Archive for the ‘Tech’ Category

Lentic is a package which implements lenticular text — two Emacs buffers that contain the same content, but are otherwise independent. Unlike indirect-buffers, which must contain absolutely identical strings, lentic buffers can contain different text, with a transformation between the two.

It was not my original plan to have another release so soon after the last release (http://www.russet.org.uk/blog/3068). However, the work that I had planned for that release turned out to be very-straightforward.

For this release, introduces a new form of buffer which is an unmatched block buffer. The details do not matter — the practical upshot is that with, for example, org-mode it is now possible to have more than one style of source block. In my examples directory, I have an org-mode file with “hello world” in three different languages (Clojure, Python and Emacs-Lisp). When lentic, you get four views, each in a different mode, and syntactically correct. Not a use I think I would suggest, but a nice demonstration.

Lentic is now available on MELPA, MELPA stable and github.

Bibliography

Lentic is a package which implements lenticular text — two Emacs buffers that contain the same content, but are otherwise independent. Unlike indirect-buffers, which must contain absolutely identical strings, lentic buffers can contain different text, with a transformation between the two.

This has several uses. Firstly, it allows a form of multi-modal editing, where each lentic buffer shows the text in a different mode. For example, this can be used to edit literate Haskell code. This should work with indirect-buffers, but in practice does not because the buffers share text-properties. These are a feature of the buffer strings in Emacs, and are used by some modes for their functionality; when two modes work on the same string, each tends to reset the text properties of the other.

It is possible to take this form of multi-modal editing further, where the different buffers contain different syntax. So, for example, one buffer might be in fully valid Emacs-Lisp, while the other might be a fully-valid org-mode buffer. This allows a literate programming technique even without specific support for this form of programming in the language. Taken to the extreme, it is even possible for the buffers to contain completely different strings; I have not found a good practical use for the extreme yet, but lentic now supports a rot-13 transformation which demonstrates its capabilities.

Lentic can also be used to create persistant views of the same text. For example, lentic could be used to maintain a view of the imports of a java file, or the namespace form in clojure, or the preamble of a latex document. Unlike a second window, this view persists even if it is not visible. Alternatively, one view could use very small text, and the other could contain larger text, allowing rapid navigation.

Lentic 0.8 contains a number of new features since the 0.7 [release] (http://www.russet.org.uk/blog/3047). The biggest change is that it is possible to produce any number of lentic buffers, rather than just two as previously. This means that its multi-modal and persistant view capabilities can be used at the same time.

Lentic is now available on MELPA, MELPA stable and github.

Bibliography

m-buffer is a package which provides a high-level convienience API for interacting, searching and changing the contents of an Emacs buffer. It abstracts away from Emacs’ use of global state, and provides list-orientated operations removing the need for explicit looping.

Hence, we can replace:

(while (re-search-forward "foo" nil t)
     (replace-match "bar"))

with:

(m-buffer-replace-match
 (m-buffer-match (current-buffer) "foo")
 "bar")

m-buffer also protects global state, so the first form actually needs to be something closer to:

(save-excursion
  (goto-char (point-min))
  (save-match-data
    (while (re-search-forward "foo" nil t)
      (replace-match "bar"))))

There have been a few changes since the 0.10 release (http://www.russet.org.uk/blog/3059). The interface has been improved slightly (in a backward compatible way). In addition some error-checking has been added to pick up what I think was counter-intuitive behaviour. So, previously:

(m-buffer-match-page (current-buffer) :regexp "this")

would return all matches to “this” rather than pages.

The biggest change, however, has been the full use of lenticular source documentation (http://www.russet.org.uk/blog/3062), which provides a richer explanation of the code base. At the moment, I think that this is very usable, although I am aware that it is less clean than, for example, the documentation for dash, but it still is a nice demonstration of how lenticular text can work.

Bibliography

Emacs, of course, has always described itself as self-documenting because it has an integrated documentation system. With my new creation, lenticular text, (http://www.russet.org.uk/blog/3035) though the world changes slightly. I can now write rich documentation embedded into the code base of my package, and can do this without comprimising either the documentation or coding environment.

For the 0.7 release of lentic (http://www.russet.org.uk/blog/3047), I added support for generating this documentation automatically on the fly from the source code. This can be viewed either internally in Emacs with EWW, or externally with a browser. In the latter case, I have used org-info.js (http://orgmode.org/worg/code/org-info-js/) which gives an info-like and navigable experience to the manual.

As yet, this process is not perfect, and the output is not ideal, but it still seems to me to be very nice. It is certainly something that I wanted to use for m-buffer (http://github.com/phillord/m-buffer-el). So I’ve extended lentic so that its documentation system can work for any package. Much to my amazement, it’s already is use by others (http://github.com/tumashu/chinese-pyim), which has to be a good sign after such a short time span.

The main issue in making lentic-doc generic is working out how to compile the completed org-mode files into HTML. I had already considered trying to the “main” file for a package (so, lentic.el for lentic or m-buffer.el for m-buffer). But, as I discovered with lentic this does not work that well; instead I use a single top level file which imports everything else. To enable this lentic-doc uses the following technique: first it looks for a variable called package-doc describing how to compile the package documentation; if that fails, lentic looks for a file called package-doc.org in which case that is used. The main entry points are two functions lentic-doc-external-view or lentic-doc-eww-view which show the documentation in the respective browser.

There has been a lot of discussion about documentation format on the Emacs-devel mailing list recently. One common complaint was that texinfo 5 is rather too slow, taking a considerable time to compile for Emacs. Now, at first sight it would appear that moving to a lentic/org mode solution would make this worse, as the combination of the two will be much slower than even texinfo. However, lentic and org have one advantage over texinfo — they are entirely implemented in Emacs lisp, so anyone using Emacs also has the ability to generate the documentation from source; this means that the documentation can be built up lazily, on-the-fly as it is requested.

This works straighforwardly for a single package at a time — on request Emacs generates the documentation for lentic or m-buffer and displays. This still leaves the problem of cross-links between the different packages, of course. If I want to hyperlink from lentic to m-buffer, then I need to ensure that the m-buffer documentation is generated. Not so easy if the documentation is being viewed inside a browser.

Fortunately, Emacs provides a solution. Actually, two, as there are now two packages implementing a webserver for Emacs. By using an Emacs web server, I can handle any requests for package documentation, generating them on-the-fly if necessary. It turns out to be relatively simple too, as you can see here: https://github.com/phillord/lentic-server.

Of course, lentic-server is just a work in progress and far from complete, but it is pleasing none the less. The use of lentic has allowed me to tie together Org and Emacs-lisp, and to make rich documentation environment from technology that already exists.

The versions of lentic and lentic-server described here are available on MELPA (http://www.melpa.org/).

Bibliography

I have just released m-buffer version 0.10. A pretty minor point release, but I thought I would post about it, as I have not done so far.

Interacting with the text in an Emacs buffer can be a little bit painful at times. The main issue is that Emacs makes heavy use of global state its functions, something which has just rather gone out of fashion since Emacs was designed, and probably for good reason. I know that I am not the only to find this painful (http://www.wilfred.me.uk/blog/2013/03/31/essential-elisp-libraries). Perhaps, most obvious, is that many functions work on the current-buffer buffer, or at point. As a result, Emacs code tends to have lots of save-excursion and with-current-buffer calls sprinkled throughout.

We see the same thing in the regexp matching code. So for instance, the Emacs Lisp manual suggests the following:

(while (re-search-forward "foo[ \t]+bar" nil t)
   (replace-match "foobar"))

to replace all matches in a buffer. As you should be able to see, the call to replace-match contains no reference to what match is to be replaced. This is passed through the use of the global match data. Again, this made managable through the use of macros to save and restore the global state, in this case save-match-data. So, you end up with:

(save-match-data
  (save-excursion
    (while (re-search-forward "foo[ \t]+bar" nil t)
      (replace-match "foobar"))))

If you want to see this taken to extreme, have a look at the definition of m-buffer-match which is a good demonstration of the problem to which it is a solution.

The use of while is also less than ideal. It may not be obvious, but the code above contains a potential non-termination. For instance, the following should print out the position of every end-of-line in the buffer. Try it in an Emacs you don’t want.

(while (re-search-forward "$" nil t)
  (message "point at: %s" (point)))

It fails (and hangs) because the regexp $ is zero-width in the length. So, the re-search-forward moves point to just in front of the first new line, reports point, and then never moves again. This problem is a real one, and happens in real code.

m-buffer.el is my solution. Inspired partially by dash.el it provides (nearly) stateless, list orientated operations of buffer contents. So:

(m-buffer-match (current-buffer) "buffer")

returns a list of match-data to every occurrence of “buffer” in the current-buffer. There are also many, many convienience functions. So to match the end of the line, you could do:

(m-buffer-match
  (current-buffer)
  "$" :post-match 'm-buffer-post-match-forward-char)

which avoids the zero-width regexp problem, but it’s even easier to do.

(m-buffer-match-line-end (current-buffer))

Or to print to mini-buffer as before, you can instead combine with dash.el.

(--map
  (message "point at: %s" it)
 (m-buffer-match-line-end (current-buffer)))

m-buffer has been written for ease of use and not performance. So, by default, it uses markers everywhere which can potentially cause slowdowns in Emacs.I have started to add some functions for dealing with markers which need explicitly clearing explicitly; a traditional use for macros in lisp of closing existing resources. So

(m-buffer-with-markers
   ((end (m-buffer-match-line-end (current-buffer))))
 (--map
    (message "point at: %s" it)
   end))

leaves the buffer free of stray markers. The m-buffer-with-markers macro behaves like let* with cleanup.

I am now working on stateless functions for point — so m-buffer-at-point does the same as point but needs a buffer as an argument. Most of the code is trivial in this case, it just needs writing.

I am rather biased, but I like m-buffer very much. It massively simplified the development of lentic (http://www.russet.org.uk/blog/3047). Writing that with embedded while loops would have been painful in the extreme. I think I saved more time in using m-buffer than it took to write it. I think it could simplify many parts of Emacs development significantly. I hope others think so to.

m-buffer is available on MELPA and github

Bibliography

With a previous release of lentic (http://www.russet.org.uk/blog/3035), I got a couple of suggestions. One of which was a complaint that it was hard to get going, because lentic lacks documentation. This is a bit unfortunate. Lentic actually did have documentation but it is hidden away as comments in source code; although, it’s not specific to it, I wanted lentic to enable literate programming and it uses itself to document itself.

Now, this makes perfect sense, but there is a problem. The end-user form of the documentation needs generating from the source code. This is true in general in Emacs land, although the standard form is texinfo. The usual solution to this problem is to generate the documentation up during the packaging process.

This should work, but it doesn’t. Or, rather, it does not work in all cases. For an archive such as Marmalade, it is entirely possible. But for MELPA it fails. The problem is that MELPA works directly from a git checkout. My documentation, of course, is not source but generated. Now MELPA has support for the generation of info from texinfo. But my source is Emacs Lisp and I need to use Lentic to generate a readable form.

One solution would, of course, be not to use MELPA. Nic Ferrier recently argued on the emacs-devel mailing list that the idea was fundamentally broken — a package is something that the developers should generate and publish as with Java or Clojure . He makes a good point, and one that is correct. Moving to Marmalade would solve this problems; after Nic’s work it is largely stable, so this was definately an option.

However, I like MELPA (although I have only used it since stable came out). It is nice that it uses what I am doing anyway (tagging, pushing, so forth). And I like the download stats. So I talked with the MELPA folks but, entirely reasonably, they did not want to add specific support to MELPA for this. Nor support for, for example, downloading the source from somewhere else.

Other possibilities did raise themselves; I could just check in my documentation. But my documentation depends on my source, so pretty much every commit would require also require a documentation commit. Not nice. I thought about adding the documentation as an independent package. Then my documentation commits would be in a repo with nothing else; but this hassles the user, even if it auto-installs. And I’d need different packages for MELPA and Marmalade. So, I was left with no good solution.

At the same time, as all of this, I was working on the documentation, generating Org files from my lisp documentation, then converting that to info. This sort of worked, but not nicely. A significant problem was that something in the toolchain did not like having multiple sections with the same names and I have a lot of these (“Commentary, Header, Code”). I have not tracked down yet whether this is a problem with Org’s texinfo export, texinfo itself or info, nor am I sure it would be worth the effort.

Instead, I decided to try HTML output. This worked quite nicely; I use a single Org driver file (called lenticular.org) and imported all the generated org files from here. I also found org-info which I had not seen before — this is Javascript which gives an Info like experience — next, previous, occur, search and so on. It’s imperfect, but pretty usable, and gives a quite nice documentation experience. It’s also possible to view in EWW although there is no Info like paging here.

Dropping info has one other big advantage — my tool chain for generating the documentation is now entirely in Emacs. So, my source code is now enough, because lentic can generate it’s own documentation on-demand after installation. The first time the user requests the documentation either in EWW or a browser, lentic generates org files from it’s own source and then the HTML (http://homepages.cs.ncl.ac.uk/phillip.lord/lentic/lenticular.html). The only limitation is that this forces the requirement for a recent Emacs version, since the org mode exporter framework has just been updated; unfortunate but acceptable for a 0.x release.

Not all problems disappear. Because my documentation fits into the Emacs-Lisp commenting standards, it is not structured-ideally for info. For instance, the headers of all the lentic Emacs-Lisp files are included. I also would like to extend org-info so I can switch the “Code” sections on and off (embedded literate sources are useful, but not for everyone). It will need some work on Lentic, and probably also the org-mode HTML exporter.

But, then, neither is it that far off. It is good enough and a bit advance on the previous situation. Perhaps, too, it demonstrates a future for Emacs documentation in general as well, with Info replaced with HTML.

The new release (http://www.russet.org.uk/blog/3047) of Lentic is now available on Marmalade and MELPA, complete with documentation avaialble from the menu. Please feel free to try both lentic and its documentation system out now.

Bibliography