Archive for the ‘Tech’ Category

While I have been working on a manual called Take Wing for Tawny-OWL (http://www.russet.org.uk/blog/3030) for some time, it is far from finished. In the meantime, I am giving a tutorial at this years ICBO, and the slides for this are now relatively advanced, although I have a few more sections and some checking to do. The full tutorial is available (http://github.com/phillord/tawny-tutorial), and I think it offers are fairly comprehensive guide to basic Tawny-OWL usage.

I am fairly pleased with the tutorial as it stands. It is written up as a semantic document using my lentic package (http://www.russet.org.uk/blog/3047) which seems to be working well (although I discovered a few bugs with the asciidoc support in the course of writing the tutorial!).

For anyone who is thinking of coming, the pre-requisities for the tutorial are now online. I am planning to do a large part of the tutorial in a “follow-my-leader” fashion; it’s always difficult to predict the state of the networks at these events, so it would help significantly if at least some of the people coming could work through this short document before hand.

Bibliography

Lentic is a package which implements lenticular text — two Emacs buffers that contain the same content, but are otherwise independent. Unlike indirect-buffers, which must contain absolutely identical strings, lentic buffers can contain different text, with a transformation between the two.

It was not my original plan to have another release so soon after the last release (http://www.russet.org.uk/blog/3068). However, the work that I had planned for that release turned out to be very-straightforward.

For this release, introduces a new form of buffer which is an unmatched block buffer. The details do not matter — the practical upshot is that with, for example, org-mode it is now possible to have more than one style of source block. In my examples directory, I have an org-mode file with “hello world” in three different languages (Clojure, Python and Emacs-Lisp). When lentic, you get four views, each in a different mode, and syntactically correct. Not a use I think I would suggest, but a nice demonstration.

Lentic is now available on MELPA, MELPA stable and github.

Bibliography

Lentic is a package which implements lenticular text — two Emacs buffers that contain the same content, but are otherwise independent. Unlike indirect-buffers, which must contain absolutely identical strings, lentic buffers can contain different text, with a transformation between the two.

This has several uses. Firstly, it allows a form of multi-modal editing, where each lentic buffer shows the text in a different mode. For example, this can be used to edit literate Haskell code. This should work with indirect-buffers, but in practice does not because the buffers share text-properties. These are a feature of the buffer strings in Emacs, and are used by some modes for their functionality; when two modes work on the same string, each tends to reset the text properties of the other.

It is possible to take this form of multi-modal editing further, where the different buffers contain different syntax. So, for example, one buffer might be in fully valid Emacs-Lisp, while the other might be a fully-valid org-mode buffer. This allows a literate programming technique even without specific support for this form of programming in the language. Taken to the extreme, it is even possible for the buffers to contain completely different strings; I have not found a good practical use for the extreme yet, but lentic now supports a rot-13 transformation which demonstrates its capabilities.

Lentic can also be used to create persistant views of the same text. For example, lentic could be used to maintain a view of the imports of a java file, or the namespace form in clojure, or the preamble of a latex document. Unlike a second window, this view persists even if it is not visible. Alternatively, one view could use very small text, and the other could contain larger text, allowing rapid navigation.

Lentic 0.8 contains a number of new features since the 0.7 [release] (http://www.russet.org.uk/blog/3047). The biggest change is that it is possible to produce any number of lentic buffers, rather than just two as previously. This means that its multi-modal and persistant view capabilities can be used at the same time.

Lentic is now available on MELPA, MELPA stable and github.

Bibliography

m-buffer is a package which provides a high-level convienience API for interacting, searching and changing the contents of an Emacs buffer. It abstracts away from Emacs’ use of global state, and provides list-orientated operations removing the need for explicit looping.

Hence, we can replace:

(while (re-search-forward "foo" nil t)
     (replace-match "bar"))

with:

(m-buffer-replace-match
 (m-buffer-match (current-buffer) "foo")
 "bar")

m-buffer also protects global state, so the first form actually needs to be something closer to:

(save-excursion
  (goto-char (point-min))
  (save-match-data
    (while (re-search-forward "foo" nil t)
      (replace-match "bar"))))

There have been a few changes since the 0.10 release (http://www.russet.org.uk/blog/3059). The interface has been improved slightly (in a backward compatible way). In addition some error-checking has been added to pick up what I think was counter-intuitive behaviour. So, previously:

(m-buffer-match-page (current-buffer) :regexp "this")

would return all matches to “this” rather than pages.

The biggest change, however, has been the full use of lenticular source documentation (http://www.russet.org.uk/blog/3062), which provides a richer explanation of the code base. At the moment, I think that this is very usable, although I am aware that it is less clean than, for example, the documentation for dash, but it still is a nice demonstration of how lenticular text can work.

Bibliography

Emacs, of course, has always described itself as self-documenting because it has an integrated documentation system. With my new creation, lenticular text, (http://www.russet.org.uk/blog/3035) though the world changes slightly. I can now write rich documentation embedded into the code base of my package, and can do this without comprimising either the documentation or coding environment.

For the 0.7 release of lentic (http://www.russet.org.uk/blog/3047), I added support for generating this documentation automatically on the fly from the source code. This can be viewed either internally in Emacs with EWW, or externally with a browser. In the latter case, I have used org-info.js (http://orgmode.org/worg/code/org-info-js/) which gives an info-like and navigable experience to the manual.

As yet, this process is not perfect, and the output is not ideal, but it still seems to me to be very nice. It is certainly something that I wanted to use for m-buffer (http://github.com/phillord/m-buffer-el). So I’ve extended lentic so that its documentation system can work for any package. Much to my amazement, it’s already is use by others (http://github.com/tumashu/chinese-pyim), which has to be a good sign after such a short time span.

The main issue in making lentic-doc generic is working out how to compile the completed org-mode files into HTML. I had already considered trying to the “main” file for a package (so, lentic.el for lentic or m-buffer.el for m-buffer). But, as I discovered with lentic this does not work that well; instead I use a single top level file which imports everything else. To enable this lentic-doc uses the following technique: first it looks for a variable called package-doc describing how to compile the package documentation; if that fails, lentic looks for a file called package-doc.org in which case that is used. The main entry points are two functions lentic-doc-external-view or lentic-doc-eww-view which show the documentation in the respective browser.

There has been a lot of discussion about documentation format on the Emacs-devel mailing list recently. One common complaint was that texinfo 5 is rather too slow, taking a considerable time to compile for Emacs. Now, at first sight it would appear that moving to a lentic/org mode solution would make this worse, as the combination of the two will be much slower than even texinfo. However, lentic and org have one advantage over texinfo — they are entirely implemented in Emacs lisp, so anyone using Emacs also has the ability to generate the documentation from source; this means that the documentation can be built up lazily, on-the-fly as it is requested.

This works straighforwardly for a single package at a time — on request Emacs generates the documentation for lentic or m-buffer and displays. This still leaves the problem of cross-links between the different packages, of course. If I want to hyperlink from lentic to m-buffer, then I need to ensure that the m-buffer documentation is generated. Not so easy if the documentation is being viewed inside a browser.

Fortunately, Emacs provides a solution. Actually, two, as there are now two packages implementing a webserver for Emacs. By using an Emacs web server, I can handle any requests for package documentation, generating them on-the-fly if necessary. It turns out to be relatively simple too, as you can see here: https://github.com/phillord/lentic-server.

Of course, lentic-server is just a work in progress and far from complete, but it is pleasing none the less. The use of lentic has allowed me to tie together Org and Emacs-lisp, and to make rich documentation environment from technology that already exists.

The versions of lentic and lentic-server described here are available on MELPA (http://www.melpa.org/).

Bibliography

I have just released m-buffer version 0.10. A pretty minor point release, but I thought I would post about it, as I have not done so far.

Interacting with the text in an Emacs buffer can be a little bit painful at times. The main issue is that Emacs makes heavy use of global state its functions, something which has just rather gone out of fashion since Emacs was designed, and probably for good reason. I know that I am not the only to find this painful (http://www.wilfred.me.uk/blog/2013/03/31/essential-elisp-libraries). Perhaps, most obvious, is that many functions work on the current-buffer buffer, or at point. As a result, Emacs code tends to have lots of save-excursion and with-current-buffer calls sprinkled throughout.

We see the same thing in the regexp matching code. So for instance, the Emacs Lisp manual suggests the following:

(while (re-search-forward "foo[ \t]+bar" nil t)
   (replace-match "foobar"))

to replace all matches in a buffer. As you should be able to see, the call to replace-match contains no reference to what match is to be replaced. This is passed through the use of the global match data. Again, this made managable through the use of macros to save and restore the global state, in this case save-match-data. So, you end up with:

(save-match-data
  (save-excursion
    (while (re-search-forward "foo[ \t]+bar" nil t)
      (replace-match "foobar"))))

If you want to see this taken to extreme, have a look at the definition of m-buffer-match which is a good demonstration of the problem to which it is a solution.

The use of while is also less than ideal. It may not be obvious, but the code above contains a potential non-termination. For instance, the following should print out the position of every end-of-line in the buffer. Try it in an Emacs you don’t want.

(while (re-search-forward "$" nil t)
  (message "point at: %s" (point)))

It fails (and hangs) because the regexp $ is zero-width in the length. So, the re-search-forward moves point to just in front of the first new line, reports point, and then never moves again. This problem is a real one, and happens in real code.

m-buffer.el is my solution. Inspired partially by dash.el it provides (nearly) stateless, list orientated operations of buffer contents. So:

(m-buffer-match (current-buffer) "buffer")

returns a list of match-data to every occurrence of “buffer” in the current-buffer. There are also many, many convienience functions. So to match the end of the line, you could do:

(m-buffer-match
  (current-buffer)
  "$" :post-match 'm-buffer-post-match-forward-char)

which avoids the zero-width regexp problem, but it’s even easier to do.

(m-buffer-match-line-end (current-buffer))

Or to print to mini-buffer as before, you can instead combine with dash.el.

(--map
  (message "point at: %s" it)
 (m-buffer-match-line-end (current-buffer)))

m-buffer has been written for ease of use and not performance. So, by default, it uses markers everywhere which can potentially cause slowdowns in Emacs.I have started to add some functions for dealing with markers which need explicitly clearing explicitly; a traditional use for macros in lisp of closing existing resources. So

(m-buffer-with-markers
   ((end (m-buffer-match-line-end (current-buffer))))
 (--map
    (message "point at: %s" it)
   end))

leaves the buffer free of stray markers. The m-buffer-with-markers macro behaves like let* with cleanup.

I am now working on stateless functions for point — so m-buffer-at-point does the same as point but needs a buffer as an argument. Most of the code is trivial in this case, it just needs writing.

I am rather biased, but I like m-buffer very much. It massively simplified the development of lentic (http://www.russet.org.uk/blog/3047). Writing that with embedded while loops would have been painful in the extreme. I think I saved more time in using m-buffer than it took to write it. I think it could simplify many parts of Emacs development significantly. I hope others think so to.

m-buffer is available on MELPA and github

Bibliography