Archive for the ‘All’ Category

m-buffer is a package which provides a high-level convienience API for interacting, searching and changing the contents of an Emacs buffer. It abstracts away from Emacs’ use of global state, and provides list-orientated operations removing the need for explicit looping.

Hence, we can replace:

(while (re-search-forward "foo" nil t)
     (replace-match "bar"))

with:

(m-buffer-replace-match
 (m-buffer-match (current-buffer) "foo")
 "bar")

m-buffer also protects global state, so the first form actually needs to be something closer to:

(save-excursion
  (goto-char (point-min))
  (save-match-data
    (while (re-search-forward "foo" nil t)
      (replace-match "bar"))))

There have been a few changes since the 0.10 release (http://www.russet.org.uk/blog/3059). The interface has been improved slightly (in a backward compatible way). In addition some error-checking has been added to pick up what I think was counter-intuitive behaviour. So, previously:

(m-buffer-match-page (current-buffer) :regexp "this")

would return all matches to “this” rather than pages.

The biggest change, however, has been the full use of lenticular source documentation (http://www.russet.org.uk/blog/3062), which provides a richer explanation of the code base. At the moment, I think that this is very usable, although I am aware that it is less clean than, for example, the documentation for dash, but it still is a nice demonstration of how lenticular text can work.

Bibliography

Emacs, of course, has always described itself as self-documenting because it has an integrated documentation system. With my new creation, lenticular text, (http://www.russet.org.uk/blog/3035) though the world changes slightly. I can now write rich documentation embedded into the code base of my package, and can do this without comprimising either the documentation or coding environment.

For the 0.7 release of lentic (http://www.russet.org.uk/blog/3047), I added support for generating this documentation automatically on the fly from the source code. This can be viewed either internally in Emacs with EWW, or externally with a browser. In the latter case, I have used org-info.js (http://orgmode.org/worg/code/org-info-js/) which gives an info-like and navigable experience to the manual.

As yet, this process is not perfect, and the output is not ideal, but it still seems to me to be very nice. It is certainly something that I wanted to use for m-buffer (http://github.com/phillord/m-buffer-el). So I’ve extended lentic so that its documentation system can work for any package. Much to my amazement, it’s already is use by others (http://github.com/tumashu/chinese-pyim), which has to be a good sign after such a short time span.

The main issue in making lentic-doc generic is working out how to compile the completed org-mode files into HTML. I had already considered trying to the “main” file for a package (so, lentic.el for lentic or m-buffer.el for m-buffer). But, as I discovered with lentic this does not work that well; instead I use a single top level file which imports everything else. To enable this lentic-doc uses the following technique: first it looks for a variable called package-doc describing how to compile the package documentation; if that fails, lentic looks for a file called package-doc.org in which case that is used. The main entry points are two functions lentic-doc-external-view or lentic-doc-eww-view which show the documentation in the respective browser.

There has been a lot of discussion about documentation format on the Emacs-devel mailing list recently. One common complaint was that texinfo 5 is rather too slow, taking a considerable time to compile for Emacs. Now, at first sight it would appear that moving to a lentic/org mode solution would make this worse, as the combination of the two will be much slower than even texinfo. However, lentic and org have one advantage over texinfo — they are entirely implemented in Emacs lisp, so anyone using Emacs also has the ability to generate the documentation from source; this means that the documentation can be built up lazily, on-the-fly as it is requested.

This works straighforwardly for a single package at a time — on request Emacs generates the documentation for lentic or m-buffer and displays. This still leaves the problem of cross-links between the different packages, of course. If I want to hyperlink from lentic to m-buffer, then I need to ensure that the m-buffer documentation is generated. Not so easy if the documentation is being viewed inside a browser.

Fortunately, Emacs provides a solution. Actually, two, as there are now two packages implementing a webserver for Emacs. By using an Emacs web server, I can handle any requests for package documentation, generating them on-the-fly if necessary. It turns out to be relatively simple too, as you can see here: https://github.com/phillord/lentic-server.

Of course, lentic-server is just a work in progress and far from complete, but it is pleasing none the less. The use of lentic has allowed me to tie together Org and Emacs-lisp, and to make rich documentation environment from technology that already exists.

The versions of lentic and lentic-server described here are available on MELPA (http://www.melpa.org/).

Bibliography

I have just released m-buffer version 0.10. A pretty minor point release, but I thought I would post about it, as I have not done so far.

Interacting with the text in an Emacs buffer can be a little bit painful at times. The main issue is that Emacs makes heavy use of global state its functions, something which has just rather gone out of fashion since Emacs was designed, and probably for good reason. I know that I am not the only to find this painful (http://www.wilfred.me.uk/blog/2013/03/31/essential-elisp-libraries). Perhaps, most obvious, is that many functions work on the current-buffer buffer, or at point. As a result, Emacs code tends to have lots of save-excursion and with-current-buffer calls sprinkled throughout.

We see the same thing in the regexp matching code. So for instance, the Emacs Lisp manual suggests the following:

(while (re-search-forward "foo[ \t]+bar" nil t)
   (replace-match "foobar"))

to replace all matches in a buffer. As you should be able to see, the call to replace-match contains no reference to what match is to be replaced. This is passed through the use of the global match data. Again, this made managable through the use of macros to save and restore the global state, in this case save-match-data. So, you end up with:

(save-match-data
  (save-excursion
    (while (re-search-forward "foo[ \t]+bar" nil t)
      (replace-match "foobar"))))

If you want to see this taken to extreme, have a look at the definition of m-buffer-match which is a good demonstration of the problem to which it is a solution.

The use of while is also less than ideal. It may not be obvious, but the code above contains a potential non-termination. For instance, the following should print out the position of every end-of-line in the buffer. Try it in an Emacs you don’t want.

(while (re-search-forward "$" nil t)
  (message "point at: %s" (point)))

It fails (and hangs) because the regexp $ is zero-width in the length. So, the re-search-forward moves point to just in front of the first new line, reports point, and then never moves again. This problem is a real one, and happens in real code.

m-buffer.el is my solution. Inspired partially by dash.el it provides (nearly) stateless, list orientated operations of buffer contents. So:

(m-buffer-match (current-buffer) "buffer")

returns a list of match-data to every occurrence of “buffer” in the current-buffer. There are also many, many convienience functions. So to match the end of the line, you could do:

(m-buffer-match
  (current-buffer)
  "$" :post-match 'm-buffer-post-match-forward-char)

which avoids the zero-width regexp problem, but it’s even easier to do.

(m-buffer-match-line-end (current-buffer))

Or to print to mini-buffer as before, you can instead combine with dash.el.

(--map
  (message "point at: %s" it)
 (m-buffer-match-line-end (current-buffer)))

m-buffer has been written for ease of use and not performance. So, by default, it uses markers everywhere which can potentially cause slowdowns in Emacs.I have started to add some functions for dealing with markers which need explicitly clearing explicitly; a traditional use for macros in lisp of closing existing resources. So

(m-buffer-with-markers
   ((end (m-buffer-match-line-end (current-buffer))))
 (--map
    (message "point at: %s" it)
   end))

leaves the buffer free of stray markers. The m-buffer-with-markers macro behaves like let* with cleanup.

I am now working on stateless functions for point — so m-buffer-at-point does the same as point but needs a buffer as an argument. Most of the code is trivial in this case, it just needs writing.

I am rather biased, but I like m-buffer very much. It massively simplified the development of lentic (http://www.russet.org.uk/blog/3047). Writing that with embedded while loops would have been painful in the extreme. I think I saved more time in using m-buffer than it took to write it. I think it could simplify many parts of Emacs development significantly. I hope others think so to.

m-buffer is available on MELPA and github

Bibliography

With a previous release of lentic (http://www.russet.org.uk/blog/3035), I got a couple of suggestions. One of which was a complaint that it was hard to get going, because lentic lacks documentation. This is a bit unfortunate. Lentic actually did have documentation but it is hidden away as comments in source code; although, it’s not specific to it, I wanted lentic to enable literate programming and it uses itself to document itself.

Now, this makes perfect sense, but there is a problem. The end-user form of the documentation needs generating from the source code. This is true in general in Emacs land, although the standard form is texinfo. The usual solution to this problem is to generate the documentation up during the packaging process.

This should work, but it doesn’t. Or, rather, it does not work in all cases. For an archive such as Marmalade, it is entirely possible. But for MELPA it fails. The problem is that MELPA works directly from a git checkout. My documentation, of course, is not source but generated. Now MELPA has support for the generation of info from texinfo. But my source is Emacs Lisp and I need to use Lentic to generate a readable form.

One solution would, of course, be not to use MELPA. Nic Ferrier recently argued on the emacs-devel mailing list that the idea was fundamentally broken — a package is something that the developers should generate and publish as with Java or Clojure . He makes a good point, and one that is correct. Moving to Marmalade would solve this problems; after Nic’s work it is largely stable, so this was definately an option.

However, I like MELPA (although I have only used it since stable came out). It is nice that it uses what I am doing anyway (tagging, pushing, so forth). And I like the download stats. So I talked with the MELPA folks but, entirely reasonably, they did not want to add specific support to MELPA for this. Nor support for, for example, downloading the source from somewhere else.

Other possibilities did raise themselves; I could just check in my documentation. But my documentation depends on my source, so pretty much every commit would require also require a documentation commit. Not nice. I thought about adding the documentation as an independent package. Then my documentation commits would be in a repo with nothing else; but this hassles the user, even if it auto-installs. And I’d need different packages for MELPA and Marmalade. So, I was left with no good solution.

At the same time, as all of this, I was working on the documentation, generating Org files from my lisp documentation, then converting that to info. This sort of worked, but not nicely. A significant problem was that something in the toolchain did not like having multiple sections with the same names and I have a lot of these (“Commentary, Header, Code”). I have not tracked down yet whether this is a problem with Org’s texinfo export, texinfo itself or info, nor am I sure it would be worth the effort.

Instead, I decided to try HTML output. This worked quite nicely; I use a single Org driver file (called lenticular.org) and imported all the generated org files from here. I also found org-info which I had not seen before — this is Javascript which gives an Info like experience — next, previous, occur, search and so on. It’s imperfect, but pretty usable, and gives a quite nice documentation experience. It’s also possible to view in EWW although there is no Info like paging here.

Dropping info has one other big advantage — my tool chain for generating the documentation is now entirely in Emacs. So, my source code is now enough, because lentic can generate it’s own documentation on-demand after installation. The first time the user requests the documentation either in EWW or a browser, lentic generates org files from it’s own source and then the HTML (http://homepages.cs.ncl.ac.uk/phillip.lord/lentic/lenticular.html). The only limitation is that this forces the requirement for a recent Emacs version, since the org mode exporter framework has just been updated; unfortunate but acceptable for a 0.x release.

Not all problems disappear. Because my documentation fits into the Emacs-Lisp commenting standards, it is not structured-ideally for info. For instance, the headers of all the lentic Emacs-Lisp files are included. I also would like to extend org-info so I can switch the “Code” sections on and off (embedded literate sources are useful, but not for everyone). It will need some work on Lentic, and probably also the org-mode HTML exporter.

But, then, neither is it that far off. It is good enough and a bit advance on the previous situation. Perhaps, too, it demonstrates a future for Emacs documentation in general as well, with Info replaced with HTML.

The new release (http://www.russet.org.uk/blog/3047) of Lentic is now available on Marmalade and MELPA, complete with documentation avaialble from the menu. Please feel free to try both lentic and its documentation system out now.

Bibliography

Lentic is an Emacs mode which supports multiple views over the same text. This can be used for a form of literate programming. It has specific support for Clojure which it can combine with either LaTeX, Asciidoc or Org-Mode.

Two lentic buffers, by default, the two share content but are otherwise independent. Therefore, you can have two buffers open, each showing the content in different modes; to switch modes, you simply switch buffers. The content, location of point, and view are shared.

However, lentic also allows a bi-directional transformation between lentic buffers — the buffers can have different but related text. This allows, for example, one buffer to contain an Emacs lisp file, while the other contains the same text but with “;;” comment characters removed leaving the content in org-mode, enabling a form of literate Emacs-Lisp programming with no change to either org-mode or Emacs-Lisp. Ready made transformations are also available for Clojure, latex and asciidoc.

Lentic is both configurable and extensible, using the EIEIO object system.

Lentic was previously known as Linked-Buffers.

The 0.7 release adds an integrated documentation system, support for Haskell, LaTeX literate programming and best of all, a ROT-13 transformation.

Available on MELPA-stable, MELPA and Marmalade https://github.com/phillord/lentic

As on a previous, happier, occasion, I ask for my readers indulgence for this personal post. This was my reading delivered today, 29th January, 2015.

My father once said that he could not understand how people found so much to say at funerals; at his, he said, his life story would be over in just a few lines. Perhaps, here, I will prove him right.

My father used to describe himself as a blessed man — he meant this in a very simple way, which was that he was a happy man, contented with his life and his family. My father did not want for things for physical possessions but he did want to look after the things that he had. He loved to make and build things, to define simple solutions to little problems; his garage has shelves and cupboards that he crafted like a fitted kitchen. He was happiest in his home, in the environment that he had helped to build.

As children, my father would take my brother and I on long cycle rides through the countryside around Worcester on many weekends. In my memory, these rides went on for a long time, though I was young and they cannot have been that far; but they were an enormously exciting adventure. They left me with a love of cycling that I retain to this day. In reality, of course, it was just my dad finding a creative way of getting us out of the house while my mum made dinner.

My father loved to talk with people, to find out how they were. After Sean, my son, was born, my dad spent many hours talking with my Italian father-in-law: first, they waved their hands around; then, after I showed them how, they used my computer to translate, passing it backward and forward between them. The lack of a common language was never going to get in the way of my dad having a good conversation.

When I was young, I did not think of him as a good father, just as my dad; I thought every dad was like that. Seeing him with my son made me appreciate him all the more; the sweet stupidity of an eighty year old man, crawling on all-fours with his grandson saying, “shall I make some silly noises, then”? He loved Sean beyond measure.

He had a deep consideration for others. When I told him how I had not been able to return this year at Christmas for his unexpected operation, he said that, he would have been devastated if I had, and that he hoped that he had not spoiled the holiday with his illness.

My dad’s life story may not be one of great deads or big adventures, but these things miss every thing that was important about him. He was a loving man, a kind man and a generous man. But above all, he was a gentle man. His loss leaves a hole in my, and our, lives that cannot be filled, but if when my, and our, time comes, if family and friends can say the same thing about us, then we, too, will have lead blessed lives.

William Henry Lord

Bill Lord

Dad

Thank you


Links

Death Thanks