L'icule gazouilleur: Thoughts on editors

(Emacs in particular)

Contents

When, after having been a passionate Emacs user for more than two decades, I switched to Vim for my day to day work, many friends and co-workers were a bit astonished. Even if I never dived into holy wars about which editor is best, some people knew my long reluctance (should I say repulsion?) to any kind of vi. Besides, with a few closer people, Emacs versus Vim was the frequent subject of friendly teases. Many people asked me to explain what they perceive as a defection, and this Web page is meant to summarise my thoughts on the matter.

However, if I had to concentrate my overall experience into a single piece of advice, it would go like this. Whatever editor you choose, learn it well. A good part of the comfort you have while programming comes from the ease you develop with an editor. It has to somehow become part of yourself. It is much better to know one well, then many partially. Yet, choose an editor with many virtues, so it will be on your side in the long run.

If you grok French, you might like to read about my very first contact with Emacs. At the time I wrote this article, I was known around here as an Emacs proponent and specialist, and this is why the article was read as humorous.

1.2 Emacs virtues

Emacs is really a wonderful editor, and newcomers to Emacs just cannot really appreciate its incredible power and versatility. The Emacs tutorial only gives the bare minimum so one is able to enter and leave the editor and do simple edit jobs. Even if the tutorial gives the base for handling many files, buffers and windows, it is likely to take some while before one becomes able to comfortably and confidently edit hundreds of files, simultaneously ☺.

The tutorial does not give a clear idea of the powerfulness of program editing modes. I especially appreciate Emacs capacity for cleverly indenting source code, moving in and out structural units, and also its ability to precisely locate (and automatically position the cursor at) syntactic errors within source files, without even having to wait until compilation has finished. So-called shell windows are extremely and constantly useful, where one may edit system commands and receive interaction results right in an editable window. And this is equally true for remote editing and execution.

There are astonishing tools in Emacs. For example, one may work on a mathematical text, and after having a written numerical, algebraic or statistical formulas, Emacs may compute the integral, resolve an equation system numerically or symbolically, do a Taylor expansion, display graphics, and many other things just as well. Emacs also offers an integrated, simultaneous view over the email one receives, Usenet News, and Web searches, so one may read, sort, process or reply to these, with a command set which likely goes beyond, in capacity, everything else one might have seen.

Finally, and this is important, there is about no limit to how one may fine tune Emacs behaviour no matter how detailed or extravagant your wishes. Besides all the predefined variables and knobs, one may override or augment on the fly the set of available commands, without the need of recompiling anything. So, if something does not fully please you, you have the means for satisfaction.

Someone was wondering if he would ever learn 10% of the Emacs keystrokes. He did not realise than pre-made keystrokes are only a small part of all available commands. And then, that all available commands are only a small part of all available functions. And, of course, that all of the above is only a small drop in the ocean of possibilities given by the extensible properties of Emacs. At times, I wonder if this should be seen as comforting or frightening. ☺

People sometimes ask me if Emacs is worth learning. One can easily get started with Emacs, but really understanding Emacs capabilities is another matter, for which the learning curve is either steep, or has to extend over a long period of time. I'm quite sure that Emacs is well worth learning, if you use computers a great deal in your professional life. Once you know it well, it speeds you up tremendously. For people using computers only casually, I sometimes feel Emacs might be overkill. It happens that I see some of my coworkers doing long, tedious and boring editing tasks using lesser editors. While observing their sorrow, I know it could have been easier. They apparently know just as well, as they sometimes ask me (and so, Emacs) for help! ☺

1.3 Why not Emacs, then?

One problem is that both Emacs and I are growing old. For Emacs, it means that its complexity is progressively increasing, and for me, that this ability to understand complex things is slowly melting. When we were both younger, I was frequently diving into Emacs sources, even C sources, for repairing the little nits I was finding, or for implementing a few ideas that were dear to me, all this with a reasonable confidence that I would succeed. But this is not true anymore.

For example, I came to much like the graphical user interface features of Emacs while working under X, even if Emacs is still very usable in more traditional text-based user interfaces. Emacs addresses the graphical facilities at Xlib level, most probably to guarantee speed, but this low level for X is foreign to me, and if I ever correctly learn to do GUI programming, I would prefer doing it at a much higher level. I reported a few tiny problems against the Emacs GUI, that are not going to be corrected until someone knowledgeable enough in this area tackles the issues. Mule, the international character support in Emacs, is in constant state of slow redesign, so its documentation is always a bit lagging. Besides very few dedicated or specialised people, no one could really say they understand Mule. Mule also ignored Unicode, and Unicode support in Emacs is currently partial, and might be for a while. I'm far from being a Unicode fanatic, but yet, something as important as Emacs may not ignore Unicode nowadays. There are Mule bugs in Emacs by which \201 characters pop up unexpectedly, here and there, which I reported many times. I have many filters that get rid of thousands of these in my files, nearly transparently, so they do not bother me that much overall.

However, what bothers me is the growing impression that being an Emacs maintainer, as time passes, becomes an increasingly difficult challenge. A lot of technical skills are required, not to speak about the ability to peacefully coordinate the work of really many volunteers. I have a lot of admiration for Emacs maintainers, yet more it goes, more it becomes impossible to recruit the right guys. And even for the right guys, organising a new distribution has become a daunting task. Most maintainers exhaust themselves at the job, and never do more than a few releases in a row. Forming a new Emacs maintainer takes a long time.

1.4 The Lisp in Emacs

I surely wrote a fair amount of Emacs Lisp code in my computer life. This language has plenty of useful features, abundant documentation, good debuggers, nice profilers, and is assorted with many good writing and stylistic conventions. In fact, this Lisp was more than once the language in which I chose to express, extensively, the algorithmic solution of some problems which were somehow related to editing tasks. But also, I sometimes needed the very same algorithms in contexts not really tied to editing. The practical choices left to me were either to artificially push myself into Emacs more often than I really needed to, or else, to execute some stunts for calling Emacs from within batch scripts. I did both for years actually, yet with the constant feeling that Emacs Lisp was not my preferred programming language, however sophisticated it may be. In fact, the language was a bit forced upon me as I sometimes or often needed these algorithms within editing contexts.

There is in Emacs a Common Lisp compatibility package, which I had to fight sometimes while writing Emacs Lisp because of naming clashes, which existed even between standard Emacs modules. Through various bug reports, these clashes were lifted and resolved along the years, but yet, I perceived the confusion as a proof of some untamed complexity. Besides a few useful macros, which Gnus amply used and demonstrated, and which were later integrated in Emacs proper, most of the Common Lisp compatibility package seemed to me like another layer of learning over something already featureful.

As far as programming languages are concerned, for all those I studied, I noticed my overall preference for those having simpler syntaxes or clean libraries. For example, I was much more on the Pascal side than on the PL/I side. Oh, no doubt that C kept me busy for a long while, it is a necessary evil when one aims wide portability. But deep down, Scheme attracted me much more. So, when Richard Stallman announced that all extensible package in GNU would eventually use Scheme, it was good news.

However, the news was not a good one for very long. We soon realised that the vision was to be a mix of Scheme and current Emacs Lisp, shamelessly altering the Scheme standard wherever it seemed interesting to do so. The result was to be named Guile, and for creating its first release, the FSF appointed a strange maintainer who was far from respecting GNU standards. (A lot of straightening occurred later, when more reasonable people worked on the project, but in my mind, the huge false start was hardly repairable.) Whenever Guile will find its way into Emacs, another layer of complexity and confusion will be created, as Emacs will host a third flavour of Lisp, and I predict that this particular confusion will last for quite a while before Guile takes over for real. Especially given, at least to my intuition, that Emacs will not likely move to Guile soon. Let me explain a bit more.

When the FSF decided that all GNU packages would switch to Rx, a replacement for regexp, it was also decided that Emacs would be one of the last to switch, as Richard did want Rx to be debugged by all other maintainers (like me) first. In his defense, Emacs could not afford being destabilised by a package like Rx. (Since then, Rx has been pronounced a failure, so I spent my time supporting it in my packages.) Despite GNU gettext has been in existence for a while now, Emacs is not using it yet; and it has been known for years that it will use it in its own special ways. I guess that Guile is just the same, Emacs will follow its acceptance by other packages in GNU, much more than it will precede them. For me, all this means that in some indeterminate future, Emacs is doomed to traverse difficult times.

1.5 Emacs and Python

In my long saga for a good programming language for day-to-day works, after Bash, GNU m4, Awk, Perl and Scheme, I finally chose Python. Python is not only good for quick scripting, it allows me to tackle big projects in a structured way, producing legible and very maintainable prototypes. Combined with profiling, then Pyrex, it also well sustains the stress of non-prototype, production environments.

One thing that happened is that I got more interested in Python than Lisp, even though I wrote a lot of Lisp. Emacs ties me to Lisp, while I knew many things I wrote in Lisp would be better written in Python. This is the first time I seriously considered getting away from Emacs, towards an editor which is more Python-aware. Looking around, I found nothing much convincing (I did not even consider vi, because I never felt comfortable with it in the past). It was also true that my Emacs customisation was rather huge by then, and I had a lot of Emacs dependencies in my work habits. I stuck with Emacs.

But I was not satisfied, and felt that the Emacs dependency grew to the point of not being that comfortable. I was still starving for Python instead of Lisp. Some people tried integrating Python within Emacs, failed, and were kind enough to tell me why. One showed me an experiment of a communication protocol between Python and Emacs kept as separate processes, as a proof of concept. I re-designed the protocol so it gets much more clean and simple, more practical than theoretical, and added clarity and elegance to the API. This has been the birth of Pymacs. Other correspondents helped me at solving some uneasy problems related to garbage collection on Emacs side.

While Pymacs is elegant in my opinion, one cannot effectively use Pymacs (the Python part) without knowing at least the specification of many Lisp functions, and I found that it requires some doing for a Pymacs developer to decouple the Emacs interaction part from the purer algorithmic part in applications. Moreover, if you do not consider speed issues, they bite you.

Guile, as being a long term direction for the FSF, is also an obstacle to any contradicting direction. Even if I, and a few of us, would consider than a tighter integration of Emacs and Python would be a nice thing, it would probably be perceived as an unwelcome distraction from the Guile plan by the FSF. And since Guile might take forever to really find its way within Emacs, Python may not become a real extension language for Emacs in our lifetime.

In September 2003, I worked at extending the Emacs Lilypond mode (Lilypond is a musical score editing system) for my own precise needs, and naturally for me, wrote these extensions in Python, using Pymacs. A few users asked if and how the extensions I wrote could be made available for Vim as well, whatever the way. One of them insisted so much that, one evening I was too tired for being useful at anything, I peeked at Vim and its Python extensibility. Although the Vim API is much weaker than Pymacs (so far that I know), the integration with Vim seemed more tight and Pythonesque. Not having to feel so much the underneath presence of Vim, at least compared to Pymacs, writing purer Python promised to be easier. The promise was so strong that I pondered for a while the idea of rewriting all Pymacs examples included in its distribution so they could be usable from Vim as well as from Emacs. I have not yet done it, but am letting the idea mature.

The good thing is that I finally recognised extensibility as being more important than the editor, given the editor is reasonable enough to start with. Whatever the editor, I will extend it, as I have so many personal habits. Python is easy to write, and Python modules are highly reusable. I took a better look at Vim itself, and it basically offers (more than vi) most things I use in Emacs all the time. Many things are still lacking for me, but Vim seems mature enough to allow them as extensions. Vim is also more in-lined with GTK and Pango, X input methods, Unicode, and those internationalisation things which were making Emacs a distinguished tool not so long ago.

I'm curious about exploring how to work in an environment of many modular tools, and learning to interact between all these tools using common idioms, like cut-and-pasting between heterogenous applications, rather than living within a single huge do-it-all design. I know that there is suffering ahead for me while re-learning to work differently, but I may not experiment these new directions (new to me, I mean!) if I continue sticking with Emacs. So, I guess, still hesitating, not fully sure, that I'll have to accept some pain sooner than later and dare diving away.

1.6 Emacs as a cargo

When Richard Stallman started the GNU project (and the FSF), long before Linux existed, he established through GNU Manifesto and standards, a firm direction, but especially through programming examples, a quality level which astonished the community and enthused many good programmers. People received Emacs, the C compiler and debugger, Make, Bison, and a flurry of other tools, as wonders. The GNU project, behind the charismatic figure that Richard was then, channelled a great deal of interest and energy. Everybody had such a strong quality prejudice for anything coming out of the FSF, that new GNU programs were immediately adopted and installed world-wide. Richard Stallman once had an immense moral authority in the world of free programming. Wherever he was going, everybody followed.

People did not always agree, but so yearned to attain the goal of a complete, free operating system, that they managed to unite despite their differences. However, when this system started to exist, and so, when the need of reaching the goal started to fade, Richard's authority soon began to dismantle. Especially since Linux has been created quite outside the FSF wing, and was initially considered by the FSF as an unwelcome drain of energies away from the Hurd, which was then the FSF project of choice. Linux development style, a kind of open collaboration between a lot of trends, was also quite different from GNU habits, and the Linux community ferociously resisted the GNU model of authority. At the time, the FSF made itself many enemies it did not need. To destroy its authority a bit more, the GNU project allowed in a few projects of lesser quality, and never paid much attention to the human qualities of its maintainers. Software is less free than it appears to be when maintainers are haughty or despising.

The FSF never fully recovered from all the damage, nor adapted to the changes. Used for a good while to going its own way without much looking around, with the planet blindly following behind, the FSF did not recognise soon enough that it will loose followers if it does not learn to really collaborate as an equal with other partners, instead of as the necessary leader of everything. A lot of worth efforts are going out in this world, which did not originate from the GNU project and which are not driven by the GNU project. (These projects might have never existed without the prior momentum created by GNU and the GPL, but this is another debate.) GNU traditionally ignores what it does not drive, until it cannot escape it. For Emacs, it means that while collaborators are expected, Emacs itself is not going to be especially collaborative.

I feel that Emacs is a heavy cargo that, even if it used to establish a direction for everybody, might now have trouble manoeuvring along directions established elsewhere, because of its terrible inertia, both political and technical. Not that it does move, but a great deal of energy is required every time.

All summarised, I perceive Emacs as one of those huge ships where people embark a luxurious life while they intend to travel around the planet. However, the ship is an heavy cargo for which it is difficult to find captains, and which consumes a great deal of fuel to effect only slight changes in direction. This cargo is not fully tracked on the mainstream routes, where the real crowd is, all active and diverse than it may be. Because captains are not that concerned with the mainstream, and also because when corrections become unavoidable, the inertia of the cargo is such that it always lags a bit from the aimed route. So this cargo very slowly diverges, progressively isolating its happy passengers from the rest of the world.

It is not easy for any passenger, abandoning all the comfort they currently enjoy, diving in the cold waters and courageously swimming back to where the bazaar really stands, and learning to survive in the happy chaos which rules there. One sure thing is that more one waits, longer and harder the swim will be. This particular concern finally triggered me into abandoning a non-sinking ship, or at least, leaving it long before we begin to perceive a degradation of life quality aboard, yet this degradation is likely unavoidable in the long run.

2 Surviving without Emacs

2.1 The overall plan

Editors are addictive. That's why so many people have quasi-religious feeling about them. They feel threatened very deeply whenever they see criticism. Salvaging the reputation of their editor is akin to salvaging themselves. ☺

With Emacs, I do a lot of special editing for special files. A big part of the special editing has already converted to Python using Pymacs, and this will ease migrating the special processing to another editor like Vim, which is Python aware and featureful enough to start with. Yet, going from Emacs to Vim leaves a few holes in work habits, that need to be filled with some replacements. Let me list a few of these, in roughly decreasing order of importance.

After a few years of Emacs abstinence (I had to heal out of a very long and deep addiction), I restarted using it, parsimoniously at first, and then more regularly. But it is now one tool among others, and not the tool as it used to be for me. Currently, I use Emacs merely to edit Erlang sources, XML code, to read Usenet News or Gmane, or to achieve big directory hierarchy cleanup.

2.2 Surviving without Gnus

For Gnus, the fact I get much less volume than I used to (like in the time I was maintaining tar, say!) might help me. I got away from Babyl files already to favour traditional Unix mailbox format, that is a great relief. Since I use the email paradigm a great deal, even to save notes within messages to myself which I then file into thematic folders, I will have to find a fairly efficient replacement.

Gnus was really marvellous, nothing can really compare with it. I looked around a bit for a replacement, and the best I came with is Mutt (supplemented with fetchmail, some procmail, Spambayes, and many home-made scripts). Getting Mutt and some of my Python scripts to really cooperate required some doing, and a few kludges, and I still feel very far from having a mail agent offering Python as an extension language. I also saw Python extensible mail agents, but which had some weakness in the underlying mail engine, so far that I could judge. Mutt is not perfect overall, but I'm trying to compromise with it for now, it's still give-and-take.

But at least, I now feel installed in a much more modular design for my work habits, and if I find something that pleases me more than Mutt, I could change this aspect without having to change everything else with it. In my case at least, the big move is now part of the past, the main suffering is behind.

After an irritating journey into Evolution, I finally switched to Thunderbird which, without being perfect, is quite reasonable.

I still use Emacs and Gnus for news reading, however.

2.3 Surviving without Shell mode

Surviving without Shell mode has been suprisingly easy for me. We have plenty of good terminal emulators around. The readline capabilities in popular shells are satisfying enough.

2.4 Surviving without Dired

For a few years, I used Tim Daneliuk's Twander, while slowly taming myself to GNOME and Nautilus, which I usually prefer nowadays. Yet, once in a while, when I really have a big perusal and cleanup to do, and seek operational speed, I still launch Emacs for its Dired mode, which is still unequaled in my experience.

2.5 Surviving without Allout

Allout has been, for quite a long time, my best choice for synoptic editing. I grew quite a lot of notes using it. Getting away of Emacs, I needed a way to access and handle all that material, so I wrote an Allout mode for Vim (using its Python extension). Later, when I found out about the very nice Tomboy tool, I patiently and tediously moved over all my Allout notes to Tomboy. Nowadays, I'm not using Allout much anymore.

2.6 Surviving without PSGML

For SGML, it might require a good amount of work before I'm happy. PSGML is really a wonder under Emacs. (I still have no idea on the Vim capabilities for controlling highlighting, which is somewhat orthogonal to SGML analysis, but especially useful there.)

I wrote something for Emacs called xxml.el, that does the job of combining colour attributes likely beyond the Vim way of doing such things. I may be wrong: my knowledge of Vim is still very fragmentary.

My associate, who is a linguist more than in computer science, usually likes to adopt and stick with my work habits, because he then gets more full support from me. He does not object switching to Vim from Emacs as I did, but only given that Vim has the equivalent of xxml.el first. The fact is that I understand him: I have been unsuccessful so far at enjoying HTML editing support in Vim. I pondered a bit about how I could rewrite xxml.el in Python, for later becoming a Vim extension say, but I guess it would require a fairly good amount of work on the Python side only, and no doubt that I would also need to learn Vim advanced usage much more deeply than I know it currently.

Nowadays, I still launch Emacs for its nXML mode, whenever I need to handle XML code, which occurs relatively often given XML popularity. I do not see much SGML anymore.

2.7 Surviving without ELSE

Using Python as my main day-to-day language much alleviated the need for ELSE, as this language is rather terse, and does not have the same syntactical overhead as other languages. I wrote Pynits to help me with various Python idiosyncrasies of mine, and the few ELSE needs I would have are covered in this tool. At the other end of the language spectrum, there is Java, for which ELSE would be useful indeed. But Java has it own flurry of problems, for which Eclipse offers a much better solution (despite its discouraging complexity and heaviness)..

2.8 Surviving without Calc

Calc, I do not use that often, but whenever I need it, it has a very efficient user API, and a wide, eclectic set of capabilities. Another wonder. To survive without it, I moved towards R mainly for most things, with bits of Maxima for symbolic computations, and a few Python scripts whenever convenient.

3 Getting used to Vim

3.1 Installing it properly

One sure thing is that I wanted Python support in Vim. This meant re-installing it from sources. Here an approximation of my installation recipe::
wget ftp://ftp.vim.org/pub/vim/unix/vim-7.1.tar.bz2 tar xfj vim-7.1.tar.bz2 cd vim71 ./configure --prefix=/usr \ --enable-pythoninterp --with-features=big --enable-multibyte make install
To quickly check that it works, once in the new Vim, try :py print 3 + 5, you should then read 8 in the bottom line. On this topic, file /usr/share/vim/vim71/doc/if_pyth.txt holds the information one needs. Succinct, but likely complete.

3.2 Teaching the fingers

Emacs and Vim key-bindings are pretty different, down to the logical organisation and structure of commands. One thing which has been difficult for me is to learn to shift the right hand one position left on the HJKL keys and unshift it on JKL; depending on what I want to do, this detail often slowed me down initially. I'm not as speedy in Vim as I was in Emacs. The more it goes, the easier it gets. I expect and accept that it will require a few more months before the spinal chord adapts. ☺

I think Vim key-bindings are a bit softer on the hand and fingers than Emacs, because most usual commands use simple lower case letters, others use upper case letters, and only rarely you have to resort to the control key. However, the Escape key is often needed, and a bit remote. This advantage somewhat vanishes when I type French text, as the cf keyboard uses a mix of dead keys (some shifted). Other combinations require a full repositioning of the right hand, Python brackets and braces in particular.

3.3 Using gvim (Vim with GTK)

While I like the standard Vim-in-a-terminal, I prefer gvim whenever possible. One of the advantage is the ability to use the mouse for some quick operations, like positioning the cursor between two characters, moving big distances by dragging scroll bars, changing window dimensions by dragging the bars between windows, wandering in help through clicking on index items, clicking within the fold margin for opening and closing folds, etc.

I'm usually not so fond on the mouse, but strangely (to me at least), gvim is effectively taming me into using it. One thing which I often do is repeating the sequence of using the mouse for fast positioning of the cursor, then using . typed with the left hand. It is surprising how many other vim operations you can quickly do with the left hand alone! ☺

One thing which helped me a lot is that, right from the beginning, I took the time of carefully customising gvim so it uses readable fonts, consistent colours, and such things. As for Python editing, of course, vim customisation is equally available within and outside X.

L'icule gazouilleur

lundi 15 décembre 2003

Thoughts on editors

1 Pondering Emacs

1.1 Introduction