Laurence Tratt's Technical Articles

In Praise of the Imperfect

August 25 2010

I recently read an article on music that, boiled down to its essentials, talked about the lack of a meaningful relationship between technical virtuosity (roughly ‘how well can you play’) and musical quality (roughly ‘how enjoyable is this piece of music’). Sometimes 3 chords and a 3 note melody played slowly and imperfectly will be exactly what is required; sometimes precise fast scales and subtle melodic variations will do the trick. When a song that should be played slowly, sloppily, and minimally is played fast, precisely, and with extraneous notes - or vice versa - the balance between technical ability and musical quality is lost. Remembering this balance is a problem many musicians have. As their technical abilities grow over time, it is common to equate technical difficulty with musical quality. Despite my extremely limited musical abilities (it is difficult to dispute one persons assessment of my playing as ham-fisted), I have on occasion suffered this problem as I have got slightly more proficient: fortunately, since many of my favourite songs are - not to put too fine a point on it - brainless three and a half minute rockers, I generally get brought back down to earth at some point.

Reading this made me realise that a similar balance applies to programming. Sometimes a heavily engineered solution is right for the job; sometimes something much simpler is just the ticket. Let me give an example of each.

First, an example of something heavily engineered. One of my favourite things that I've done is extsmail which is a traditionally constructed, robust Unix daemon for mail delivery. In extsmail, I cared about every possible detail: the program is carefully decomposed into its constituent parts; every error condition is explicitly handled and dealt with appropriately; the documentation is complete; I audit the code regularly; and I've run it through every static analyser I've been able to get my hands on. In short, its 1,600 lines of code are probably the highest quality I've ever personally written. Given its tiny potential audience, you could well argue that it is grossly over-engineered, but it solves a real problem that I had for years. It performs its task - delivering e-mail to a remote machine via SSH - admirably and hasn't crashed in almost 2 years. Anything less than that wouldn't have been right for what I needed - it has to do the job as near to perfect as is possible, every time. Heavy engineering was the right choice here.

Second, an example of something simple. A colleague once dismissed a program I'd written with an unrepeatable 4 letter word, saying that it was poorly organised and unreadable. He was right in one way. The program in question was about 4000 lines of code, in an area of which both of us had previously been entirely ignorant - the fact that I had to learn a new programming language was one of the lesser challenges in the whole thing - and the result was slapped together in varying stages of incomprehension over roughly 3 or 4 man weeks. I pointed out to him two things about the result: first, it showed that what we wanted to do was possible and plausible, when we both had doubts originally; second, that he'd written precisely zero lines of code in the same period. That code, incidentally, is now part of one of the more widely used pieces of software I've been involved with. No user will ever know that it's the product of someone learning on the job, and that, internally, things could have been done much better. For the job it does it works reliably enough and - more importantly - it exists. If I'd taken the high-road, it still wouldn't exist now, and nobody would have been able to benefit from it. Quick and dirty was the right choice.

As my tactless colleague shows, there is a bias in programming against imperfect solutions. I can understand where this bias comes from: the worst of all worlds is an inappropriate imperfect solution. Trying to ensure that this doesn't happen has some odd side-effects. One is a tendency to assume that one language, one tool, one technique should be appropriate for all tasks. Different communities fixate on different things. For the last 10 years, industrial management has often thought Java was the answer; web-sites PHP; and academics strongly typed pure functional languages. In my opinion, they are all wrong today - and, I suspect, they will all be wrong in the future, even as their favourite things change. There is no one size fits all. Different tasks demand different approaches. For example:

• If I need to do a sysadmin job across my servers, I'll use the Unix shell or Python. If it goes wrong, it's not a huge problem. What is important is having something now. Having it in a form that can be easily altered to suit changing needs is also useful.
• If I'm writing a Unix daemon I'll use C. I prefer it not to go wrong, but I can tolerate very occasional failures. If it takes a while to create, I can live with that. Having it in a form that can be tweaked a bit is useful, though I don't expect it to change a great deal in the future.
• If you're writing software for a plane or nuclear reactor, I hope you use the strongest statically typed language you can, every static analyzer available, and as many formal techniques as you can shake a stick at. Failure is not an option, since the result will be loss of human life. Nor is cost - if it takes hundreds of man years of effort, it's probably worth it. Since such software must be precisely specified up-front, there is relatively little need for it to amenable to change.

No one in their right mind would say that the highly practiced virtuosity of a classical orchestra could replace the raw emotion of a simple folk singer - both have their own merits and their place in the musical landscape. So too should we give imperfect, pragmatic software the respect that it is due.

Link to this entry

A Modest Attempt to Help Prevent Unnecessary Static / Dynamic Typing Debates

April 7 2010
Updated: April 8 2010

Several times recently, memories of wasted hours have flooded back to me as I have been an unwilling participant in one of the longest-standing debates in programming languages - statically vs. dynamically typed languages. Frankly, such debates haven't become any less painful since the passing of my teenage years. The problem in such debates remains the same: both sides over-inflate the advantages of their favoured approach while wilfully maintaining their ignorance of the other. Add to this the common tendency of any side in a debate to assume the worst of their opposition, and you have a situation that resembles Northern Europe in WWI: grim, unrelenting trench warfare, with neither side making any real advances.

To make matters worse, the debate in this particular case is asymmetric. For decades now, the computing intelligentsia have been overwhelmingly in favour of static typing - in academia, this bias is almost total. While the intellectual basis of static typing is both well established and frequently promulgated, the dynamic typing community has been far less intellectually confident. Please note, I'm not saying that dynamic typing has a less firm intellectual basis - simply that it is infrequently articulated. Let me give two simple, but different, examples:

• The first is terminological. Dynamically typed languages are frequently dismissed as scripting languages, which is at best an odd classification (languages such as Python have all the modularisation-type features of equivalent statically typed languages) and, at worst, deliberately derogatory. But labels are just labels: there's no point getting too worried about them. What's more frustrating is a lack of understanding of the difference between statically and dynamically typed languages. Many (though, in fairness, not all) static typing advocates confuse dynamic typing with no typing. Dynamic typing doesn't mean that programs have no types - rather the typing discipline is enforced at run-time rather than at compile-time. Compare this with strong and weak typing. In a dynamically, strongly typed language like Python, types have no compile-time effect, but they can not be overruled at run-time (adding an integer to a string causes a run-time type exception); conversely, a statically, weakly typed language such as C enforces types at compile-time, but allows them to be overridden at run-time (to sometimes hilarious, though generally annoying, effect).
• The second is practical. Refactoring is an umbrella term (in my world, at least) for the activity of reworking a program to improve its internal quality. Any program that's been around a while will have had unanticipated changes made to it, and will require some refactoring. Small exploratory tweaks are the hallmark of refactoring in dynamically typed languages; they often temporarily break the system as a whole (so program execution tends to terminate prematurely), but give the programmer confidence that the particular part of the program they are concentrating on is not adversely impacted by the larger change they intend making. Unfortunately, static typing inhibits this type of program refactoring, since even the smallest of tweaks must respect the static typing system. If the whole thing doesn't compile, it can't be run; and if a small part of the change can't be tested on a running system, the programmer will often not have the confidence to spend days (or longer) changing the whole system, only to find that the original idea was a bad one.

To my mind, both statically and dynamically typed languages have a place. If you're building software for a nuclear reactor, I want you to use every tool at your disposal to reduce errors, even if that increases the cost of producing the software by a factor of ten or more - and statically typed languages will undoubtedly catch some errors that might otherwise go undiscovered, so they're the right tool for the job. If you're building a website, I want you to make it as easy as possible for yourself to modify it to reflect rapidly changing requirements - which probably means using a dynamically typed language.

One day I hope that someone will put together a comprehensive and unbiased comparison of the two paradigms since, to the best of my knowledge, nothing currently exists which does the job. Until then, I immodestly offer a pre-print of a modest book chapter I wrote last year (and, if you're really interested, its corresponding BibTeX entry) where I tried to give an introduction to dynamically typed languages. Inevitably this involves a comparison to statically typed languages, defining terminology, and trying to enumerate the relative strengths and weaknesses of each approach. As mentioned in the chapter, I personally dislike the terms statically typed and dynamically typed because they mislead so many people; unfortunately, changing them to something less misleading is a battle I could never win, so I start from the assumption that we're stuck with those terms. In retrospect, the chapter is far from perfect: while I tried to suppress my own biases, I didn't fully succeed; there are many parts which I'd like to expand; and even more parts which I'd like to change. Yet, despite this, I can't help feel that some of the points it makes (even though none of them are particularly original) might have helped avoid some of the more tedious aspects of the static vs. dynamically typed debates that I was forcibly involved in. Until something better comes along, it might fill a useful niche.

Updated (April 8 2010): Fred Blasdel points me at Chris Smith's What To Know Before Debating Type Systems which covers much (though not all) of the same ground as my chapter, although more briefly, and from a statically-typed perspective. You may find it interesting to compare and contrast.

Link to this entry

A Proposal for Error Handling

December 14 2009

The two approaches

1. Error checking is where functions return a code denoting success or, otherwise, an error; callers check the error code and either ignore it, perform a specific action based on that code, or propagate it to their caller. Error checking does not require any support from language, compiler, or VM; it depends entirely on programmers following the convention. Languages which use this idiom include C.
2. Exceptions imply a secondary means of controlling program flow other than function calls and returns. When an exception is raised, this secondary control flow immediately propagates the exception to the functions caller; if a try block exists, the exception can be caught and handled (or rethrown); if no such block exists, the exception continues propagating up the call stack. Exceptions require certain features in the language, and support from both compiler and VM. Languages which use this idiom include Java, Python, and Converge.

Outline of the problem

The problem comes when one is trying to write highly fault tolerant code, by which I mean systems which try to keep on running even when undesirable events or situations arise. I documented my experiences when writing the (tiny) e-mail sending program extsmail earlier. Highly fault tolerant systems need to deal explicitly with almost every error that can occur; with judicious thought, it is amazing how many errors can either be partly or fully recovered from. In extsmail, the only fatal error (i.e. the system immediately exits) is when memory is exhausted. I would guess that about 40% of extsmail's code is to do with error checking and handling - such fault tolerant systems are much more expensive to produce than the alternative. extsmail is written in C, a language which much folklore would suggest is fundamentally ill-suited to the task. The fact that extsmail - and indeed, most of the other C software I use - works and, I hope, works reasonably well, continues to grate against my long-held prejudices; yet I honestly don't think I could write a system which is as fault tolerant in any other comparable language - in particular, in any exception-based language of my acquaintance.

Some readers might interpret the above as being a position based either on ignorance, or a hatred of exceptions. While I generally plead guilty to charges of ignorance, I can provide some evidence that in this rare situation it's not the case. The language that I designed - Converge - is an exception-based language. To the charge of hatred I say that, except when writing highly fault-tolerant systems, I believe exceptions are the best way of dealing with errors.

If I don't hate exceptions, what's the problem? Here's the rub. In theory there is no real problem with exceptions; they're clearly a more pleasing solution to the problem than error checking. The problem comes because they seem to inevitably lead to a certain style of programming that is not suitable for highly fault tolerant code. This style permeates every exception based language and library I have seen. Before I go into more detail as to what the problem is, it's best to take a step back and look at the competing solutions.

Error checking

Let's consider error checking in a C-like environment (I say C-like because I wish to avoid the horror that is is errno; let us also assume that this C-like environment allows functions to return multiple values). A possible example is the following:

int err;
if ((err = write(...)) != 0) {
  switch (err) {
    case EINTR:
      ...
    case EAGAIN:
      ...
    case ...:
      ...
  }
}

In general, it's unusual to explicitly deal with each different error that a function can return. At the other end of extreme, one can choose to simply ignore the error returned by a function:

write(...);

if (write(...) != 0)
  err(1, "Fatal error.\n");

As the above hopefully suggests, error checking code suffers several problems including: verbosity; the ease with which minor typos escape notice; and the difficulty of retrospectively changing APIs (extending the errors that a function returns would, in general, necessitate changing - or at least checking - all callers of that function). However there is one point which, while it might initially seem a problem, turns out to have interesting positive consequences. Since error checking relies entirely on convention, every function that follows this idiom must carefully document what errors it can return; without that, the idiom would be unusable. We'll return to this shortly.

Exceptions

try {
  err = write(...);
}
catch Interrupt_Exception e {
  ...
}
catch Again_Exception e {
  ...
}
catch ... {
  ...
}

try {
  err = write(...);
}
catch IO_Exception e {
  ...
}

Of course, the beauty of exceptions is that they don't need to be explicitly handled. Furthermore, there is much less potential to silently, and accidentally, swallow an error (as can easily happen in error-checking systems; and assuming one doesn't have a brain-dead checked exception system as in Java). A system which uses exceptions is entirely predictable: it will run reliably and tend to fail reliably and quickly. In contrast, a buggy error-checking system will often fall over long after the real error occurred, in a way which makes debugging painful at best.

The problem with exceptions

Ironically it is the ease of exceptions which I believe is their downfall.

1. The first problem is theoretical in nature. Most languages don't include exceptions in their typing system. This means that there is no way of knowing what exceptions a function will throw. As such, this isn't a problem: I'm not known as a big fan of static typing anyway. The problem then becomes a cultural one: the exception based languages with which I am familiar only lightly document what exceptions a function can throw. Even when they document the exceptions, it is rarely clear exactly what circumstances will lead to the exception being raised; and it is generally equally ambiguous as to exactly what the exception being raised means. Compare that with the terse, but invariably precise, descriptions of the C Unix man pages. Each function religiously documents the error codes it will return and what they mean. Clearly this is a cultural problem at heart - there's no real reason why exception based systems have to have such poor documentation. But, I would argue, that because most programs have no need to precisely know what exceptions a function could throw, there is no cultural pressure to document such things. And, as soon as one function is imprecisely documented, any function which calls it is inevitably going to be imprecise in its documentation (even if it doesn't realise it). Poor documentation of exceptions is thus difficult to avoid.
2. The second problem is due to the ease with which exceptions can be thrown. Most functions, if carefully analysed, can potentially throw a vast number of exceptions. This is hardly surprising. Every look-up of an element in a list; every division of an integer; not to mention the fun that polymorphism can bring to the table; all of these things can potentially raise an exception. The culture of most exception based languages is not to see this as problem and thus not to be defensive: exceptions safely catch problems, so little to nothing is done to check in advance that a given action will not raise an exception. In contrast, error-checking languages have to be defensive: if the pre-conditions for an action are not checked in advance, the system is likely to go wrong in unpredictable fashion. Therefore error-checking systems force programmers to explicitly consider every error that a function might have to throw; and most of them are either dealt with explicitly or, at least, documented. In general, functions in error-checking languages return far fewer distinct errors than an exception-based function can raise exceptions.
3. The third problem is related to the second. Philosophically speaking, I believe that there are two types of exceptions:
   1. Programmer cock-ups (e.g. pop'ing an empty list; division by zero).
   2. The inability of an external thing to fulfil its contract (e.g. writing to a network socket which has been closed by the other end).
   Programmer cock-ups are frequent (at least, they are in the programs I write), and mean that one or more assumptions underlying the program in question have been violated. These to me are very bad things: in general, I wish the program to be immediately terminated so that the program doesn't limp on to an even worse death later, and so that I can easily pinpoint the underlying cause. Furthermore, by definition, I as the programmer don't mean to make these cock-ups, so I don't put try ... catch statements in to handle them. In other words, I think that exceptions cover programmer cock-ups very well. In contrast, the inability of an external thing to fulfil its contract is different. We all know that network sockets can close at any time; so whenever reading from a network socket, it is reasonable to expect to check for read errors. In fault tolerant systems, one is likely to try and deal with all such errors explicitly.
4. The fourth problem relates to the try { ... } catch { ... } construct. Any code in the try block which raises an exception will be dealt with appropriately. The difficulty here is the frequency with which too much code is put into this try block. The classic case I see looks as follows:

   try {
     f(g(...));
   }
   catch (Exception) {
     ...
   }
   

   The intention here is that if f throws an exception, the system still soldiers on. The problem is that the ease with which code can be put in the try block means that f's parameters' evaluations are also included; and they are rarely meant to be. In other words, any exceptions which g raises will be silently - and in this case unintentionally - swallowed. In an error checking approach, the call to g would have to be explicitly checked for errors, largely preventing this incorrect idiom.

For average programs, none of these points is a huge problem; in fact, they arguably make a programmers life easier. But for fault-tolerant systems all are genuine issues: it is impossible to write a fault tolerant system if you are unsure what errors you need to deal with; if the number of errors you are required to deal with becomes too great, the sheer magnitude of task will overwhelm you; if you're unclear what errors are the result of your mistakes and which aren't, debugging becomes a philosophical minefield; and if you tend to be too crude in the granularity with which you check errors, mistakes will happen.

A proposal

What both error-checking and exceptions provide is a way of returning two things from a function: an error code / exception, and the results of the function. The beginning part of my proposal is for an operator which pulls these two things apart in one go. For arguments sake, let's use back-slash \ as this operator. On the left hand side of \ are the functions normal return values; on the right-hand side its error code. Error codes are integers (reflecting one of my prejudices; but the type of the error code is not hugely important), with 0 indicating success. We can then thus write code such as:

bytes_written \ err = write(...);
if (err == EINTR)
  ...

Although it's not directly related, an obvious problem with error checking is the difficulty with extending an API; any errors codes not explicitly checked for will be silently swallowed. To some extent this is a deliberate design decision: it should be culturally difficult (not impossible; but definitely difficult) to extend the range of errors that a function returns (exceptions provide no encouragement to follow this rule whatsoever). However, I also assume that any language with this feature in will have an equivalent of Converge's ndif statement which raises an exception if none of its branches match.

So what has all this bought us? Well, at first glance, we've just got an error-checking system which formalises the way in which errors are returned. Indeed, this is a large part of the story. We now have a simple, uniform way of performing the error-checking approach. But, if this was all we'd got, we wouldn't have advanced much beyond C. By formalising how errors are returned, we can also define what happens if the error code is not explicitly checked for. In other words, what does the following code do?

bytes_written = write(...);

Finally, I expect exceptions to be maintained almost exactly as they exist in current languages, including the throws / raise construct (which can also be called with an error code, so that if a calling function doesn't know what to do with a particular error code, it can be turned into an exception by that caller). The only difference I would make is that I would forbid exception hierarchies. This is because in this proposal, it's not really expected that exceptions will ever be caught (with one caveat): they're really just a debugging aid and, as such, exception hierarchies only muddy the waters. Because there are fault-tolerant systems such as network servers - where staying alive is the number one priority - I would also maintain try ... catch, though I would expect it to be little used other than to surround a top-level call, with the catch block restarting the server (or a similar recovery mechanism) in the event of an exception.

Intentions

The intention of the proposal is twofold:

1. Programmer cock-ups are handled as normal exceptions, don't require any extra effort on the part of the user, and lead to immediate and easy-to-debug program failure.
2. It allows programs which want to do explicit error-checking to do so, and to do so in a framework in which error-checking is culturally practical; in other words, it doesn't suffer from the practical, mostly cultural, problems noted with exceptions.

Conclusion

My thanks to Martin Berger for commenting on a draft of this article. Any errors and infelicities are my own, as are all the bad ideas.

Link to this entry

The Missing Level of Abstraction?

September 15 2009

Levels of abstractions

1. That X is at a higher level of abstraction than Y (alternatively one could say that X is more abstract than Y).
2. That X does not introduce anything fundamentally new over Y; indeed, it may well remove fundamental things, or present them in an easier fashion.
3. Assuming that one does not need any of the fundamental things in Y that may have been lost in the abstraction X, then X is in some sense easier to use than Y.

Abstractions are typically relative things. From the statements X is at a higher level of abstraction than Y and Y is at a higher level of abstraction than Z we can deduce that X is at a higher level of abstraction than Z, but we can not say that Z is the lowest level of abstraction of all - it may well be at a higher level of abstraction than some other thing of which we are currently unaware.

Regrettably, my professional life has taught me that the notion of relative levels of abstraction is not universally shared, probably because it is harder to understand than the notion of absolute levels of abstraction. This fallacy is commonplace; I first encountered it in the MDA world where the terms PIM (Platform Independent Model) and PSM (Platform Specific Model) are widely, and incorrectly, abused to imply absolute levels of abstraction. Similarly one often hears talk of high-level versus low-level languages as if they were absolutes when they clearly are not. Provided one bears in mind that these notions are always relative and that omitting the relative qualification is a simple brevity aid, then many things make a good deal more sense.

Objective-C

Objective-C, alas, I find a more difficult language to like. In small part this is because it generally implies (and did in my case) working under OS X, an operating system beloved of those who do not truly love computers - its idiot-proof lack of customisation and innumerable small bugs came close to breaking me. Objective-C grafts higher-level Smalltalk-like features onto a lower-level C base. The resulting language is not only more verbose than either of its parents - I particularly enjoyed needing to type class attributes into three different places - but distorts many of their defining features. For example, Smalltalk's collection hierarchy (i.e. lists, sets etc.) is a thing of genuine beauty and utility, allowing a syntactically minimalist language to succinctly express complex constraints; since Objective-C doesn't allow blocks (small anonymous functions), not to mention that its collection hierarchy is not as well designed, this is impossible. Another aspect is that C is a statically (if weakly) typed language, catching errors at compile-time that would otherwise cause hard to debug segfaults; however Objective-C's message sending is, like Smalltalk, dynamically typed. In Smalltalk this is not an issue - type errors are simply triggered, lead to predictable backtraces, and are easily fixed. While some dynamic typing errors in Objective-C lead to similar backtraces, others fall foul of C's free-wheeling approach to memory management and cause horrible run-time results, leading to little more than a splat sound; debugging those is a chore.

Memory management

As I understand things, modern Objective-C under OS X uses garbage collection. On some other platforms, such as the one I was targeting, there is no garbage collection. I initially assumed that in such cases Objective-C would revert to a standard C-style system of malloc and free - to my surprise, it does not. Instead, Objective-C uses an unusual system of sometimes-implicit, sometimes-explicit memory management. For someone used to traditional high-level garbage collection or low-level C-style memory management, it's rather hard to get your head around: there don't seem to be hard and fast rules; some of the documentation is ambiguous about the users responsibilities; and there is a good deal of obfuscating cruft which I assume has gathered over time. However, the basic idea is as follows. If a user explicitly allocates memory (via an alloc message), he is responsible for freeing it. If another function allocates memory, it will (or, more accurately, should) be put into a memory pool; when the memory pool is freed, the objects in it are freed too. New memory pools can be created, and they are kept in a stack so objects are put into the most recent memory pool.

Memory pools are an attempt to allow implicit memory allocation (something which is, for good reason, deeply frowned on in traditional C libraries) with implicit memory deallocation. In practice, they resemble a poor-mans attempt at garbage collection or, perhaps, garbage collection as designed by someone who hadn't fully grasped the idea. For the life of me, I can not fully prevent memory leaks in a medium-sized Objective-C system; ironically, I have found it much easier to debug memory leaks in pure C applications. Different parts of code can retain and release objects, which I assumed was a synonym for reference counting, but in reality doesn't always seem to quite work: what, according to the documentation, are valid combinations of calls can cause crashes or leaks. Was this due to bugs in my code, someone else's, or a flaw in the memory management design? Who knows - at some point, I got the leaks down to the level of a few bytes a minute and gave up.

The middle-level of abstraction

The interesting thing to me is not that Objective-C is the middle-level of abstraction between C and Smalltalk as such - after all, given two points on the abstraction scale, there's a high chance that there are other levels in-between - but that it appears to me to have been deliberately designed to slot into this place. This made me wonder about two things. First, why are there not more systems designed to slot between two existing levels of abstractions? Second, why have I never heard the term middle-level of abstraction before?

A couple of answers immediately suggested themselves to me; no doubt others can think of better ones. Perhaps systems designed to slot between two existing levels are more likely than not (as Objective-C) to be worse than the things either side of it? Alternatively, perhaps under normal circumstances we only need to think of one higher level of abstraction thing and one lower-level thing in order to work satisfactorily, and what comes in the middle is generally irrelevant? Whatever the reason may be, I suspect we're unlikely to ever hear many uses of the term middle-level of abstraction - it may remain forever the missing level of abstraction.

Link to this entry

Good Programmers are Good Sysadmins are Good Programmers

March 20 2009

Of course, my life thus far has not just involved my surprise at other peoples habits; on occasions (less rare than my ego might have preferred), other people have been surprised at mine. Recently a non-computing friend saw my main computer workspace - a Unix setup with 4 xterms displayed - and asked, jokingly, if I was plotting to take over the world (I blame the media for this particular image). I'm so used to my own setup that I no longer think of it as odd but, when suitably prompted, I can see why other people might think so. In comparison to a Windows or Mac machine, full of little visual goodies, and perhaps with only a web browser or word processor loaded, a number of tiny xterms filled with half-executed commands does look odd.

It's not really surprising that a non-computing person would find my setup odd - after all, I spend a lot of time on computers, so it's to be expected that some things that I have come to find natural scare casual users. What has surprised, and continues to surprise me, is how many computing people I come across find my setup odd - sufficiently odd that it attracts comment. Some people are baffled as to why my systems are as they are, some are curious as to how it works, and some people sneer at the way I do things. There is no good answer to the sneer, nor any great reason to answer that person (although, possibly due to a deep character flaw, I find the sneer rather amusing). However the how and why are interesting questions, which raise interesting points, and are more closely integrated than they may first appear.

Here's the broad setup I use. I have a desktop machine (because it's fast and comfortable), a laptop (which I use only when out and about, because laptops are slow and ergonomically disastrous), a main server (where you're probably reading this from), and a backup server (for the next time the water company cuts through the cable powering the main server; in an attempt to salve my environmental qualms, the backup server is a very low power device that also serves some domestic purposes). Though various people have some sort of access to the servers, I personally administer all 4 machines. This raises two immediate problems: how to keep the administration overhead to a minimum; and how to keep files synchronised between each machine.

The answer to the administration overhead question is, for me, simple: use the same operating system for all machines. That way, the lessons learned on one machine apply trivially to the others (and, when things go belly up, machines can relatively easily stand in for one another). As someone who (through a quirk of geography and history) never passed through the DOS / Windows world, I eventually gravitated towards Unix operating systems and, after a brief flirtation with Linux, I've exclusively used OpenBSD for nearly 10 years. This immediately scares most people off, or confirms their worst suspicions of me - to give you an idea of the popularity of this OS, at the time of writing, I've met precisely 1 OpenBSD user in real life. Why did I chose OpenBSD? Simple: it's simple. OpenBSD does very little by default, and what it does, it does well, consistently, with minimal configuration, good documentation, and is easily administered remotely. I can have a blank box turned into a complete OpenBSD install with everything I want, setup how I need, in a couple of hours (most of which is automatic downloading of stuff, and doesn't require my presence). I keep an open mind about OS replacements, but so far none of them appears an improvement, or even a sideways step. Of course, using what is often dismissively called a server operating system does involve some compromises, although less than you might think - the increasing diversity of real-world OS's (thanks indirectly, I think, to OS X) has meant that running a minority platform involves fewer compromises than it did 5 or 6 years back.

The file synchronization problem is a little more subtle, but arguably more important. I outlined my mechanism for this a while back and while it's changed in detail quite a bit, in spirit it's still the same: I use a version control system (git these days) for my important files and Unison for large files that I can recreate via other mechanisms.

A corollary of using a decent Unix, and synchronizing files automatically, is that virtually everything is configured by simple text files, so to a large extent my configuration also propagates across machines. I have also tried over the years to accept, whenever possible, the default configuration on a machine. The reason for this is simple: the less I feel the need to change, the easier it is to move between machines (and different OS's). Of course, there's a limit to how far I'm prepared to accept someone else's choices, and so I do change a reasonable number of settings; but, compared to most people I know, I change relatively little.

As well as trying to use the default configuration as far as is practicable, I also try to maximise my use of tools supplied with the OS and, failing that, to use the simplest tool that does the task I require. A decent Unix comes with a wide variety of little tools, most of them neglected by most users; it continues to amaze me as to how many tasks can be expressed in terms of these little tools. Using tools that are standard across many different machines and OS's again lowers the barriers to moving between machines. It also generally implies a greater consistency of user experience, since tools from the same providers tend to be more consistent; some providers (particularly commercial) seem to delight in perverse user interface choices, which means that installing and learning new tools can be an uncomfortable experience. I also try, whenever possible, to use command-line tools not because - despite what some of my friends think - I like being obscure (my formative years were spent on RISC OS, where the GUI was King and the default assumption was that the command-line was for the mentally unsound) but because it's easier to control command-line tools and maintain consistency across platforms.

Most of my time on a computer is spent either doing e-mail (I use mutt because it's the least annoying mail client I've yet found, despite its obvious limitations), web browsing, programming, or writing. The latter two tasks are the most interesting. If for you, as for me, an average day is a wild trip of programming in several languages, and working on several papers of dubious literary merit, then you'll know how much time one can spend editing text; I often have 20 or 30 files open for editing. The sad truth of the matter is that, as far as I can tell, the modern computing world does not contain a single decent text editor (whereas RISC OS, which I mentioned earlier, had at least 2 excellent text editors). Most text editors are either arcane (e.g. vi and emacs) or bloated (e.g. Eclipse). Since I am not clever enough for the former, and far too impatient to wait for the latter to load (I had a massive shock 4 or 5 years back when, on a powerful machine, I found to my horror that if I typed at full speed in a well known IDE, there was a noticeable lag in text appearing on screen), I use a half-way option, NEdit. NEdit has many limitations and flaws, but it's simple, loads almost immediately, and its syntax highlighting is just about powerful enough to satisfy me.

Let's return to the 4 xterms I mentioned at the beginning of the article, which scared my non-computing friend - it's both worse and better than it seems. I setup KDE so that it has 12 virtual desktops (one of the main reasons I used KDE in the early days was because it binds sensible keys to virtual desktop selection by default), of which I typically use 7 to 8 at any given point. The first is my main work area; one of the xterms has mutt permanently loaded (I occasionally load other mutts in different xterms to simultaneously read multiple folders; an advantage of using a simple tool), one is mostly used for downloading e-mail, and the other two are for random commands and ssh sessions. Desktop 2 is for web browsing and desktop 3 for web page editing. Desktop 4 is my calendar. Desktops 5 and 6 are for programming. Desktops 7 and 8 are for paper writing, with 9 and 10 being used for secondary paper writing. The remaining desktops are spares. This may seem complex or unduly pernickety. The answer to both points is essentially the same: I evolved this setup organically over time so it seems natural, to me at least. Because of virtual desktops, I need only a single 19" monitor, although these days it's a struggle to find a sensibly sized monitor. Unfortunately, computer people are generally gadget people, and gadget people are easily fooled by bigger, faster, better, so monitors these days have largely useless resolutions. Wide-screen monitors, for example, are (I assume) good for watching films, but they're fairly useless for text editing, where screen height is more important than width. Furthermore, the pointless fixation on resolution means that many fixed size things (some fonts, icons etc.) appear tiny, so I increasingly see people having to put their nose virtually to their screen to read things. 1280x1024 works for me and, until someone doubles the resolution without increasing the screen size (since then one can imagine that old apps could be transparently run with two physical pixels for every logical pixel they perceive, while new apps will have access to the genuine high resolution, making everything a bit smoother and sharper), I will try to resist the siren call of the resolution junkies.

In the above I've tried to give a brief outline of how I use computers - a modern reader may well detect a certain Luddite tendency in some of the choices, but hopefully I've also provided some small justification for each of the detailed choices. Of course, none of the above is really a high-level why. Why go to all this effort? Why these particular choices? Although I didn't explicitly think of it this way when I first started down the path that led me to my current mode of operation, there is a solid reason behind it. When I look at the really good programmers I've come across (directly and indirectly), then, with only one exception I can really think of, they seem to share one thing in common: they're also good sysadmins. Their machine(s) are in good order, simple, with the right hardware for the job, and ready for the task at hand; and they can whip a new machine into shape quickly. When the muse strikes, there is little to get in the way of good work: they know their machines inside out, the tools inside out, all their files are easily available, everything used frequently loads quickly, and they can flick rapidly between the sub-tasks that constitute a larger job. Similarly, the best sysadmins I've seen are also good programmers. I don't think it's possible to understand a modern OS without being a decent programmer - more importantly, it's certainly not possible to tame and control an OS in the desired way without programming being involved. There's a symbiosis between these two activities that seems to me undeniable; being really good in one requires being at least fairly good in the other.

So, in conclusion, my computer setup is an attempt to emulate, in my own small way, the best habits I've been able to pick up from those more able than myself. It's a continual work in progress, but it does the trick for me.

Link to this entry