Laurence Tratt's Technical Articles

A Proposal for Error Handling

December 14 2009

A while ago I wrote about my experience of writing extsmail, and how surprised I was that highly reliable and fault tolerant programs could be written in C. In large part I attributed this to the lack of exceptions in C. In this article, I expand upon this point, consider some of the practical issues with exceptions based language, and present a candidate language design proposal that might partly mitigate the problem. I don't promise that this is a good design; but it does present some of the issues in a different way than I've previously seen and if it encourages a debate on this issue, that might be use enough.

The two approaches

There are two main approaches to trapping and propagating errors which, to avoid possible ambiguity, I define as follows:

Error checking is where functions return a code denoting success or, otherwise, an error; callers check the error code and either ignore it, perform a specific action based on that code, or propagate it to their caller. Error checking does not require any support from language, compiler, or VM; it depends entirely on programmers following the convention. Languages which use this idiom include C.
Exceptions imply a secondary means of controlling program flow other than function calls and returns. When an exception is raised, this secondary control flow immediately propagates the exception to the functions caller; if a try block exists, the exception can be caught and handled (or rethrown); if no such block exists, the exception continues propagating up the call stack. Exceptions require certain features in the language, and support from both compiler and VM. Languages which use this idiom include Java, Python, and Converge.

Outline of the problem

The fundamental problem as I see it can be concisely expressed: exceptions allow predictable systems to be written with much less code than with error checking; but exceptions make writing highly fault tolerant code difficult. The former point is, I'm fairly sure, uncontentious; with exceptions, one needn't explicitly check and propagate most errors, negating the need for vast wodges of code. For many systems, this is fine - indeed, it is desirable. In general, most of the small and medium sized systems I write have little to no exception checking; if something odd happens (e.g. a file is missing) I prefer such systems to immediately exit rather than limp on to an unpredictable end.

The problem comes when one is trying to write highly fault tolerant code, by which I mean systems which try to keep on running even when undesirable events or situations arise. I documented my experiences when writing the (tiny) e-mail sending program extsmail earlier. Highly fault tolerant systems need to deal explicitly with almost every error that can occur; with judicious thought, it is amazing how many errors can either be partly or fully recovered from. In extsmail, the only fatal error (i.e. the system immediately exits) is when memory is exhausted. I would guess that about 40% of extsmail's code is to do with error checking and handling - such fault tolerant systems are much more expensive to produce than the alternative. extsmail is written in C, a language which much folklore would suggest is fundamentally ill-suited to the task. The fact that extsmail - and indeed, most of the other C software I use - works and, I hope, works reasonably well, continues to grate against my long-held prejudices; yet I honestly don't think I could write a system which is as fault tolerant in any other comparable language - in particular, in any exception-based language of my acquaintance.

Some readers might interpret the above as being a position based either on ignorance, or a hatred of exceptions. While I generally plead guilty to charges of ignorance, I can provide some evidence that in this rare situation it's not the case. The language that I designed - Converge - is an exception-based language. To the charge of hatred I say that, except when writing highly fault-tolerant systems, I believe exceptions are the best way of dealing with errors.

If I don't hate exceptions, what's the problem? Here's the rub. In theory there is no real problem with exceptions; they're clearly a more pleasing solution to the problem than error checking. The problem comes because they seem to inevitably lead to a certain style of programming that is not suitable for highly fault tolerant code. This style permeates every exception based language and library I have seen. Before I go into more detail as to what the problem is, it's best to take a step back and look at the competing solutions.

Error checking

Let's consider error checking in a C-like environment (I say C-like because I wish to avoid the horror that is is errno; let us also assume that this C-like environment allows functions to return multiple values). A possible example is the following:

int err;
if ((err = write(...)) != 0) {
  switch (err) {
    case EINTR:
      ...
    case EAGAIN:
      ...
    case ...:
      ...
  }
}

In other words, a function write is called and, if it returns an error, all the possible errors that the function can return are explicitly dealt with. There are three important things implicit in the above. First, functions uniformly denote success or error (most Unix C functions denote success by returning 0; values other than 0 indicate failure). Second, errors are simple integer values. This means that they can easily be stored and compared against pre-defined error codes. Third, functions carefully document which errors they can return so that the caller need only handle those errors.

In general, it's unusual to explicitly deal with each different error that a function can return. At the other end of extreme, one can choose to simply ignore the error returned by a function:

write(...);

which is equivalent to the exception-based code try { write } catch (Exception) { } in, say, Java - although noticeably terser. In practice, one will also see many instances of the following idiom:

if (write(...) != 0)
  err(1, "Fatal error.\n");

This is a rough approximation of the concept in exception-based systems of exiting the whole program if an exception is not caught at some level. This particular idiom in error-checking systems is a pain in the rear end: it's tedious and verbose to write; and if one has several points that share the error message Fatal error then it may not be possible to easily distinguish which one of them triggered.

As the above hopefully suggests, error checking code suffers several problems including: verbosity; the ease with which minor typos escape notice; and the difficulty of retrospectively changing APIs (extending the errors that a function returns would, in general, necessitate changing - or at least checking - all callers of that function). However there is one point which, while it might initially seem a problem, turns out to have interesting positive consequences. Since error checking relies entirely on convention, every function that follows this idiom must carefully document what errors it can return; without that, the idiom would be unusable. We'll return to this shortly.

Exceptions

Most readers are probably familiar with exception based systems as the majority of modern languages utilise them in one form or another. It is this familiarity which I suspect numbs us to one of the major practical problems with exceptions. Let's recast the initial error checking example into exceptions:

try {
  err = write(...);
}
catch Interrupt_Exception e {
  ...
}
catch Again_Exception e {
  ...
}
catch ... {
  ...
}

As this suggests, it's possible to almost exactly emulate the error checking approach in an exception based language. However, one will almost never see the above idiom in an exception based language (I don't think I've ever seen it). Exceptions are typically organised into a hierarchy so one is much more likely to see:

try {
  err = write(...);
}
catch IO_Exception e {
  ...
}

Oddly enough, this organisation of exceptions into hierarchies, while convenient, is not something I'm keen on for fault tolerant systems. The reason is that, by abstracting away from the specific error that occurred, it tends to give a false sense of security that one is dealing with a given exception in the right way. For example, most systems have a multitude of IO related exceptions, yet it is rare for anyone to deal with anything other than the top-level IO exception class; in a truly fault tolerant system many of those sub-exceptions are likely to be best dealt with differently than others.

Of course, the beauty of exceptions is that they don't need to be explicitly handled. Furthermore, there is much less potential to silently, and accidentally, swallow an error (as can easily happen in error-checking systems; and assuming one doesn't have a brain-dead checked exception system as in Java). A system which uses exceptions is entirely predictable: it will run reliably and tend to fail reliably and quickly. In contrast, a buggy error-checking system will often fall over long after the real error occurred, in a way which makes debugging painful at best.

The problem with exceptions

Ironically it is the ease of exceptions which I believe is their downfall.

The first problem is theoretical in nature. Most languages don't include exceptions in their typing system. This means that there is no way of knowing what exceptions a function will throw. As such, this isn't a problem: I'm not known as a big fan of static typing anyway. The problem then becomes a cultural one: the exception based languages with which I am familiar only lightly document what exceptions a function can throw. Even when they document the exceptions, it is rarely clear exactly what circumstances will lead to the exception being raised; and it is generally equally ambiguous as to exactly what the exception being raised means. Compare that with the terse, but invariably precise, descriptions of the C Unix man pages. Each function religiously documents the error codes it will return and what they mean. Clearly this is a cultural problem at heart - there's no real reason why exception based systems have to have such poor documentation. But, I would argue, that because most programs have no need to precisely know what exceptions a function could throw, there is no cultural pressure to document such things. And, as soon as one function is imprecisely documented, any function which calls it is inevitably going to be imprecise in its documentation (even if it doesn't realise it). Poor documentation of exceptions is thus difficult to avoid.
The second problem is due to the ease with which exceptions can be thrown. Most functions, if carefully analysed, can potentially throw a vast number of exceptions. This is hardly surprising. Every look-up of an element in a list; every division of an integer; not to mention the fun that polymorphism can bring to the table; all of these things can potentially raise an exception. The culture of most exception based languages is not to see this as problem and thus not to be defensive: exceptions safely catch problems, so little to nothing is done to check in advance that a given action will not raise an exception. In contrast, error-checking languages have to be defensive: if the pre-conditions for an action are not checked in advance, the system is likely to go wrong in unpredictable fashion. Therefore error-checking systems force programmers to explicitly consider every error that a function might have to throw; and most of them are either dealt with explicitly or, at least, documented. In general, functions in error-checking languages return far fewer distinct errors than an exception-based function can raise exceptions.
The third problem is related to the second. Philosophically speaking, I believe that there are two types of exceptions:
1. Programmer cock-ups (e.g. pop'ing an empty list; division by zero).
2. The inability of an external thing to fulfil its contract (e.g. writing to a network socket which has been closed by the other end).
Programmer cock-ups are frequent (at least, they are in the programs I write), and mean that one or more assumptions underlying the program in question have been violated. These to me are very bad things: in general, I wish the program to be immediately terminated so that the program doesn't limp on to an even worse death later, and so that I can easily pinpoint the underlying cause. Furthermore, by definition, I as the programmer don't mean to make these cock-ups, so I don't put try ... catch statements in to handle them. In other words, I think that exceptions cover programmer cock-ups very well. In contrast, the inability of an external thing to fulfil its contract is different. We all know that network sockets can close at any time; so whenever reading from a network socket, it is reasonable to expect to check for read errors. In fault tolerant systems, one is likely to try and deal with all such errors explicitly.
The fourth problem relates to the try { ... } catch { ... } construct. Any code in the try block which raises an exception will be dealt with appropriately. The difficulty here is the frequency with which too much code is put into this try block. The classic case I see looks as follows:
```
try {
  f(g(...));
}
catch (Exception) {
  ...
}
```
The intention here is that if f throws an exception, the system still soldiers on. The problem is that the ease with which code can be put in the try block means that f's parameters' evaluations are also included; and they are rarely meant to be. In other words, any exceptions which g raises will be silently - and in this case unintentionally - swallowed. In an error checking approach, the call to g would have to be explicitly checked for errors, largely preventing this incorrect idiom.

For average programs, none of these points is a huge problem; in fact, they arguably make a programmers life easier. But for fault-tolerant systems all are genuine issues: it is impossible to write a fault tolerant system if you are unsure what errors you need to deal with; if the number of errors you are required to deal with becomes too great, the sheer magnitude of task will overwhelm you; if you're unclear what errors are the result of your mistakes and which aren't, debugging becomes a philosophical minefield; and if you tend to be too crude in the granularity with which you check errors, mistakes will happen.

A proposal

At a high-level, what I believe might be an improvement on the current situation is a feature which, to some extent, merges the better parts of error-checking and exceptions. Let us therefore consider what the best parts of each approach are. Error checking forces programmers to be considerate both of the errors that they have to deal with, and the quantity of errors they return to the caller. Exceptions make code far smaller and are particularly good at dealing with programmer or cock-ups and very unusual situations (e.g. out of memory errors) which virtually no-one wants to deal with explicitly.

What both error-checking and exceptions provide is a way of returning two things from a function: an error code / exception, and the results of the function. The beginning part of my proposal is for an operator which pulls these two things apart in one go. For arguments sake, let's use back-slash \ as this operator. On the left hand side of \ are the functions normal return values; on the right-hand side its error code. Error codes are integers (reflecting one of my prejudices; but the type of the error code is not hugely important), with 0 indicating success. We can then thus write code such as:

bytes_written \ err = write(...);
if (err == EINTR)
  ...

On the other side of the coin, we need a way for functions to return errors (or not). Mirroring the above syntax, return x returns x as per normal and sets the returned error code as 0 (or whatever the no error value is). return x \ e returns x and sets the returned error code to e (note that it would be valid, though syntactically redundant to explicitly return the no error value).

Although it's not directly related, an obvious problem with error checking is the difficulty with extending an API; any errors codes not explicitly checked for will be silently swallowed. To some extent this is a deliberate design decision: it should be culturally difficult (not impossible; but definitely difficult) to extend the range of errors that a function returns (exceptions provide no encouragement to follow this rule whatsoever). However, I also assume that any language with this feature in will have an equivalent of Converge's ndif statement which raises an exception if none of its branches match.

So what has all this bought us? Well, at first glance, we've just got an error-checking system which formalises the way in which errors are returned. Indeed, this is a large part of the story. We now have a simple, uniform way of performing the error-checking approach. But, if this was all we'd got, we wouldn't have advanced much beyond C. By formalising how errors are returned, we can also define what happens if the error code is not explicitly checked for. In other words, what does the following code do?

bytes_written = write(...);

Answer: if write raises an error and it is not dealt with, the error turns from an integer error code into a standard exception. In general, I would not expect such exceptions to be caught; they would terminate the program, leaving behind a simple to debug line-by-line backtrace.

Finally, I expect exceptions to be maintained almost exactly as they exist in current languages, including the throws / raise construct (which can also be called with an error code, so that if a calling function doesn't know what to do with a particular error code, it can be turned into an exception by that caller). The only difference I would make is that I would forbid exception hierarchies. This is because in this proposal, it's not really expected that exceptions will ever be caught (with one caveat): they're really just a debugging aid and, as such, exception hierarchies only muddy the waters. Because there are fault-tolerant systems such as network servers - where staying alive is the number one priority - I would also maintain try ... catch, though I would expect it to be little used other than to surround a top-level call, with the catch block restarting the server (or a similar recovery mechanism) in the event of an exception.

Intentions

The intention of the proposal is twofold:

Programmer cock-ups are handled as normal exceptions, don't require any extra effort on the part of the user, and lead to immediate and easy-to-debug program failure.
It allows programs which want to do explicit error-checking to do so, and to do so in a framework in which error-checking is culturally practical; in other words, it doesn't suffer from the practical, mostly cultural, problems noted with exceptions.

What's worth noting is that use of the error-checking facility is entirely optional: if one doesn't use the \ construct, what's left is a standard (if slightly simplified) exception-based system. In other words, the error-checking approach is a bonus which doesn't interfere with normal exception-based practices; it degrades gracefully into using standard exceptions.

Conclusion

In one way, what this article has done is to note problems with the practice of exceptions, and then suggest a partial return to the stone-age of explicit error checking. In that sense, one counter-argument is that I am aiming to throw the baby out with the bath water; and there is merit in that argument. There are also a couple of obvious questions which it raises. First, has this scheme been proposed elsewhere? Not that I know of, but there are a vast amount of relatively obscure languages lurking around, so it's not impossible. Second, would this work in a practical language? I don't know - this is a classic paper design which may well not survive its first pass through a compiler. One day I hope to find out.

My thanks to Martin Berger for commenting on a draft of this article. Any errors and infelicities are my own, as are all the bad ideas.

Link to this entry

The Missing Level of Abstraction?

September 15 2009

Levels of abstractions

I feel fairly confident in stating that everyone who is familiar with the details of computing will have encountered the phrases high level of abstraction and low level of abstraction - probably rather often. Abstraction is one of those words which is used rather more frequently than it is considered. In short, an X that is an abstraction of Y implies three things in our context:

That X is at a higher level of abstraction than Y (alternatively one could say that X is more abstract than Y).
That X does not introduce anything fundamentally new over Y; indeed, it may well remove fundamental things, or present them in an easier fashion.
Assuming that one does not need any of the fundamental things in Y that may have been lost in the abstraction X, then X is in some sense easier to use than Y.

Abstractions abound in computing, as they do in life in general. To take a simple example, the first computers were programmed in machine code (zeros and ones); the second generation, in assembly code which easily translated into zeros and ones, but provided an easier syntax for humans; the third generation and beyond, in languages such as C whose translation into machine code became increasingly complex. Though they are often nebulous and, indeed, often rather hard to spot and understand, abstractions are what make modern computing possible; life without them would be like trying to walk from Lands End to John o' Groats with ones eyes pointed downwards and half an inch above the ground.

Abstractions are typically relative things. From the statements X is at a higher level of abstraction than Y and Y is at a higher level of abstraction than Z we can deduce that X is at a higher level of abstraction than Z, but we can not say that Z is the lowest level of abstraction of all - it may well be at a higher level of abstraction than some other thing of which we are currently unaware.

Regrettably, my professional life has taught me that the notion of relative levels of abstraction is not universally shared, probably because it is harder to understand than the notion of absolute levels of abstraction. This fallacy is commonplace; I first encountered it in the MDA world where the terms PIM (Platform Independent Model) and PSM (Platform Specific Model) are widely, and incorrectly, abused to imply absolute levels of abstraction. Similarly one often hears talk of high-level versus low-level languages as if they were absolutes when they clearly are not. Provided one bears in mind that these notions are always relative and that omitting the relative qualification is a simple brevity aid, then many things make a good deal more sense.

Objective-C

I have recently been doing some programming in Objective-C for reasons that most people can probably easily guess. Learning a new language is rarely a wasted opportunity, and this has certainly been an interesting experience. As I have noted before, I have a definite soft spot for C as a language. If it wasn't for its brain-dead approach to arrays (which don't know their size, and therefore can't automatically resize themselves, bloating code and causing untold bugs) and, to a lesser extent, the baroque system of standard types that has accreted during decades of porting to different platforms (and consequent misunderstandings), I would not have a bad word to say about it. This praise is of course conditional on the fact that C has a particular niche: it is commonly called a low-level language, because it is little more than a slim layer above assembly code.

Objective-C, alas, I find a more difficult language to like. In small part this is because it generally implies (and did in my case) working under OS X, an operating system beloved of those who do not truly love computers - its idiot-proof lack of customisation and innumerable small bugs came close to breaking me. Objective-C grafts higher-level Smalltalk-like features onto a lower-level C base. The resulting language is not only more verbose than either of its parents - I particularly enjoyed needing to type class attributes into three different places - but distorts many of their defining features. For example, Smalltalk's collection hierarchy (i.e. lists, sets etc.) is a thing of genuine beauty and utility, allowing a syntactically minimalist language to succinctly express complex constraints; since Objective-C doesn't allow blocks (small anonymous functions), not to mention that its collection hierarchy is not as well designed, this is impossible. Another aspect is that C is a statically (if weakly) typed language, catching errors at compile-time that would otherwise cause hard to debug segfaults; however Objective-C's message sending is, like Smalltalk, dynamically typed. In Smalltalk this is not an issue - type errors are simply triggered, lead to predictable backtraces, and are easily fixed. While some dynamic typing errors in Objective-C lead to similar backtraces, others fall foul of C's free-wheeling approach to memory management and cause horrible run-time results, leading to little more than a splat sound; debugging those is a chore.

Memory management

One of C's many lower-level delights is its approach to memory management: put simply, the user is responsible for all memory allocation and deallocation. The virtue of C's approach is that it's simple to understand (for users) and implement (for compiler and library writers). The disadvantage is that it's easy to use incorrectly; in particular, it's very easy to forget to free memory, causing hard to debug memory leaks. A less common, but more dangerous, error is to try and free an already freed chunk of memory (the so-called double free). It is thus reasonable to say that C's memory management is low-level. In contrast, high-level memory management has, in my mind, been largely synonymous with garbage collection, which automatically frees memory (and implicitly prevents double frees). Garbage collection is not without some negative implications - for example, it is notoriously difficult to work out when garbage collection pauses will strike - but overall is generally a good thing. Indeed, garbage collection is now largely ubiquitous outside of systems and embedded programming; anyone weened on Java (or even much older languages such as Lisp and Smalltalk) will know nothing else.

As I understand things, modern Objective-C under OS X uses garbage collection. On some other platforms, such as the one I was targeting, there is no garbage collection. I initially assumed that in such cases Objective-C would revert to a standard C-style system of malloc and free - to my surprise, it does not. Instead, Objective-C uses an unusual system of sometimes-implicit, sometimes-explicit memory management. For someone used to traditional high-level garbage collection or low-level C-style memory management, it's rather hard to get your head around: there don't seem to be hard and fast rules; some of the documentation is ambiguous about the users responsibilities; and there is a good deal of obfuscating cruft which I assume has gathered over time. However, the basic idea is as follows. If a user explicitly allocates memory (via an alloc message), he is responsible for freeing it. If another function allocates memory, it will (or, more accurately, should) be put into a memory pool; when the memory pool is freed, the objects in it are freed too. New memory pools can be created, and they are kept in a stack so objects are put into the most recent memory pool.

Memory pools are an attempt to allow implicit memory allocation (something which is, for good reason, deeply frowned on in traditional C libraries) with implicit memory deallocation. In practice, they resemble a poor-mans attempt at garbage collection or, perhaps, garbage collection as designed by someone who hadn't fully grasped the idea. For the life of me, I can not fully prevent memory leaks in a medium-sized Objective-C system; ironically, I have found it much easier to debug memory leaks in pure C applications. Different parts of code can retain and release objects, which I assumed was a synonym for reference counting, but in reality doesn't always seem to quite work: what, according to the documentation, are valid combinations of calls can cause crashes or leaks. Was this due to bugs in my code, someone else's, or a flaw in the memory management design? Who knows - at some point, I got the leaks down to the level of a few bytes a minute and gave up.

The middle-level of abstraction

A good question at this point is: what does all this have to do with abstractions? Well, the two examples above are instances of something that I've seen once or twice before, but which is hardly common. In normal computing conversation, C is at a low-level of abstraction, and Smalltalk is at a high-level. As I wrote earlier, levels of abstraction are relative, but that doesn't mean we can't talk about the gaps between two levels. Objective-C, which is a child of these two languages, explicitly pitches itself as being the middle-level of abstraction between C and Smalltalk. Similarly, its memory management is the middle-level of abstraction between malloc/free and garbage collection.

The interesting thing to me is not that Objective-C is the middle-level of abstraction between C and Smalltalk as such - after all, given two points on the abstraction scale, there's a high chance that there are other levels in-between - but that it appears to me to have been deliberately designed to slot into this place. This made me wonder about two things. First, why are there not more systems designed to slot between two existing levels of abstractions? Second, why have I never heard the term middle-level of abstraction before?

A couple of answers immediately suggested themselves to me; no doubt others can think of better ones. Perhaps systems designed to slot between two existing levels are more likely than not (as Objective-C) to be worse than the things either side of it? Alternatively, perhaps under normal circumstances we only need to think of one higher level of abstraction thing and one lower-level thing in order to work satisfactorily, and what comes in the middle is generally irrelevant? Whatever the reason may be, I suspect we're unlikely to ever hear many uses of the term middle-level of abstraction - it may remain forever the missing level of abstraction.

Link to this entry

Good Programmers are Good Sysadmins are Good Programmers

March 20 2009

It is human nature to assume that what is familiar to us, is familiar to all. I can still clearly remember, when I was around 12, going to a friends house, and being astonished at the fact that around my dinner plate were three pairs of knives and forks. In fact, petrified would probably be a better word - not only had I never personally seen anything like it, I had never imagined that anyone would, or indeed could, use more than one knife and fork in a single meal.

Of course, my life thus far has not just involved my surprise at other peoples habits; on occasions (less rare than my ego might have preferred), other people have been surprised at mine. Recently a non-computing friend saw my main computer workspace - a Unix setup with 4 xterms displayed - and asked, jokingly, if I was plotting to take over the world (I blame the media for this particular image). I'm so used to my own setup that I no longer think of it as odd but, when suitably prompted, I can see why other people might think so. In comparison to a Windows or Mac machine, full of little visual goodies, and perhaps with only a web browser or word processor loaded, a number of tiny xterms filled with half-executed commands does look odd.

It's not really surprising that a non-computing person would find my setup odd - after all, I spend a lot of time on computers, so it's to be expected that some things that I have come to find natural scare casual users. What has surprised, and continues to surprise me, is how many computing people I come across find my setup odd - sufficiently odd that it attracts comment. Some people are baffled as to why my systems are as they are, some are curious as to how it works, and some people sneer at the way I do things. There is no good answer to the sneer, nor any great reason to answer that person (although, possibly due to a deep character flaw, I find the sneer rather amusing). However the how and why are interesting questions, which raise interesting points, and are more closely integrated than they may first appear.

Here's the broad setup I use. I have a desktop machine (because it's fast and comfortable), a laptop (which I use only when out and about, because laptops are slow and ergonomically disastrous), a main server (where you're probably reading this from), and a backup server (for the next time the water company cuts through the cable powering the main server; in an attempt to salve my environmental qualms, the backup server is a very low power device that also serves some domestic purposes). Though various people have some sort of access to the servers, I personally administer all 4 machines. This raises two immediate problems: how to keep the administration overhead to a minimum; and how to keep files synchronised between each machine.

The answer to the administration overhead question is, for me, simple: use the same operating system for all machines. That way, the lessons learned on one machine apply trivially to the others (and, when things go belly up, machines can relatively easily stand in for one another). As someone who (through a quirk of geography and history) never passed through the DOS / Windows world, I eventually gravitated towards Unix operating systems and, after a brief flirtation with Linux, I've exclusively used OpenBSD for nearly 10 years. This immediately scares most people off, or confirms their worst suspicions of me - to give you an idea of the popularity of this OS, at the time of writing, I've met precisely 1 OpenBSD user in real life. Why did I chose OpenBSD? Simple: it's simple. OpenBSD does very little by default, and what it does, it does well, consistently, with minimal configuration, good documentation, and is easily administered remotely. I can have a blank box turned into a complete OpenBSD install with everything I want, setup how I need, in a couple of hours (most of which is automatic downloading of stuff, and doesn't require my presence). I keep an open mind about OS replacements, but so far none of them appears an improvement, or even a sideways step. Of course, using what is often dismissively called a server operating system does involve some compromises, although less than you might think - the increasing diversity of real-world OS's (thanks indirectly, I think, to OS X) has meant that running a minority platform involves fewer compromises than it did 5 or 6 years back.

The file synchronization problem is a little more subtle, but arguably more important. I outlined my mechanism for this a while back and while it's changed in detail quite a bit, in spirit it's still the same: I use a version control system (git these days) for my important files and Unison for large files that I can recreate via other mechanisms.

A corollary of using a decent Unix, and synchronizing files automatically, is that virtually everything is configured by simple text files, so to a large extent my configuration also propagates across machines. I have also tried over the years to accept, whenever possible, the default configuration on a machine. The reason for this is simple: the less I feel the need to change, the easier it is to move between machines (and different OS's). Of course, there's a limit to how far I'm prepared to accept someone else's choices, and so I do change a reasonable number of settings; but, compared to most people I know, I change relatively little.

As well as trying to use the default configuration as far as is practicable, I also try to maximise my use of tools supplied with the OS and, failing that, to use the simplest tool that does the task I require. A decent Unix comes with a wide variety of little tools, most of them neglected by most users; it continues to amaze me as to how many tasks can be expressed in terms of these little tools. Using tools that are standard across many different machines and OS's again lowers the barriers to moving between machines. It also generally implies a greater consistency of user experience, since tools from the same providers tend to be more consistent; some providers (particularly commercial) seem to delight in perverse user interface choices, which means that installing and learning new tools can be an uncomfortable experience. I also try, whenever possible, to use command-line tools not because - despite what some of my friends think - I like being obscure (my formative years were spent on RISC OS, where the GUI was King and the default assumption was that the command-line was for the mentally unsound) but because it's easier to control command-line tools and maintain consistency across platforms.

Most of my time on a computer is spent either doing e-mail (I use mutt because it's the least annoying mail client I've yet found, despite its obvious limitations), web browsing, programming, or writing. The latter two tasks are the most interesting. If for you, as for me, an average day is a wild trip of programming in several languages, and working on several papers of dubious literary merit, then you'll know how much time one can spend editing text; I often have 20 or 30 files open for editing. The sad truth of the matter is that, as far as I can tell, the modern computing world does not contain a single decent text editor (whereas RISC OS, which I mentioned earlier, had at least 2 excellent text editors). Most text editors are either arcane (e.g. vi and emacs) or bloated (e.g. Eclipse). Since I am not clever enough for the former, and far too impatient to wait for the latter to load (I had a massive shock 4 or 5 years back when, on a powerful machine, I found to my horror that if I typed at full speed in a well known IDE, there was a noticeable lag in text appearing on screen), I use a half-way option, NEdit. NEdit has many limitations and flaws, but it's simple, loads almost immediately, and its syntax highlighting is just about powerful enough to satisfy me.

Let's return to the 4 xterms I mentioned at the beginning of the article, which scared my non-computing friend - it's both worse and better than it seems. I setup KDE so that it has 12 virtual desktops (one of the main reasons I used KDE in the early days was because it binds sensible keys to virtual desktop selection by default), of which I typically use 7 to 8 at any given point. The first is my main work area; one of the xterms has mutt permanently loaded (I occasionally load other mutts in different xterms to simultaneously read multiple folders; an advantage of using a simple tool), one is mostly used for downloading e-mail, and the other two are for random commands and ssh sessions. Desktop 2 is for web browsing and desktop 3 for web page editing. Desktop 4 is my calendar. Desktops 5 and 6 are for programming. Desktops 7 and 8 are for paper writing, with 9 and 10 being used for secondary paper writing. The remaining desktops are spares. This may seem complex or unduly pernickety. The answer to both points is essentially the same: I evolved this setup organically over time so it seems natural, to me at least. Because of virtual desktops, I need only a single 19" monitor, although these days it's a struggle to find a sensibly sized monitor. Unfortunately, computer people are generally gadget people, and gadget people are easily fooled by bigger, faster, better, so monitors these days have largely useless resolutions. Wide-screen monitors, for example, are (I assume) good for watching films, but they're fairly useless for text editing, where screen height is more important than width. Furthermore, the pointless fixation on resolution means that many fixed size things (some fonts, icons etc.) appear tiny, so I increasingly see people having to put their nose virtually to their screen to read things. 1280x1024 works for me and, until someone doubles the resolution without increasing the screen size (since then one can imagine that old apps could be transparently run with two physical pixels for every logical pixel they perceive, while new apps will have access to the genuine high resolution, making everything a bit smoother and sharper), I will try to resist the siren call of the resolution junkies.

In the above I've tried to give a brief outline of how I use computers - a modern reader may well detect a certain Luddite tendency in some of the choices, but hopefully I've also provided some small justification for each of the detailed choices. Of course, none of the above is really a high-level why. Why go to all this effort? Why these particular choices? Although I didn't explicitly think of it this way when I first started down the path that led me to my current mode of operation, there is a solid reason behind it. When I look at the really good programmers I've come across (directly and indirectly), then, with only one exception I can really think of, they seem to share one thing in common: they're also good sysadmins. Their machine(s) are in good order, simple, with the right hardware for the job, and ready for the task at hand; and they can whip a new machine into shape quickly. When the muse strikes, there is little to get in the way of good work: they know their machines inside out, the tools inside out, all their files are easily available, everything used frequently loads quickly, and they can flick rapidly between the sub-tasks that constitute a larger job. Similarly, the best sysadmins I've seen are also good programmers. I don't think it's possible to understand a modern OS without being a decent programmer - more importantly, it's certainly not possible to tame and control an OS in the desired way without programming being involved. There's a symbiosis between these two activities that seems to me undeniable; being really good in one requires being at least fairly good in the other.

So, in conclusion, my computer setup is an attempt to emulate, in my own small way, the best habits I've been able to pick up from those more able than myself. It's a continual work in progress, but it does the trick for me.

Link to this entry

How can C Programs be so Reliable?

November 11 2008

C is, today, a unique programming language. Surprisingly few people can really program in C and yet many of us have quite strong opinions about it. Buffer overflows, stack smashing, integer overflows - C has many well publicised flaws, and these terms are often bandied about confidently, even by those unfamiliar with C. Personally I shied away from C for a decade, for one reason or another: originally, compilers were expensive (this being the days before free UNIX clones were readily available) and slow; the culture was intimidatory; and, of course, all the C scare stories made me think that a mere mortal programmer such as myself would never be able to write a reliable C program.

Discounting a couple of tiny C modules that I created largely by blindly cutting and pasting from other places, the first C program I wrote was the Converge VM. Two things from this experience surprised me. First, writing C programs turned out not to be that difficult. With hindsight, I should have realised that a youth misspent writing programs in assembler gave me nearly all the mental tools I needed - after all, C is little more than a high-level assembly language. Once one has understood a concept such as pointers (arguably the trickiest concept in low-level languages, having no simple real-world analogy) in one language, one has understood it in every language. Second, the Converge VM hasn't been riddled with bugs as I expected.

In fact, ignoring logic errors that would have happened in any language, only two C-specific errors have thus far caused any real problem in the Converge VM (please note, I'm sure there are lots of bugs lurking - but I'm happy not to have hit too many of them yet). One was a list which wasn't correctly NULL terminated (a classic C error); that took a while to track down. The other was much more subtle, and took several days, spread over a couple of months, to solve. The Converge garbage collector can conservatively garbage collect arbitrary malloc'd chunks of memory, looking for pointers. In all modern architectures, pointers have to live on word-aligned boundaries. However, malloc'd chunks of memory are often not word-aligned in length. Thus sometimes the garbage collector would try and read the 4 bytes of memory starting at position 4 in a chunk - even if that chunk that was only 5 bytes long. In other words, the garbage collector tried to read in 1 byte of proper data and 3 bytes of possibly random stuff in an area of memory it didn't theoretically have access to. The rare, and subtle, errors this led to were almost impossible to reason about. But let's be honest - in how many languages can one retrospectively add a garbage collector?

My experience with the Converge VM didn't really fit my previous prejudices. I had implicitly bought into the idea that C programs segfault at random, eat data, and generally act like Vikings on a day trip to Lindisfarne; in contrast, programs written in higher level languages supposedly fail in nice, predictable patterns. Gradually it occurred to me that virtually all of the software that I use on a daily basis - that to which I entrust my most important data - is written in C. And I can't remember the last time there was a major problem with any of this software - it's reliable in the sense that it doesn't crash, and also reliable in the sense that it handles minor failures gracefully. Granted, I am extremely fussy about the software I use (I've been an OpenBSD user for 9 years or so, and software doesn't get much better than that), and there are some obvious reasons as to why it might be so reliable: it's used by (relatively) large numbers of people, who help shake out bugs; the software has been developed over a long period of time, so previous generations bore the brunt of the bugs; and, if we're being brutally honest, only fairly competent programmers tend to use C in the first place. But still, the fundamental question remained: why is so much of the software I use in C so reliable?

After a dark period of paper writing, I've recently been doing a little bit of C programming. As someone who, at some points, spends far too much time away from home, reliably sending e-mail has always been an issue. For several years I have sent e-mail by piping messages to a sendmail process on a remote machine via ssh. While this solves several problems (e.g. blacklisting), it has the problem that on many networks (particularly wireless networks) a surprising number of network connections get dropped. Checking that each e-mail has been sent is a frustrating process. So, having mulled on its design for a little while, I decided to create a simple utility to robustly send e-mail via ssh. The resulting program - extsmail - has more features than I'd originally expected, but the basic idea is simply to retry sending messages via an external command such as ssh, until the message has been successfully sent. I also wanted the utility to be as frugal with resources as practical, and to be as portable as possible. This inevitably led to extsmail being written in C. I then decided, as an experiment, to try and write this, as far as possible, in the traditional UNIX way: only to rely on features found in all sensible UNIX clones and to be robust against failure. In so doing, I made two observations, new to me, about writing software in C.

The first observation is semi-obvious. Because software written in C can fail in so many ways, I was much more careful than normal when writing it. In particular, anything involved in manipulating chunks of memory raises the prospect of off-by-one type errors - which are particularly dangerous in C. Whereas in a higher-level language I might be lazy and think hmm, do I need to subtract 1 from this value when I index into the array? Let's run it and find out, in C I thought OK, let's sit down and reason about this. Ironically, the time taken to run-and-discover often seems not to be much different to sit-down-and-think - except the latter is a lot more mentally draining.

The second observation is something I had not previously considered. In C there is no exception handling. If, as in the case of extsmail, one wants to be robust against errors, one has to handle all possible error paths oneself. This is extremely painful in one way - a huge proportion (I would guess at least 40%) of extsmail is dedicated to detecting and recovering from errors - although made easier by the fact that UNIX functions always carefully detail how and when they will fail. In other words, when one calls a function like stat in C, the documentation lists all the failure conditions; the user can then easily choose which errors conditions he wishes his program to recover from, and which are fatal to further execution (in extsmail, out of memory errors are about the only fatal errors). This is a huge difference in mind-set from exception based languages, where the typical philosophy is to write code as normal, only rarely inserting try ... catch blocks to recover from specific errors (which are only sporadically documented). Java, with its checked exceptions, takes a different approach telling the user you must try and catch these specific exceptions when you call this function.

What I realised is that neither exception-based approach is appropriate when one wishes to make software as robust as possible. What one needs is to know exactly which errors / exceptions a function can return / raise, and then deal with each on a case-by-case basis. While it is possible that modern IDEs could (indeed, they may well do, for all I know) automatically show you some of the exceptions that a given function can raise, this can only go so far. Theoretically speaking, sub-classing and polymorphism in OO languages means that pre-compiled libraries can not be sure what exceptions a given function call may raise (since subclasses may overload functions, which can then raise different exceptions). From a practical point of view, I suspect that many functions would claim to raise so many different exceptions that the user would be overwhelmed: in contrast, the UNIX functions are very aware that they need to minimise the amount of errors that they return to the user, either by recovering from internal failure, or by grouping errors. I further suspect that many libraries that rely on exception handling would need to be substantially rewritten to reduce the number of exceptions they raise to a reasonable number. Furthermore, it is the caller of a function who needs to determine which errors are minor and can be recovered from, and which cause more fundamental problems, possibly resulting in the program exiting; checked exceptions, by forcing the caller to deal with certain exceptions, miss the point here.

Henry Spencer said, Those who don't understand UNIX are doomed to reinvent it, poorly. And that's probably why so many of the programs written in C are more reliable than our prejudices might suggest - the UNIX culture, the oldest and wisest in mainstream computing, has found ways of turning some of C's limitations and flaws into advantages. As my experience shows, I am yet another person to slowly realise this. All that said, I don't recommend using C unless much thought has been given to the decision - the resulting software might be reliable, but it will have taken a significant human effort to produce it.

Link to this entry

Free Text Geocoding

September 1 2008

I think nearly everyone would agree that maps are useful things; I've only met a couple of people who labour under the illusion that they know the way from anywhere to anywhere. Personally I find maps more than interesting - they're often fascinating. Maps are one thing the British do immensely well (judging by some of the inaccurate doodles I've seen abroad). By looking at the detailed Ordnance Survey map of the area where I grew up, I can easily see layers of historical information such as: the Roman road a mile away; the long-abandoned railway lines upon which roads were later built; the now-drained marshes and the canal network later built upon them. Of course, these are fairly standard maps with a well-understood relationship to the real world; sites such as Strange Maps and Mark Easton's blog show how maps can present data in many other forms. My headmaster at school thought that geography as a subject had lost its way; instead of focusing on rock strata, it should focus on teaching children where places where. At the time, we all thought he was a touch eccentric; in retrospect, he was absolutely right. Huge tracts of politics, economics, and history only make sense in the context of a given place(s). I could go on, but I hope my point is clear: maps are important, and not just for finding our way around.

A big problem with traditional maps is finding things: places, areas, and so on. Even for a medium sized paper map, the size of the index is huge; and the index for a good quality London A-Z is often almost as big as the map itself. Paper map indexes have their own funny little language, trying to squeeze as much data in as possible. However paper indexes have two problems. First, how many times have you forgotten what square you were supposed to look at when you get to the proscribed page? Second, indexes can only grow to a certain size before they are unusable.

Computer geocoding

The advent of computer mapping has been a real boon, and not to just map fiends such as I. The first site I used regularly was Street Map. Having the map data for the whole UK was incredibly useful and the search facilities, while crude, were a huge improvement on a paper index. When later sites such as Google Maps became available, I was stunned. Suddenly freely viewable map data (though note I do not use the phrase free map data) for much of the world was coupled with a new way of searching. No paper index, or Streetmap's cloying need to be told what type of search was being performed (e.g. place name, street name, post code etc.). Instead is what I call (for want of a better name) free text geocoding: that is, where one types in something in the format used in real-life, and the search engine finds the right place automatically. One doesn't need to tell the search engine that the search is for a postcode or a place-name, or even which country one is searching - it magically does the right thing. Well, of course - not always the right thing. For example, at the time of writing, if I do a search for Penrith in Google Maps UK, I get taken to a suburb of Sydney in Australia. The poor residents of Penrith in the North of England don't even have their town mentioned as a possible match for the search; one must instead search for something like Penrith Cumbria or Penrith UK. There are a range of related minor infelicities in both Google Maps and Yahoo Maps. However on average, both do an adequate job.

There are several reasons why sites such as Google Maps are not as widely used as they might be. Regrettably the chief reason is legal: there are restrictions on the way that map data can be used. Fortunately an initiative - OpenStreetMap - to create much freer map data was started a few years back (and started by Brits - we are, it appears, a nation of map lovers) and is now at the stage where, despite many huge holes in its data, it is semi-usable: in central London, for example, it already has arguably the highest quality maps. Some of the uses of OpenStreetMap are already quite astonishing (the UK postcode layer is a simple, but effective, example of what can be done - click the little + icon in the top-right of the map for more fun), and its accelerating progress is impressive.

One part of OpenStreetMap that frustrates me a little is its search (the Name Finder). Although it does a reasonable job in many respects, the results it returns are hard to interpret (type in Penrith and then try and work out which result is the UK town - I got this wrong on my first go and I'm English!), and it's not easy to use it outside of a website (because it's written in PHP). Furthermore the version running on OpenStreetMap's front page is painfully slow (possibly because it's running on overloaded servers). [At the time of writing, I can't even work out how to get it to find Penrith in the UK automatically. Queries like Penrith, UK crash it. Penrith - please don't take the cold-shoulder from the map search engines personally!]

Free text geocoding

Since one of the huge benefits of using computer mapping is its search ability, I thought a fun little summer project would be to create my own free text geocoder. I started with only a vague idea of what a free text geocoder should (or could) do. While I don't now claim to have thought of everything, I do now have a much clearer idea of what a good free text geocoder should do. I've split these into must and should haves. A free text geocoder must:

give accurate results (meaning it always, somewhere in the list of matches, gives the place the user is looking for). While this might seem obvious, some of the existing free text geocoders don't do this, as we saw earlier.
require as little formatting from the user as practicable (e.g. London SW1 and SW1 London are both likely searches).
return any unmatched text (thus allowing searches like cafes in Pimlico to return Pimlico as the place matched and cafes in as unmatched text which an application can then use as it pleases).
be fast: results shouldn't take more than a second on average (and preferably should be much quicker).
be localisable. This is both linguistic and cultural. Searching should take place disregarding the users input language, but results should be localised when possible. Results should be formatted relative to the users local cultural expectations (e.g. in the US, state names are always shown; in the UK county names are nearly always shown).

If possible, it should:

be possible to use it do more than just find the latitude and longitude of a place.
try and weight the results so that the most likely match(es) are given higher priority (e.g. an Englishman searching for Penrith should see the English town as the first match, while an Australian should see the Sydney suburb).
be usable in different contexts (e.g. in websites, or in applications).
be amenable to (possibly quite low-level) customization.

Fetegeo

To this end I created, and have now released, Fetegeo with a BSD / MIT licence. Using Fetegeo's included client / server interface, queries can be performed on the command-line:

$ fetegeoc geo London
Match #1
  ID: 719913
  Name: London
  Latitude: 51.508415
  Longitude: -0.125533
  Country ID: 233
  Parent ID: 1262
  Population: 7421209
  PP: London, United Kingdom
  Dangling text:

Of course, there are a lot of London's in the world and I haven't copied all of Fetegeo's output. Notice though that, since the preferred country of the user wasn't specified, it's chosen what most people are likely to consider to be the London as the first match. If the user specifies that their country is Canada then London in Ontario is the first match:

$ fetegeoc -c ca geo london
Match #1
  ID: 2984878
  Name: London
  Latitude: 42.983389283
  Longitude: -81.233042387
  Country ID: 39
  Parent ID: 540
  Population: 346765
  PP: London, Ontario
  Dangling text:

Fetegeo can be instructed to allow dangling (i.e. unmatched) text in matches:

$ fetegeoc -d geo Museums in London
Match #1
  ID: 719913
  Name: London
  Latitude: 51.508415
  Longitude: -0.125533
  Country ID: 233
  Parent ID: 1262
  Population: 7421209
  PP: London, United Kingdom
  Dangling text: Museums in

If you're interested, there's a slightly more thorough description of the ways that Fetegeo can be used, and a simple demo which geocodes results and shows them on an OpenStreetMap map.

How Fetegeo works

Internally, Fetegeo's search is fairly simple and its approach is easily described. Strings in Fetegeo are always normalised; in particular punctuation is removed, and strings are lower-cased. String queries to the database are always on hashes of normalised words. Given a normalised string S, Fetegeo breaks it into a list of words. It then works right-to-left in the list, trying to find all possible matches. Whenever a match within S occurs, a counter is decremented meaning that subsequent matching takes place n elements from the end of the list. Matches are greedy; they always try to first match the maximum number of words possible, before gradually trying fewer words. Matches are exhaustive in the sense that all possibilities are tried; however only the longest matches are eventually returned to the user (some obvious optimisations are used here, so that some possibilities aren't tried if it's obvious they won't work). Fetegeo first of all tries to match the entirety of S as being a place within the users country; if that fails, it tries to match a country name at the right-hand side of the string. It then tries to match places and postcodes; if a place is found, it is considered to be a parent area; subsequent matches can only match places within that area or (recursively) a sub-area. Postcodes can occur at any point in the match. Of course, there's a lot more detail than this in the code, but this is the essence of Fetegeo's fast free text geocoding.

How Fetegeo compares

How does Fetegeo score on the must / should chart?

It's too early to say how accurate Fetegeo's results are. First, Fetegeo has only been relatively lightly tested so far: it's inevitable that there will be bugs and oversights. Second, any free text geocoder is subject to something I'm tentatively calling Tratt's First Law of Free Text Geocoding (though I doubt I'm the first to think of it): the upper bound for results quality is determined by your dataset. The best free text geocoder in the world can only give iffy results with an iffy dataset. Fetegeo's initial dataset is based on Geonames data (and postcode data from various other sources). While Geonames should be saluted as the first serious attempt to collate freely-available place data, the structure of the data is less than ideal, and the data itself is of variable quality, suffering from frequent inaccuracies and duplication. Because of this, Fetegeo has been designed to be relatively independent of any particular dataset; I hope one day in the not too distant future that OpenStreetMap's data will be sufficiently broad in scope to replace Geoname's (OpenStreetMap's data is already deeper in the sense that it includes roads, which Geonames doesn't).

Fetegeo is already reasonably fast, given that it's only been semi-optimised. On my 3-ish year old desktop machine, using the stock install of PostgreSQL (a few tweaks would, I suspect, make it perform much better - if only I could work out what those tweaks were amongst the mass of overlapping configuration options!), typical queries are answered in less than 0.1s. Fetegeo makes use of simple caching internally to speed things up. Someone who understands databases better than I could almost certainly make things run much faster.

Fetegeo has the beginnings of being usable for more than just longitude / latitude searches, but there is some way to go yet to prove this is feasible. In particular I would like to see it capable of being used by applications to classify things as being in particular areas. Imagine you have a website listing X's in Britain (where X could be just about anything), where each X is located at a particular latitude / longitude. This allows one to easily search for all X's near place P. However users often want to perform area searches such as all X's in London or all X's in Rutland. Exposing the identifiers of places, counties, states (and so on) makes this latter type of query feasible.

Fetegeo is usable in a number of different ways. As such, Fetegeo is just a Python library which can be included and used in any application. Fetegeo also comes with a standard internet server and (command-line) client, which can receive and answer XML queries (as an aside, the XML parser used is often the slowest part of querying). This means that even a simple web-site can query a single Fetegeo server and make use of its caching facilities and so on.

Conclusion

I have no idea whether anyone will find Fetegeo useful. It seems to me that, even in its current embryonic form, it fills an unoccupied niche, at least in terms of its licence if not its functionality (yet). I hope that other people might find it interesting, and start to extend its functionality to make it more widely applicable. If you want to find out more, and contribute, please waltz on over to fetegeo.org.

Link to this entry

Earlier articles

All articles

Last 10 articles

A Proposal for Error Handling

The Missing Level of Abstraction?

Good Programmers are Good Sysadmins are Good Programmers

How can C Programs be so Reliable?

Free Text Geocoding

Extended Backtraces

Designing Sane Scoping Rules

Some Lessons Learned from Icon