Oryx failed

About ten years ago, Abhijit Menon-Sen and I discussed email storage, archiving and access, and set out to write something good. We failed. It's time to think about why; if this reads like a stream of consciousness it's because I'm thinking.

We were right: Archiving all mail and searching it is the right approach. Gmail has a billion users. (Hm. How many of the people who told me I was wrong in 2002 use gmail now? I can think of at least five.) But we still failed.

I've always blamed the onset of sleep problems. When I lost the knack of falling asleep, I started doing really stupid things. Pissing people off, writing bad code, crossing the street in front of cars. Not good. But as an explanation it has a certain air of comfort. I distrust comfortable explanations.

We also made the classic mistake: We never showed any real advance over classic mail storage. We had the advance, we just never showed it. But that too isn't really an answer, because it merely asks why again.

I think IMAP is part of the answer. We spent too much attention on IMAP, and robbed ourselves of much function and even more performance. We needed search, and IMAP's builtin search is much too primitive. Our efforts to extend searching ended in committee failure, and we didn't understand the implications of that. The phrase tunnel vision springs to mind.

We needed performance, and I knew even then that accessing the email database via other means is much faster than via IMAP. I'm not sure why that is so. I think there must be several reasons. But whatever the reasons, aoxexport can export my complete archive faster than Thunderbird can read my new mail most mornings. It's hard to show real advance wearing leg irons.

Gmail, on the other hand, has less powerful search, but it uses json and a web browser instead of imap and client programs, and by that route its search performance and capabilities actually reach the user.

We also made installation too difficult. Noone has a better database/schema installer than we wrote, and noone has better configuration file handling and error messages. I fooled myself into thinking that was enough. The way Archiveopteryx upgrades the database is wonderful, but who notices that until they're already a long-time user?

I see now that others ship virtual machines, or provide hosted services on one of the big clouds, or both. Much, much easier. Simple evaluation, too. In this respect, as in several others above, we failed to take a step back and look at our status.

We provided no spam filtering, or integration with spam filtering. Ditto virus. We provided no calendaring/groupware functionality, or integration with that. We provided no Outlook compatibility. Zero points out of four possible. I don't think we needed four out of four. Many people would say we did, but many people are far too quick to make long requirements lists. Still, zero is a small number.

I came across this photo while the text above was half-done. Google and Tineye find a lot of sites with this photo and stupid funny captions. I don't think it's at all funny.

I tried to find the rights holder for the photo and, fittingly, failed.

Update: Source. It's a Reuters photo. I just spoke to a friendly lady at Reuters, who will send me an invoice.