Cryptlib in a sea of OpenSSL

Archiveopteryx uses Cryptlib. Still. It's good code, and Abhijit and I trust Peter. Almost everyone else on the planet uses OpenSSL. A few outcasts use something that shares code with OpenSSL, such as GnuTLS or SSLeay (why do these people all have a MiXEDcaps fetish?), and there's an even smaller lot that uses cryptlib, matrixssl etc.

Naturally, there are interoperability issues. There always are, when one implementation dominates overwhelmingly. Doesn't matter whether it's Windows as OS, Word as editor, Netscape/Mozilla/Firefox as browser, OpenSSL for TLS: The dominant implementation sets the standard, and the alternative contenders have to be compatible or deal with the pain.

So Archiveopteryx still uses Cryptlib, but maybe not for much longer. Like the FreeBSD user who finally caves in and switches to Windows in order to get interoperability, we may switch to OpenSSL in one of the next releases.

Some reasons:

Chained certificates are very nice. The CA root signs A, A signs B, B signs C and the (mail/web/…) server presents C as certificate. Would be nice if we could get chains to work with Archiveopteryx. We can't, though, the certificate conversion fails with a strange error. Peter Gutmann thinks there is some problem in the source. That's correct, it's in an OpenSSL-specific format and probably depends on OpenSSL-specific behaviour, but all that doesn't help. At the end of the day Archiveopteryx must be able present certificates signed by the regular CAs. If such a certificate depends on nonstandard features, then so be it.

Strange warnings cause support load, and this bug affects connections between most mail readers and Archiveopteryx. We can't reproduce it, we can't fix it, what can we do? People call support when it happens, and it's been unsolved for years now.

Mozilla Thunderbird sometimes says Error -12244 when it talks to Archiveopteryx. No idea what it means (beyond received an unexpected Certificate handshake message), but we've had no luck making it go away (or even reproducing it) and it too causes support load.

It's all very frustrating. I really don't want to struggle with OpenSSL again. I've dealt with that before and it was no fun.

Update: We switched.

Update: I should point out that the bugs mentioned above are very unlikely to be Cryptlib bugs. For instance, Thunderbird is known to emit the -12244 message with some mail servers that do not use Cryptlib. But when you're interoperating with a gorilla, the gorilla's bugs are your problems.