Arnt Gulbrandsen
About meAbout this blog

The changing nature of breakage

Software is changing. So are its problems.

Software today is developed in a more regimented and orderly manner than twenty years ago. A few good practices have taken hold:

  • Version control
  • Code review
  • A sort of of agile development
  • Coding standards
  • Wide use of libraries, toolkits and frameworks

These are good things. Even if the agile development is often a parody of what it could be, it's still a net positive factor, in my opinion.

These good things have bad effects, too. Perhaps most notably, developers don't know what their third-party code does, and that affects what they know about their own software, and what the users can know and do.

Code gets used because of some known function, without anyone learning the full scope of functionality. Take Ruby on Rails and the action-mailer package, for example. If you want to send HTML, you probably want an add-on package that modifies your site's CSS for common mail reader compatibility, Premailer is the most common one.

There will be an issue in the issue tracker to add Premailer and fix CSS compatibiliy. Some team member will take the issue, add the necessary five lines of code, test that it looks right in gmail and outlook, write a test, and that was it. The premailer web site doesn't say whether premailer takes care of printouts (@media print {...} in CSS), and most likely noone on the team will ever know whether that works or not.

That lack of knowledge results in poor documentation and bugs around the edges. The team doesn't know everything its software actually does, and as a result, the team writes bland, incomplete documentation. It's growing harder and harder to read about something and think ok, i know what this does, and one reason is that the authors of the documentation don't know what the software does.

Today's software has lots of unintended and untested functions. If the software uses Ruby on Rails, it may well use a gem that reads and converts two hundred image formats. Those formats aren't supported, they just happen to work. They're supported and tested in the third-party gem, not in the software users use. The users have to experiment to see what the software does. Do features such as colour spaces or translucency work?

Software suffered from a lack of support in the old days. Most of the 200 image formats weren't handled at all, CSS in email wasn't generated at all. Now it suffers from a lack of insight into what the code actually does. The maintainers don't know whether colour spaces are handled in the way users may expect, the documentation doesn't say, and the users have to experiment as soon as they use anything other than JPEG.

A second ill effect is that the lack if insight dampens the vision into what the software ought to be, making today's software incoherent. In the old days it was more likely to be incomplete than incoherent.

Wietse Venema and Postfix are a great example of how this can work. I've written on this blog that if you combine any two Postfix settings, the combination works as intended because Wietse has thought about that combination. Wietse knows Postfix deeply. If someone comes along with a proposal to implement a new feature, Wietse will know how that actually interacts with every existing feature and also how it ought to interact.

That depth of vision is difficult to achieve if your software handles image files, but you don't even know whether or how your code handles colour spaces in some formats. Or if you do know it, you don't know whether that's an accidental or intentional feature of the third-party library you use.

Some people do have vision. I've come across several CEOs and startup founders who used their product harshly, knew what it did and knew its soul, too. Not so many developers any more. A decade or two ago, some developers knew their code inside-out too, that's impossible now.

A small tool I worked on in 2016 had 1514 third-party dependencies, and that was one of 50+ systems being maintained by a team of 10-15. Just knowing all the 1514 by name is inhuman.

Most developers don't have any chance to attain complete insight, so things like Scrum meetings typically happen without anyone really insightful in the room.


Three programs, one feature

It's not something one does often, but I've implemented the same feature in three different programs. Not very different, all are written in the same programming language for the same platform, and all are servers.

Same platform, same language, same task, same developer... you would think the three patches would end up looking similar? They did not, not at all.

The feature I wrote is is support for using UTF8 on SMTP, which I've implemented for Postfix, Sendmail and Qmail, which all run on linux/posix systems. I tried to follow the code style for each of them, and surprised myself at how different my code looked.

One patch is well-engineered, prim and proper.

The next is for an amorphous blob of software. The patch is itself amorphous, and makes functions even longer that were too long already. Yet it's half as long as the first patch. The two are, in my own judgment, about equally readable. One wins on length, the other on readability, they're roughly tied overall. This surprised me not a little.

The third is a short, readable patch which one might call an inspired hack. It's a much smaller than the others and easily wins on readability too.

It wasn't supposed to be like that, was it? Good engineering shouldn't give the most verbose patch, and the hack shouldn't be the most lucid of the three.

I see two things here:

First: Proper engineering has its value, but perhaps not as much as common wisdom says. Moderately clean code offers almost all of the value of really clean code.

Second: A small program is easy to work with, such as the MVPs that are so fashionable these days. But ease of modification isn't all, the smallest among the three servers has fallen out of use because the world changed and it stopped being viable.

Some random verbiage on each of the three servers and patches: (more…)


Code review times two

I like code review but complain about it all the time. Why? The other day I had an eureka moment: There are two quite different kinds, which I will give the misleading names agile review and waterfall review, and I'm annoyed when the reviewers perform one kind of review and I want the other.

Waterfall development is centered around planning, development and deployment in stages: First plan well and get rid of most bugs, then implement what's planned, then review and test to fix the remaining bugs, then deploy. In this model, code review focuses on catching minor bugs, ensuring code style harmony, proper use of internal/thirdparty APIs, etc. Big questions such as is this feature a good idea at all? are out of scope, because that's been settled in the planning phase. (I'm not saying waterfall works in practice, only describing the theory.)

Agile development is different. Instead of planning well, agile development is based around an iterative process: Implement something that seems to be a good idea, consider it, learn something from it, finish it and then deploy, then pick something else to do, something one now knows more about. Code review can be the first part of the consideration and learning process, so it's a good time to discover that although a feature seemed like a good idea, it's going to be a morass and should be dropped at once. (more…)


Morbid bindings

The first thing I learned about programming in-person (not from a book or from code) was morbid bindings, a strangely unused term.

I learned it from a university employee called Eric Monteiro, who was oddly clueful in a generally sparsely beclued environment. Can't remember his title, or what he taught.

I had just listened to Eric answer someone else about multiple inheritance, and asked a followup question, which he answered, and then he digressed into relationships in general: When two things are connected in a program, the binding is always one of three: A kind, a belonging, or else it's morbid, and morbid bindings always turn out to be bad in one way or another. He gave me a quick example: This number in this file has to be at the same as that number in that file and I think I answered, such as maximum line length in two functions that read/write the same file? (more…)


The value of features

A programmer doesn't always know whether a new feature of a program will turn out to be valuable or not. Perhaps: doesn't even often know. I've just had a repeat lesson on that topic.

I have a new phone. The manufacturer brags about a high-resolution camera and many other things, but I bought it because it's the smallest phone with a good screen and an up-to-date version of Android. (Yes it fits in a pocket, even sideways in some pockets.) I noticed in a review on the web that the phone's watertight, complete with an underwater photo of a beauty in bikini, but didn't give any thought to it. After all I don't spend much time in pools or at beaches, and if I do the bikini beauties don't gravitate towards me. When I bought the phone I had no idea that I would care about its being waterproof.

But I do care. The summer rains here in Munich can be impressively intense. Until now I've always been conscious that I was exposing an expensive and fragile electronic device to water when I used a phone in the rain. I have done that when I needed to, but in a corner of my mind I was always aware of the risk. Now I just do whatever I need to do, rain or shine, and don't worry about the device.


On software architecture

It's not particularly sensible, and not related to any software architecture I deal with at the moment, but I do want to post this photograph. Nominally it's is a photo of the concrete kind of architecture, not the software kind, but doesn't it look like an enterprise-ready, flexible, feature-rich and polished staircase framework?

It's from a hotel interior, so I expect there's a lift off-camera that people use when they want to get anywhere.


Testing versus code layout

Here is a simplified version of function I once wrote:

    if(foo() && bar())
        return true; // common case
    else if(foo())
        return false; // happens too...
    else if(mumble() == jumble() && bar())
        return true;
    else if(mumble() == jumble() && rotten())
        throw new MalevolentClientException( ... );
    else if(mumble() == null)
        return true;
    return false;

The real foo() was something along the lines of getRequest()­.getClientIpAddress()­.isRoutable(), so not awfully cheap, but it was O(1).

As you've noticed, foo() may be called twice even though the second return value has to be the same as the first. The function could be more efficient:

    if(foo()) {
            return true;
        return false;
    } else if(… (more…)


Making X11 usable across the internet. A sigh.

I am the fool who tweaked Qt to work with long-distance X11. The main problem was slow startup; if the ping time to the display was 0.1s, then applications needed 0.5s or more to start. So I fixed that, tested it using Qt/X11 applications on a transatlantic link, then fixed whatever else that I noticed because of the slow link.

As far as I know, noone ever made use of this work.

That didn't prevent someone at Trolltech from trying some of the same again twelve years later, except this time with better buzzwords and more effort.

So why did I do it? I don't remember. Perhaps someone had told me that X11 was network-agnostic by design and Qt's implementation fell short of the design, and I let myself be persuaded. Perhaps I told myself. One thing is sure: Noone had told me that this was a problem for their actual, concrete, existing application.

Maybe my work helped sell a seat or two for Qt. Many people are willing to pay for features or performance they don't need, so it's entirely possible. But I think it was probably waste. I feel good about it, though, because quite recently I was able to avoid a mistake by having learned from those two.

I've been a professional programmer for twenty years now. That seems to be long enough that for whatever mistake anyone proposes to make, I have made or watched a similar mistake at least once before. I'm not sure whether that's good or bad. Seeing a mistake be repeated is such a sad feeling.