A unit test failure

A few months I had a bug in a class without unit tests, and I can't stop thinking about the case.

There's a reason why I had no unit tests. I do use unit tests by default, but there are exceptions: If there is no real unit smaller than the program, I don't use unit tests. If the program only has to work once, I generally don't bother to test units. If correctness isn't formally knowable, ditto. And finally, if a unit test wouldn't really test.

Fuzzy correctness is the most common reason I skip unit tests. Earlier this year I skipped one because the goal of the change was to get rid of some unpleasant flicker. I managed to make the flickering cases uncommon, but I still don't know what unpleasant really means, so how would I write a unit test? I've heard games have this problem quite often, because correctness can mean is fun to play. But if flicker becomes bothersome again later, I know I will wish there were a unit test.

Tests that don't test are also a serious problem. In this case, the function that broke looked like return "foo" + bar(42, false), and the bug was that I used the wrong string constant. I very much suspect that the test would have been another one-line function with the same wrong string. Code review ought to pick such things up. Most code reviewers I've known wouldn't, though.

Most of the functionality in the class I was writing is outsourced to a backend. The class itself writes simple requests and processes equally simple responses. The caller calls millions of lines of code, but what broke was the simple, straight-line function that creates the request. And it broke spectacularly — the result was valid, it just changed the meaning of the request, made the wrong thing happen in the backend, which in turn instructed the frontend to change its state and usually issue some followup requests, the backend processed those too, and eventually the screen would show a valid but usually puzzling or misleading state.

I think that in such cases, I want protocol review, to answer a question such as is this a good frontend↔backend conversation given this user input? or perhaps based on this frontend↔backend chatter, what do you think the user did?.