Good unit tests, bad unit tests

The author of the most pleasant java code I've ever seen wrote the following paragraph on an internal mailing list last year, or perhaps the year before:

By the time I met my last significant mentor I had been programming for 15-16 years. I was skeptical to much of what he wanted me to do, but I shut up and gave it a few weeks — eventually incorporating much of his thinking into how I work and evolving my own style. He was amazingly bad at explaining things verbally, and to be honest: his code looks like shit. But he kept coming into my office and demanding all the right things of me; that I think about what I output (seriously, do you need that log output!?), that I have near 100% test coverage even if it means the test code is 5 times larger than the code it tests, that I always remove code that isn't used, that I never implement methods that I don't need right now, that all public methods and classes are always documented and that the documentation be kept in sync with the implementation, that the code will build and run its tests on any machine and without screwing over other developers by assuming what is and isn't installed on the machine, that it compiles without warnings, that software is trivial to fire up, that any and all performance critical code is benchmarked, that code is never prematurely optimized but that design-work always consider performance implications of choices etc etc. I got yelled at every day for at least 3 months.

That mailing list was internal to a company that cared deeply about doing all the right things, avoided almost all of the cliché mistakes and yet missed deadlines serially. (I didn't know it yet, but later, the company would cancel its main product, weeks before release.) The demanding mentor was someone whose code looked like shit, yet demanded all the right things. There were parallels there, I thought, maybe I could use one to learn about the other. So I put it aside for a think and a ramble. Then time passed, as is its habit.

Are those are really all the right things, or even the right things? I think not. It's a summary of best practices. It's all about means, not about ends, and if you stop thinking about the goal you may never get there.

Wrote a great swordsman: The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him.

I don't want to cut anyone. I define the goal — Goal — as that the code we write be used by people to do something useful. I am going to collect those right things as four main points, and for each of them I will argue that keeping the goal in mind is key to working well.

1. Write code. This is our most important activity. Without code, all we have is vapourware and grand plans.

Automated testing serves this, like most of the mentor's list. Automated tests exist so you won't have to slowly repeat manual tests, so you can test many things quickly, so you'll even test the code you know cannot be broken.

Most the mentor's list is things that serve the code, and I think this is why it rubbed me the wrong way: The list focuses on indirect means. On things that help things that help reach the goal. That's a little too indirect. That's indirect enough to let mortals lose track of the goal, even as they think they're focused.

It's just the right degree of indirection to lose focus, but keep an illusion of focus. The goal is close enough to feel focused, but in reality, the goal has slipped into the background, behind Key Quarterly Objectives, Sprint Targets and other unreal substitutes.

One of my last changes at that company was to one to reduce the frequency of one cause of false monitoring alarms. Normally I find it easier to write code with unit tests than without, but in this case writing the unit test was a lot more work than the code itself. So did that change require a unit test, or would it be better to save time and skip the test? That code wasn't shipped to anyone. Customers used it via the network and cared only about uptime, not about monitoring alarms. Too many false alarms might confuse the on-call people and thereby harm uptime, but that's no reason to add this particular unit test. It may be a reason to add an acceptance test for the total number of false alarms.

Adding a unit test for this change shows that you've lost touch with the goal and are adding unit tests because unit tests are a corporate Key Quarterly Objective.

(I didn't write that unit test, and I'm proud not to. In general I did as badly as the rest of the team, but this instance I got it right.)

Compiling without warnings is also on the list. A good thing indeed. If you agree and are focused on the goal, the code you compile and check for warnings is precisely the code users (will) use. If, instead, you're mindlessly doing right things, you may easily find yourself obsessing over the quality of some code while other code slides through without any review at all. For example, you may have elaborate review of inhouse code and at the same time ship third-party binaries of open-source libraries. CI process here, absolute vacuum there.

2. Focus on the code and how it's used by people. The code we write is to be used by people, to do things. Some of the mentor's points are corrollaries of this. He says to not write unneded code. I say that if you focus on the code being used, you don't write unneeded code.

On the other hand, if you focus on not writing unneeded code, then you won't write unneeded code, as you see it. But you won't have been really thinking about what the users need. One colleague of mine wrote 17,000 lines of math library for something that needed clever math; we threw it away and shipped on time. Eighteen months later he wrote a fine database portability layer so that unit testing didn't require the same DBMS as the production system. It was right according to the checklist, the code will build and run its tests on any machine and without screwing over other developers by assuming what is and isn't installed on the machine, but it was sorely lacking in user focus. Only proper focus tells you whether code is really needed.

Many people would use this as a rationale for shipping early. Perhaps it is.

3. When something hurts: Stop doing that. Things always change. Some changes happen to make development faster or better, others harm. I state, without supporting argument, that the first kind of change is good and the second is bad. Then I use that unsupported statement to support this one: If you recognize and roll back bad changes, you'll do better than if you don't.

What feat of logic.

Telling whether a change is/was bad is the most difficult part, and that part is much easier when you have a clear goal clearly in mind. Focus matters.

I cannot believe it now, but I have worked on myself to suck more. I have tried to teach myself to talk to people before writing code, even though talking to two people invariably meant waiting for at least one of them. I have tried to teach myself that waiting entire days for code reviews was okay, filling my time with unimportant tasks or worse. It's wrong. The result may may be billable hours, it may comply with company policy, it may get something (unimportant) done, but it's still wrong, because it's weakly tied to the goal, and the goal defines what's right.

Note that I'm not saying that harmful changes are to be avoided. Can't do that: Some changes are imposed by the world. The key is to keep the good changes and roll back the bad ones before they cause much harm.

4. Be willing to throw code away. In the current buzzword religion this is called fail fast, which sounds nice. The way I phrase it is blunt, not nice, and that's because throwing away code is a blunt thing to do.

Even Microsoft hasn't enough power to force its users, by some counts Windows has lost lost sixty per cent of its market share in the past years. Google is at the peak of its power now, yet G+ is going nowhere.

If something's going to fail, throwing bits of it away earlier can turn the rest into a success, which means that the throwing-away is best done early, by the people who have the code first: The developers. Or if you like pretty words: Only software developers can execute the miracle of transubstantiating code from a failing product into greatness.

Forecasting whether users will reject a particular thing is not easy. Habitual focus on actual users and actual use helps.

3. When something hurts: Stop doing that. Really. And I want to repeat another sentence too: If you stop thinking about the goal you may never get there.