Arnt Gulbrandsen
About meAbout this blog
2015-06-24

IP geolocation using Maxmind and Digital Envoy

Back in 2013 I set up customer geolocation for a now-defunct video streaming service. The company needed to stream things only to the countries for which it held content rights, and to choose the right UI language and content catalog by default. The database was also used for smaller chores, like suggesting the right country in forms.

My task was to make sure we picked the right countries with few enough errors that the content rights owners were happy, and that we did it quickly and reliably.

Two vendors offered the right kind of geoip database: Maxmind and Digital Envoy. I first spoke to Maxmind, who handled me well. It took a couple of days to get answers from them, in one case more than a week, but my questions were competently answered. For instance, I had a problem where lookup of one particular IP address crashed, and Maxmind did find the reason: Maxmind's library code didn't handle countries that were newer than the code itself, and that address was in South Sudan, proudly independent since 2011. An upgrade solved the problem, but more importantly, Maxmind's support passed my test.

I first checked Maxmind's geoip database against about 80,000 IP addresses in our target markets and found it acceptable. I checked the database using ping time (packets cannot go faster than light, which provides solid geographical proof), knowledge of router naming schemes and IP exchange locations, and of course whois registrations. Fewer than 100 IP addresses needed more manual work such as looking at the customer's billing address. Digital Envoy cold-called us a little later, so I went on to check Digital Envoy's database against a small subset of the 80,000, including many of the addresses for which Maxmind was wrong or not precise enough. Then I compared all three results sets (mine, DE's and Maxmind's).

The Maxmind and DE answers were not 100% comparable, because their categories didn't match up. One had categories like Tor/VPN user and somewhere in Europe, not sure which country that the other did not. Based on my set of IP addresses and using only my requirements to decide whether each difference counted as an error against each vendor, I counted both as more than 99% accurate for the purpose described above, and Digital Envoy a smidgen better than Maxmind.

The errors were mostly acceptable, ie. the results were good enough that the rights owners wouldn't yell at us. One error I remember was a consulting company that used the same IP subnet for several small offices in neighbouring countries. Another was a multinational ISP that had recently reallocated an address block to a different country.

I chose Maxmind even though Digital Envoy had slightly better geolocation data. Maxmind gave me solid data and solid answers, Digital Envoy gave me solid data but manure for answers. Their salesman claimed variously 99.9 and 99.99% accuracy (what's an extra nine among friends?) and said things about Maxmind that I knew to be false.

Then he gave a PDF to help inform me, supposedly a more thorough comparison than what I had done. I had to sign an NDA, but that NDA has expired now, so here's the PDF Digital Envoy gave me. I was quite shocked by its cavalier inaccuracies or even lies.

I measured roughly the same database accuracy, DE claims to be 500 times better than Maxmind. I counted six Maxmind committers just on the code I saw, DE claims there are only a couple of developers in total. I received updates from Maxmind every Tuesday and noticed changes from the past week in an update, DE claims Maxmind publishes stale data once per month. As to accuracy, various addresses in my sample set were correctly located at village level by Maxmind, even though I could find no public data source to confirm that. I don't know what sources Maxmind uses, but whatever they are, they're better than DE claims.

And then there's support... I suppose I should have sent DE a question. I had one, a tricky tricky question, but I had already decided against DE because of their errors and exaggerated promises.

According to the PDF's metadata, it was written by Digital Envoy's VP of Operations and was five years old when Digital Envoy gave it to me. Quite an age for a document that brags about constantly updating data.

It's difficult to believe, but the DE salesman gave me this PDF after I had told him that I had measured Maxmind's performance, and offered no explanation for the discrepancies. Did he expect me to trust this FUD rather than my own measurements? Or to trust DE to be accurate after having seen all these inaccuracies?