Numbers Aren't Always Useful

A while back I worked with a client who used a popular service that provides percentage figures for visitation of others' websites. This service contracts with ISPs to get sanitized web visitation figures from its subscribers, some ten million of them last I heard. Then it reports mass figures on who went where. The problem is that it's impossible to find out just what those numbers are - all you get from the service is percentages, which are presumably percentages of the subset of ten million that went to that particular site on any given day. How many is that total? Nobody knows, and the service isn't telling. In my client's case, they were getting figures like .0016%, which is so low that it isn't hardly worth knowing, but they were very keen on it, watching the numbers shift daily from .0016% to .0021%, and cheering lustily at the uptick. Of course, .000016 X 10^6 is only 160 (visits, users? who knows?), and the surge amounted to only 50, a number that didn't make them so cheerful when I pointed it out. And that assumes that the denominator is indeed ten million, which is probably isn't - it's doubtful that all ten million subscribers are online on any given day. The actual improvement might have been as little as 10, or less.

The problem is taking percentages as real numbers. They need to be scrutinized, and you need to know what the denominator is. We called the service to find out just what those percentage figures meant, but they either wouldn't, or couldn't, tell us. The people we talked to were frankly ignorant of simple statistics, calling the ten million subscribers a "sample". I had to tell them that it wasn't a sample, it was a sampling frame, and that without knowing the denominator for the percentage, the tiny percentages the client tracked were all but meaningless. The service had no answer, and the call ended unsatisfactorily. And the client continues using the service to this day, and happily reporting the results to higher-ups.

The service may be worthwhile for larger sites that get a sizable percentage of whatever data points the service is tracking, but any sites smaller than that are probably not getting their money's worth. I used to think that people with marketing degrees would be more conscious of the fallibility of numbers and the need to understand analytics, but I've been wrong more often than right.