Third-Party Perils

A lot of client sites that I evaluate have tagging problems that aren't really of their own making. We have clients "tag" their sites for analytics purposes to send data back to our mothership, which is then returned to the client as reports. As you undoubtedly know, it's been getting commonplace to "farm out" a certain part of a site to third-party suppliers. Many clients, for example, now out-source their employment pages, with just enough matching page elements to make the visitor think they're still somewhere in the same site. Same thing with newsletters and emails - other people handle it for you. The problem is that those sites usually aren't tagged, so you can't track them. No tracking, no evaluation. Again, small sites aren't deeply affected, but bigger ones are. If you can, work it out with your vendor to let you tag their pages, or have them tag the pages. It's not a new request for most of them. Don't ignore such vital functions as recruitment and marketing contacts.

Engage This!

I'm afraid that I have to take exception to yet another Web buzzword. This time it's "engagement". It's hot right now. Just ask Eric Peterson, who's making a little cottage industry out of his own "Engagement Index". Please. Make it stop. "Engagement" is no better defined than "intelligence", "happiness", or "it sucks".

I'm really a numbers kinda guy, with the heart of a researcher. That means I resist sloppy thinking. And "engagement" is just that, sloppy thoughts. Naming something and believing that you've driven to the heart of it. Peterson's various components may have merit, but he's going about this all the wrong way. Ideally, you study a big group of things and then derive patterns using standard statistical techniques. You don't just wish them into being, no matter how sure you are that they exist. Then you validate your model against a known situation and see if it holds up. If it wavers, fragments, or veers wide of the mark, then your model is faulty.

So far as I can tell, Peterson has never subjected his model to rigorous validation. His various engagement components in the index aren't weighted, so as one rises another could fall, leaving you with the same EI, but with a different situation entirely. I think anybody who relies on a single-measure EI to make expensive business decisions is playing with a loaded gun with the barrel plugged.

That's not to say that "engagement" could never be defined. It can. But it can be defined only as a series of KPIs that shouldn't be arbitrarily added together. A simple radar chart could show them all. So could time series charts. And it should be defined anew for each site. The quest for a standardized index will go on, but in the end I think it's futile. Adopt a Deming approach and keep working on your own special site. I don't think there are any shortcuts.

Reading TeaLeaf

Sorry to have been away so long. Complications of various kinds. But now I'm back, and with tea.

Have you seen TeaLeaf? It's a snazzy app that sits athwart your Web traffic, sniffing and recording every user's session. A bit disconcerting, that. But its benefits are undeniable. It stores thirty days (or more, at your discretion) of user transactions, at the user level. It aggregates them too. I've long been a proponent of continual usability checking. Our profession seems to put all its emphasis on initial design and testing, while utterly neglecting Web analytics and other red-flag functionality that can signal usability leaks. Traditional Web analytics is good, but it isn't always granular, meaning that its results are en masse, not at the level of the individual user. It's great for marketing departments, but not as good for usability concerns. TeaLeaf shows the actual user transactions - where people go, what they click, what choices they make, and whether their conversions are successful.

For example, you can lose users at any turn in the road, but especially during checkout. Many visitors drop off when money becomes an issue, and understandably so, since they had no intention of paying anyway; they're just here for the experience, or the knowledge. But others experience technical problems or usability pitfalls. TeaLeaf generates a report on who converted and who didn't, and then you can track out why the failures happened, following every user's trail.

Conversion Rates and The Types of Visitors

Wendy Moe and Peter Fader published a paper in March 2003 titled "Dynamic Conversion Behavior at e-Commerce Sites". In it, they talk (among other things) about the types of visitors to e-commerce sites. This struck me because in analytics we tend to lump all visitors together, just because we can't easily define the segments. Moe and Fader mention the obvious: conversion rates on e-commerce sites are spectacularly low compared with physical stores, below 5% in most cases. Any brick-and-mortar store would have closed up within a week at that rate. They analyze why rates are so low, mostly because so many visitors aren't really immediate buyers. Moe and Fader classify visitors into four groups, only one of which is the get-in-get-out immediate buyer type. Note that I've added notes from my own perspective, so Moe and Fader may take issue with how I'm using their categorization.

• Direct buyers. They come, they choose, they slap down the plastic. They enter knowing what they're looking for. The site can be marginally usable and unattractive, and they'll still probably buy.

• Indirect buyers. They know generally what they want, but they're browsing. Probably will buy, but will take a while and lots of pages. May be influenced somewhat by site characteristics, but not extensively.

• Threshold buyers. These aren't ready to buy, but they're curious and skittish. They're window-shopping. For these visitors, site elements are everything. If the site isn't sticky, they'll leave. Likely influenced by usability and attractiveness. Store impression is as important as product.

• Never-buy visitors. These are seeking knowledge, not product. Not likely to be influenced by site appearance or usability. May look at lots of pages or very few. No intention of buying.
  

Now, these aren't mutually exclusive. I switch modes. I may go to Amazon to order a book I've been wanting, or I may go there just to see what's inside of a particular book that my campus will order.

The relative size of each group has a huge bearing on site owner strategy, but all four groups are typically crammed into the data and dashboard indiscriminately. In statistics, we call this "conflating populations". We see it often in distributions with multiple "hills". And it makes analysis almost a blind operation. For example, if you're seeing a large per-session page view rate, but the conversion rate refuses to rise from 2%, is it because you haven't satisfied the threshold visitors, or because you have too many never-buys, or because the population is mostly indirect buyers? One solution won't hit all of them, so choosing to optimize something on the site may not be the answer. For example, if you streamline the checkout, that may help to capture more of the threshold buyers, but if they're actually only a small percentage of the visitor count, you won't see much of a bump in conversion.

For very large sites, picking the right optimization strategy may make the difference between a huge loss and massive improvement, six figures or more. So how do we segment these populations? Surveys always beckon to us, but I'm skeptical. Surveys online are always self-selected, and self-selection seems to me to invalidate most survey data. There may be clues in the analytics data itself, but I have yet to find a formula. Moe and Fader propose a formula (in fact that's the major purpose of the paper). It would provide a good basis in the real world if only our figures for new and returning visitors were accurate, and they're decidedly not. Several studies have confirmed that cookie-based figures for new visitors can be off by a factor of 2 or more. Unfortunately, if we don't know who's coming to the site, we can't segment them, and without registration we just can't be sure.

Still, keeping these four categories in mind will help when doing site analysis and optimization. If we can make an informed guess about which category of visitor is dominating, we can advise the client accordingly. For example, if we get a lot of visitors to particular pages that have a lot of information, and the pages are obviously being read, and the product is unusual or truly new, then we may have a lot of never-buys. This intuitive approach isn't completely satisfying to me, but it may be all we have.

Scott Adams and the Demise of Common Sense

Scott Adams, the creator of Dilbert, has announced on his blog that he'll be blogging less often. It seems that his original common sensical expectations about how the blog would turn out aren't coming out well at all.

They original expectations included:

1. Advertising dollars
2. Compiling the best posts into a book.
3. Growing the audience for Dilbert
4. Artistic satisfaction.

Of these, only number 4 has worked out. RSS has made visitors go around the ads, the book hasn’t done all that well, and the audience for Dilbert hasn’t been correlated at all with the growth of the blog. As the blog has exploded, the benefits to him haven’t. So he’s talking about blogging less often. It’s a great illustration of how common sense is a lousy predictor of future events. Viva testing and statistics.

Numbers Aren't Always Useful

A while back I worked with a client who used a popular service that provides percentage figures for visitation of others' websites. This service contracts with ISPs to get sanitized web visitation figures from its subscribers, some ten million of them last I heard. Then it reports mass figures on who went where. The problem is that it's impossible to find out just what those numbers are - all you get from the service is percentages, which are presumably percentages of the subset of ten million that went to that particular site on any given day. How many is that total? Nobody knows, and the service isn't telling. In my client's case, they were getting figures like .0016%, which is so low that it isn't hardly worth knowing, but they were very keen on it, watching the numbers shift daily from .0016% to .0021%, and cheering lustily at the uptick. Of course, .000016 X 10^6 is only 160 (visits, users? who knows?), and the surge amounted to only 50, a number that didn't make them so cheerful when I pointed it out. And that assumes that the denominator is indeed ten million, which is probably isn't - it's doubtful that all ten million subscribers are online on any given day. The actual improvement might have been as little as 10, or less.

The problem is taking percentages as real numbers. They need to be scrutinized, and you need to know what the denominator is. We called the service to find out just what those percentage figures meant, but they either wouldn't, or couldn't, tell us. The people we talked to were frankly ignorant of simple statistics, calling the ten million subscribers a "sample". I had to tell them that it wasn't a sample, it was a sampling frame, and that without knowing the denominator for the percentage, the tiny percentages the client tracked were all but meaningless. The service had no answer, and the call ended unsatisfactorily. And the client continues using the service to this day, and happily reporting the results to higher-ups.

The service may be worthwhile for larger sites that get a sizable percentage of whatever data points the service is tracking, but any sites smaller than that are probably not getting their money's worth. I used to think that people with marketing degrees would be more conscious of the fallibility of numbers and the need to understand analytics, but I've been wrong more often than right.

Web Analytics

One of the things I wish my colleagues knew more about was Web analytics. We've ceded this important facet of usability to the marketing folks, and we need to get a piece of it back.

Web analytics is the ability to get constant data on where users have been in your website, what they looked at, where they came from, what they bought, how long they lingered, and so forth. Much of this stuff used to be in log files, but they proved to be too feeble for prediction. Current analytics packages have much more functionality than log files ever did. For example, in WebTrends I can see what proportion of users went in one of several directions from the main page, and where they went after that, and after that. I can see what keywords they used in searches, both on the site and in search engine queries to find the site.

Web analytics has become the province of marketing departments everywhere, but often they don't know how to use the data for usability or IA purposes. We do, but most of us never look at the data, or don't know what to do with it when we do. When you can see clickpaths through your site, you can begin to optimize your navigational structure. You can gradually make the site better. If you're using a WA package now, I'd suggest you start mining it for every bit of gold you can find.