Third-Party Perils

A lot of client sites that I evaluate have tagging problems that aren't really of their own making. We have clients "tag" their sites for analytics purposes to send data back to our mothership, which is then returned to the client as reports. As you undoubtedly know, it's been getting commonplace to "farm out" a certain part of a site to third-party suppliers. Many clients, for example, now out-source their employment pages, with just enough matching page elements to make the visitor think they're still somewhere in the same site. Same thing with newsletters and emails - other people handle it for you. The problem is that those sites usually aren't tagged, so you can't track them. No tracking, no evaluation. Again, small sites aren't deeply affected, but bigger ones are. If you can, work it out with your vendor to let you tag their pages, or have them tag the pages. It's not a new request for most of them. Don't ignore such vital functions as recruitment and marketing contacts.

Engage This!

I'm afraid that I have to take exception to yet another Web buzzword. This time it's "engagement". It's hot right now. Just ask Eric Peterson, who's making a little cottage industry out of his own "Engagement Index". Please. Make it stop. "Engagement" is no better defined than "intelligence", "happiness", or "it sucks".

I'm really a numbers kinda guy, with the heart of a researcher. That means I resist sloppy thinking. And "engagement" is just that, sloppy thoughts. Naming something and believing that you've driven to the heart of it. Peterson's various components may have merit, but he's going about this all the wrong way. Ideally, you study a big group of things and then derive patterns using standard statistical techniques. You don't just wish them into being, no matter how sure you are that they exist. Then you validate your model against a known situation and see if it holds up. If it wavers, fragments, or veers wide of the mark, then your model is faulty.

So far as I can tell, Peterson has never subjected his model to rigorous validation. His various engagement components in the index aren't weighted, so as one rises another could fall, leaving you with the same EI, but with a different situation entirely. I think anybody who relies on a single-measure EI to make expensive business decisions is playing with a loaded gun with the barrel plugged.

That's not to say that "engagement" could never be defined. It can. But it can be defined only as a series of KPIs that shouldn't be arbitrarily added together. A simple radar chart could show them all. So could time series charts. And it should be defined anew for each site. The quest for a standardized index will go on, but in the end I think it's futile. Adopt a Deming approach and keep working on your own special site. I don't think there are any shortcuts.

Reading TeaLeaf

Sorry to have been away so long. Complications of various kinds. But now I'm back, and with tea.

Have you seen TeaLeaf? It's a snazzy app that sits athwart your Web traffic, sniffing and recording every user's session. A bit disconcerting, that. But its benefits are undeniable. It stores thirty days (or more, at your discretion) of user transactions, at the user level. It aggregates them too. I've long been a proponent of continual usability checking. Our profession seems to put all its emphasis on initial design and testing, while utterly neglecting Web analytics and other red-flag functionality that can signal usability leaks. Traditional Web analytics is good, but it isn't always granular, meaning that its results are en masse, not at the level of the individual user. It's great for marketing departments, but not as good for usability concerns. TeaLeaf shows the actual user transactions - where people go, what they click, what choices they make, and whether their conversions are successful.

For example, you can lose users at any turn in the road, but especially during checkout. Many visitors drop off when money becomes an issue, and understandably so, since they had no intention of paying anyway; they're just here for the experience, or the knowledge. But others experience technical problems or usability pitfalls. TeaLeaf generates a report on who converted and who didn't, and then you can track out why the failures happened, following every user's trail.

Conversion Rates and The Types of Visitors

Wendy Moe and Peter Fader published a paper in March 2003 titled "Dynamic Conversion Behavior at e-Commerce Sites". In it, they talk (among other things) about the types of visitors to e-commerce sites. This struck me because in analytics we tend to lump all visitors together, just because we can't easily define the segments. Moe and Fader mention the obvious: conversion rates on e-commerce sites are spectacularly low compared with physical stores, below 5% in most cases. Any brick-and-mortar store would have closed up within a week at that rate. They analyze why rates are so low, mostly because so many visitors aren't really immediate buyers. Moe and Fader classify visitors into four groups, only one of which is the get-in-get-out immediate buyer type. Note that I've added notes from my own perspective, so Moe and Fader may take issue with how I'm using their categorization.

• Direct buyers. They come, they choose, they slap down the plastic. They enter knowing what they're looking for. The site can be marginally usable and unattractive, and they'll still probably buy.

• Indirect buyers. They know generally what they want, but they're browsing. Probably will buy, but will take a while and lots of pages. May be influenced somewhat by site characteristics, but not extensively.

• Threshold buyers. These aren't ready to buy, but they're curious and skittish. They're window-shopping. For these visitors, site elements are everything. If the site isn't sticky, they'll leave. Likely influenced by usability and attractiveness. Store impression is as important as product.

• Never-buy visitors. These are seeking knowledge, not product. Not likely to be influenced by site appearance or usability. May look at lots of pages or very few. No intention of buying.
  

Now, these aren't mutually exclusive. I switch modes. I may go to Amazon to order a book I've been wanting, or I may go there just to see what's inside of a particular book that my campus will order.

The relative size of each group has a huge bearing on site owner strategy, but all four groups are typically crammed into the data and dashboard indiscriminately. In statistics, we call this "conflating populations". We see it often in distributions with multiple "hills". And it makes analysis almost a blind operation. For example, if you're seeing a large per-session page view rate, but the conversion rate refuses to rise from 2%, is it because you haven't satisfied the threshold visitors, or because you have too many never-buys, or because the population is mostly indirect buyers? One solution won't hit all of them, so choosing to optimize something on the site may not be the answer. For example, if you streamline the checkout, that may help to capture more of the threshold buyers, but if they're actually only a small percentage of the visitor count, you won't see much of a bump in conversion.

For very large sites, picking the right optimization strategy may make the difference between a huge loss and massive improvement, six figures or more. So how do we segment these populations? Surveys always beckon to us, but I'm skeptical. Surveys online are always self-selected, and self-selection seems to me to invalidate most survey data. There may be clues in the analytics data itself, but I have yet to find a formula. Moe and Fader propose a formula (in fact that's the major purpose of the paper). It would provide a good basis in the real world if only our figures for new and returning visitors were accurate, and they're decidedly not. Several studies have confirmed that cookie-based figures for new visitors can be off by a factor of 2 or more. Unfortunately, if we don't know who's coming to the site, we can't segment them, and without registration we just can't be sure.

Still, keeping these four categories in mind will help when doing site analysis and optimization. If we can make an informed guess about which category of visitor is dominating, we can advise the client accordingly. For example, if we get a lot of visitors to particular pages that have a lot of information, and the pages are obviously being read, and the product is unusual or truly new, then we may have a lot of never-buys. This intuitive approach isn't completely satisfying to me, but it may be all we have.

Scott Adams and the Demise of Common Sense

Scott Adams, the creator of Dilbert, has announced on his blog that he'll be blogging less often. It seems that his original common sensical expectations about how the blog would turn out aren't coming out well at all.

They original expectations included:

1. Advertising dollars
2. Compiling the best posts into a book.
3. Growing the audience for Dilbert
4. Artistic satisfaction.

Of these, only number 4 has worked out. RSS has made visitors go around the ads, the book hasn’t done all that well, and the audience for Dilbert hasn’t been correlated at all with the growth of the blog. As the blog has exploded, the benefits to him haven’t. So he’s talking about blogging less often. It’s a great illustration of how common sense is a lousy predictor of future events. Viva testing and statistics.

File Sharing and Offshoring

I see double standards everywhere, I guess. It's not so much a moral failing as a human condition. Take file sharing and offshoring. Proponents of free file sharing like Slashdot and BoingBoing speak for a huge number of users and techies who dismiss the entertainment industry's hissy fits over the practice with replies like "Get used to it", "Globalization has made your old business model obsolete", and "File sharing opens up the market with more diversity".

My personal take on file sharing is that is indeed a new game that threatens the business model of the choke point that entertainment companies have profited from for more than a century. It's never safe to dismiss the creative energies of millions of users who want to circumvent the old restrictions on their pleasure. Getting around "The Man" is also more a human condition than a moral failing. The simple fact for the entertainment moguls is that file sharing exists and can't be effectively stopped, so they will indeed have to learn how to live with it. Further, file sharing has begun to live up to its potential as a redistributor of talent, a true exercise in globally democratic artistry.

But then there's B side of the technological album - offshoring. The same globalization and foreign talent development that opened up music and movies has also enabled engineers and developers in Eastern Europe, Asia, and other places to cater to the markets of America. I've heard the anecdotes about how foreign code is often bug-ridden and flaky, but so is much of American code. Further, the quality of the code continues to rise as foreign programmers become university-trained. Only a fraction of the toys from China are lead-coated, and only a fraction of the code from India is trash. We can't confuse media hype with reality.

I have to admit to being conflicted about offshoring. As jobs drift away from Americans, even in small percentages, the net effect is to make talented high schools pause at the door to computer science and engineering. Both the now-historical dot-bomb and current tales of offshoring have combined to devastate computer-related programs in higher education all over the country. We're losing a generation. Offshoring itself doesn't scare me, but its reputation does.

On the other hand, the egalitarian meritocrat within me can't help but marvel at what the other nations on Earth can accomplish with some money and markets. A recent review of a Korean Kia model, for example, compared it favorably with a Lexus, and for half the price. India's Bollywood is now producing movies that can compete in quality with many indies in America. "Dil Se", a Bollywood film that broke into the UK top ten, features an energetic crowd dance on the top of a moving train. Why not put more money and projects into these people's hands? If technology is an unstoppable force for globalization, so is commerce. If we can't stop offshoring because it works, we have to change our business model, don't we? We need to stop bemoaning it and get on with doing whatever we find we can do best.