Old Houses and Portals

I live in an older home (1920s) in a historic neighborhood. It's not a particularly wealthy neighborhood. Most historic ones aren't. Money flees its breeding ground. But the neighborhood is comfortable and reasonably vibrant. I always wondered why I loved older homes, and finally one of my gurus, Stewart Brand, might have explained it in his book How Buildings Learn: What Happens After They're Built. He says that older buildings exude what we call "charm" or "character" because they've been altered over time to suit both changing infrastructure needs (the arrival of central heating, air conditioning, indoor plumbing, electricity) and the changing life needs of its occupants (bigger kitchens, more light, more entertainment at home). They grow, morph, and gradually conform more closely to actual human life, like an old pair of jeans. New homes are raw despite their efforts to "design for life". Brand points out that buildings can't be designed up-front for our lifestyles, because no designer can get it right the first time. That's why a home needs so much time to find its proper shape.

What's the lesson for Web designers? Alas, probably not much, despite my most earnest desires to bring the analogy across. The missing element is time. Websites don't give you time. Portals were supposed to let users modify their views quickly, compressing the decades of home conformity into minutes online. Never worked. The vast majority of visitors never knew about customization or took the time to mess with it. Personalization works to an extent, but not completely. Web users are now used to their comfortable sites changing regularly, and although they may not approve, they rarely boycott on that basis.

That said, for years I've been fascinated with the idea of a personalization engine that would track Web user behavior and subtly shift the interface to suit. I've never bothered to fully flesh out the concept, but in general it would work much like Microsoft's failed personalization functionality in Office, the one that gave you chevrons instead of full menus. It was a good idea, but possibly the wrong place to use it. Office users are almost all repeat visitors. Website visitors aren't. Amazon does a good job with personalization, but I'd extend it from "you might also like this stuff" to actually shifting controls and navigational paths. A pipe dream, certainly, but given a huge pile of cash something I'd be interested in researching.

Reading TeaLeaf

Sorry to have been away so long. Complications of various kinds. But now I'm back, and with tea.

Have you seen TeaLeaf? It's a snazzy app that sits athwart your Web traffic, sniffing and recording every user's session. A bit disconcerting, that. But its benefits are undeniable. It stores thirty days (or more, at your discretion) of user transactions, at the user level. It aggregates them too. I've long been a proponent of continual usability checking. Our profession seems to put all its emphasis on initial design and testing, while utterly neglecting Web analytics and other red-flag functionality that can signal usability leaks. Traditional Web analytics is good, but it isn't always granular, meaning that its results are en masse, not at the level of the individual user. It's great for marketing departments, but not as good for usability concerns. TeaLeaf shows the actual user transactions - where people go, what they click, what choices they make, and whether their conversions are successful.

For example, you can lose users at any turn in the road, but especially during checkout. Many visitors drop off when money becomes an issue, and understandably so, since they had no intention of paying anyway; they're just here for the experience, or the knowledge. But others experience technical problems or usability pitfalls. TeaLeaf generates a report on who converted and who didn't, and then you can track out why the failures happened, following every user's trail.

Conversion Rates and The Types of Visitors

Wendy Moe and Peter Fader published a paper in March 2003 titled "Dynamic Conversion Behavior at e-Commerce Sites". In it, they talk (among other things) about the types of visitors to e-commerce sites. This struck me because in analytics we tend to lump all visitors together, just because we can't easily define the segments. Moe and Fader mention the obvious: conversion rates on e-commerce sites are spectacularly low compared with physical stores, below 5% in most cases. Any brick-and-mortar store would have closed up within a week at that rate. They analyze why rates are so low, mostly because so many visitors aren't really immediate buyers. Moe and Fader classify visitors into four groups, only one of which is the get-in-get-out immediate buyer type. Note that I've added notes from my own perspective, so Moe and Fader may take issue with how I'm using their categorization.

• Direct buyers. They come, they choose, they slap down the plastic. They enter knowing what they're looking for. The site can be marginally usable and unattractive, and they'll still probably buy.

• Indirect buyers. They know generally what they want, but they're browsing. Probably will buy, but will take a while and lots of pages. May be influenced somewhat by site characteristics, but not extensively.

• Threshold buyers. These aren't ready to buy, but they're curious and skittish. They're window-shopping. For these visitors, site elements are everything. If the site isn't sticky, they'll leave. Likely influenced by usability and attractiveness. Store impression is as important as product.

• Never-buy visitors. These are seeking knowledge, not product. Not likely to be influenced by site appearance or usability. May look at lots of pages or very few. No intention of buying.
  

Now, these aren't mutually exclusive. I switch modes. I may go to Amazon to order a book I've been wanting, or I may go there just to see what's inside of a particular book that my campus will order.

The relative size of each group has a huge bearing on site owner strategy, but all four groups are typically crammed into the data and dashboard indiscriminately. In statistics, we call this "conflating populations". We see it often in distributions with multiple "hills". And it makes analysis almost a blind operation. For example, if you're seeing a large per-session page view rate, but the conversion rate refuses to rise from 2%, is it because you haven't satisfied the threshold visitors, or because you have too many never-buys, or because the population is mostly indirect buyers? One solution won't hit all of them, so choosing to optimize something on the site may not be the answer. For example, if you streamline the checkout, that may help to capture more of the threshold buyers, but if they're actually only a small percentage of the visitor count, you won't see much of a bump in conversion.

For very large sites, picking the right optimization strategy may make the difference between a huge loss and massive improvement, six figures or more. So how do we segment these populations? Surveys always beckon to us, but I'm skeptical. Surveys online are always self-selected, and self-selection seems to me to invalidate most survey data. There may be clues in the analytics data itself, but I have yet to find a formula. Moe and Fader propose a formula (in fact that's the major purpose of the paper). It would provide a good basis in the real world if only our figures for new and returning visitors were accurate, and they're decidedly not. Several studies have confirmed that cookie-based figures for new visitors can be off by a factor of 2 or more. Unfortunately, if we don't know who's coming to the site, we can't segment them, and without registration we just can't be sure.

Still, keeping these four categories in mind will help when doing site analysis and optimization. If we can make an informed guess about which category of visitor is dominating, we can advise the client accordingly. For example, if we get a lot of visitors to particular pages that have a lot of information, and the pages are obviously being read, and the product is unusual or truly new, then we may have a lot of never-buys. This intuitive approach isn't completely satisfying to me, but it may be all we have.

Scott Adams and the Demise of Common Sense

Scott Adams, the creator of Dilbert, has announced on his blog that he'll be blogging less often. It seems that his original common sensical expectations about how the blog would turn out aren't coming out well at all.

They original expectations included:

1. Advertising dollars
2. Compiling the best posts into a book.
3. Growing the audience for Dilbert
4. Artistic satisfaction.

Of these, only number 4 has worked out. RSS has made visitors go around the ads, the book hasn’t done all that well, and the audience for Dilbert hasn’t been correlated at all with the growth of the blog. As the blog has exploded, the benefits to him haven’t. So he’s talking about blogging less often. It’s a great illustration of how common sense is a lousy predictor of future events. Viva testing and statistics.

Numbers Aren't Always Useful

A while back I worked with a client who used a popular service that provides percentage figures for visitation of others' websites. This service contracts with ISPs to get sanitized web visitation figures from its subscribers, some ten million of them last I heard. Then it reports mass figures on who went where. The problem is that it's impossible to find out just what those numbers are - all you get from the service is percentages, which are presumably percentages of the subset of ten million that went to that particular site on any given day. How many is that total? Nobody knows, and the service isn't telling. In my client's case, they were getting figures like .0016%, which is so low that it isn't hardly worth knowing, but they were very keen on it, watching the numbers shift daily from .0016% to .0021%, and cheering lustily at the uptick. Of course, .000016 X 10^6 is only 160 (visits, users? who knows?), and the surge amounted to only 50, a number that didn't make them so cheerful when I pointed it out. And that assumes that the denominator is indeed ten million, which is probably isn't - it's doubtful that all ten million subscribers are online on any given day. The actual improvement might have been as little as 10, or less.

The problem is taking percentages as real numbers. They need to be scrutinized, and you need to know what the denominator is. We called the service to find out just what those percentage figures meant, but they either wouldn't, or couldn't, tell us. The people we talked to were frankly ignorant of simple statistics, calling the ten million subscribers a "sample". I had to tell them that it wasn't a sample, it was a sampling frame, and that without knowing the denominator for the percentage, the tiny percentages the client tracked were all but meaningless. The service had no answer, and the call ended unsatisfactorily. And the client continues using the service to this day, and happily reporting the results to higher-ups.

The service may be worthwhile for larger sites that get a sizable percentage of whatever data points the service is tracking, but any sites smaller than that are probably not getting their money's worth. I used to think that people with marketing degrees would be more conscious of the fallibility of numbers and the need to understand analytics, but I've been wrong more often than right.

Common Sense Sucks

As HCI practitioners who believe in making technology slicker to use, our biggest opponents may be budgets, but coming up strong on the outside is stupidity based on common sense. Human psychology is a weird and wonderful thing, and I love studying it because there's always something unexpected waiting to mug you around the corner. Where others love bar fights, I love getting pasted by new knowledge. And research has shown for a long time that common sense is a lousy predictor of future events. You'd think that humans would be good at prediction by now, but we're not. For one thing, we think in linear terms - one cause, one effect. But the universe is gloriously nonlinear, and every event has many causes. Further, humans tend to predict based on what's going on now. When we ask "how would you like this?" or "how would you feel about this?", the respondent has to extrapolate based solely on how he feels at the moment. When we put this to the test, we find that the respondents don't really feel that way later on. This is why I put no faith in predictive surveys.

When those in power use common sense to make sweeping laws or spend huge sums of money, they too usually screw the pooch. The Freakonomics blog has a short piece on how abstinence-only sex education has actually resulted in more teen pregnancies, something any psychologist could have predicted. The urge to merge is far too strong to educate away, especially in adolescents who haven't yet developed much control over their impulses. The victims of abstinence-only education aren't given the tools to prevent disease or pregnancy, but they're driven to give in to the mating call anyway, resulting in more pregnancies. It's obvious from studies that abstinence-only ed doesn't work, and New York has appropriately dropped it, despite losing millions in federal funds. But the US Congress stubbornly sticks to the plan. Common sense is dooming teenagers and costing millions, all with no substantial foundation, but that doesn't matter.

The lesson for us is to mistrust common sense, both our own and our clients'. I've seen clients cling to old, unusable designs simply out of faith. Analytics, psychological principles, and user testing will eliminate most of the problems if they're used, but they won't be applied if common sense has anything to say about it.

Why a Messy Room is a Good Thing

I saw a comic the other morning that featured a young man and a thoroughly trashed room, with items strewn all over the place. In doubtless a whiny voice, the youngster pleaded "but when I put things away, I can't find anything". Parents will smile smugly in silent rebuttal, but an HCI'er and part-time economics junkie like me can't help but wonder if those legions of room-messers don't have a point. When we humans do things frequently and in large numbers, there's usually something to it.

Advocates for clean rooms (like rabid inspecting drill sergeants) like to say that everything has its place, and that's where it should be. But is that really true? After you reach a certain threshold of object ownership, is there a single place for everything? Children mostly keep toys in a box, where toys on top obscure the ones underneath. Folded clothing suffers the same fate in a drawer, where you have to pull out the stuff on top to get to the items below.

In effect, it seems to me that a messy room is actually a form of shallow navigation where little is hidden badly enough to be overlooked. The same pertains to a cluttered desk. I often keep a wide, low pile of file folders, papers, notes, books, and pads on half my desk. A neat freak might object that I could just as easily keep all that in their respective drawers and bookcases, but I'm convinced (without much evidence, I have to add) that doing so is less efficient. I can riffle through the pile faster than I can flip through file folders in a file drawer, even in alphabetical order. The pile is for things I'm using frequently at the moment, and overcomes the problem of filing materials under the wrong headings. Our users tend to like shallow navigation, and I'm convinced that I do, too.

Give Me Your Huddled Masses...

When I do usability work, it's astonishing how often a client won't give me access to their user base. They're so used to protecting that base that they can't immediately swing their thinking around to working with that base, rather than just wining and dining them. Sales personnel are the most protective, in my experience. They seem to live in terror that somebody will screw up the precarious relationship they have with their customers. Even when I get to customers, I'm usually given only a small number of very carefully selected and primed representatives, and sometimes not the right people within the customer organizations.

The excuses are legion - customers are too busy, they're too far away, the right people aren't available. But perhaps the most ironic is the excuse that I'll raise customer expectations by letting them think that their wishes will become features in the next release, and if they aren't then the customers will get crabby and disappointed. A lot of salespeople leave me with the impression that their customers are generally vocal and upset about something, interspersed with short interludes of grumpy acceptance. The salesperson doesn't want that sleeping dog disturbed by the slightest zephyr.

But what they can't seem to get straight is that I'm not after user wishes; I'm after user processes. I'm not going to talk much about desires, but about business. I'm trying to understand how customers operate. They'll tell me their desires, of course, but I'm always careful to be noncommittal, with comments like "I'll see what I can do about that in the design, but it may have to be in version two".

Letting users into the planning and design stages is good business, and establishes a level of trust that can't be won any other way. Now if I can just make clients see that...

Another Hideous Example

Perhaps the most fun entries to write are about awful sites, and here's a real winner, courtesy of BoingBoing. The site is by the British government, and it's trying to discourage knife violence. But the site has two major failings: it's all in Flash, and it has enough stupid lawyerly language to discourage a Supreme Court justice from reading the site. The BoingBoing article is here. The awful site is here.

Microsoft Is After Your Brain

New Scientist is reporting a Microsoft patent application for monitoring a test subject with an EEG during usability testing, so that testers can see how people really react, instead of relying on second-hand information like behaviors or think-aloud protocols. Apparently Microsoft believes it has a way to separate out the dozens of noisy signals from the ones they believe relate to user responses. Are all the rest of us out of work now?