As a user experience designer, A/B testing is not only something I design for, but something I advocate that all my clients implement. It’s one of the best ways we can both provide users with the best and most effective experience and provide businesses with ongoing opportunities to optimize.
The reality is that like almost anything in design, some do it really well, while others fail in appalling and reasonably smile-inducing ways. Getting caught doing A/B testing by your most unassuming and uninformed user is like getting caught with your pants down—embarrassing.
One of the great examples of this occurred in 2000 when Amazon customers found themselves paying different prices for the same DVDs. Ouch!
This brings me to eHarmony, an online dating site that advertises some well-known commercials on television. If you’ve watched TV in the past year on a TV chances are you know the Dr. Neil Clark Warren mantra I’m talking about;)
Throughout 2009 I found myself in a position where I experimented with online dating for two reasons:
- the experience of how dating companies message and market to people
- the dates (er, yes, I do mean the actual dates themselves)
Both taught me quite a bit.
Match, Chemistry, and a handful of others never shocked me with their emails (some of the usernames that men choose did). However, eHarmony stood out of the crowd for a few reasons:
- rapid-fire morning messages, usually between 4 and 8 in a row (like I was under fire from the “matchmaking tool”)
- radically different branded email templates
- really bad subject lines, all different, but with the same purpose
It’s the really bad subject lines that I’d like to focus on, and they relate to the rapid-firing in a row, because together they produced an experience that killed the romance.
Here’s a typical morning from 2009:
All of these messages, though different, say the same thing: “Get to know so-and-so.” They would have been more scanable and less distracting if they used the same subject line emphasizing the users’ names.
I never wanted to share these observations with anyone in my profession, because it’s inevitably tied to a much larger issue of my love life, and was ready to let this go on January 4 at the same time as my subscription ran out, except then eHarmony took thing a step further.
Beginning January 4, 2010, my inbox filled with an even more challenging set of differing subject lines. More challenging because the messaging was all over the place. Was I a user more interested in “activities,” “spark,” “unique,” “common,” or greatness? I felt like it was eHarmony having the problem ordering off a menu of men and not me. I wanted simple and given to me straight up.
Here’s a typical morning in 2010 (*pardon that they’re all in my trash now):
Distracting and disruptive, right?
This kind of A/B testing made me feel more like an experiment and less like a client. Moreover, my perception of eHarmony became characterized by the realization that this was not a thoughtful process that took pride in emotionally connecting, but that it was more of a massive warehouse churning out widgets.
A/B Testing Should Not Be:
- Transparent
- Radical
- Isolated
- Hurried
- Fleeting
A/B Testing Should Be:
Subtle—User shouldn’t be aware that they are being tested.
She should feel like she’s experiencing the very best your company has to offer from personalization to copy to look and feel. You’re learning from her actions, so make it impossibly easy for her to accomplish tasks that teach your team without drawing attention to your team.
Incremental—Don’t break things that are currently working well.
Change the text on the button v. changing the button entirely simultaneously altering its shape, color, size, and text.
Aware—Web apps don’t exist alone the world.
News events, seasons, weather, traffic sources, search engine patterns, and even the economy shape the ways in which users discover, interpret, and engage with information.
As you A/B test you should make decisions based on data that’s measured and tracked over time and interpreted with a sensitivity to the world-at-large.
Paced—It’s not a sprint, and when developing apps there is no finish line online.
A/B testing takes time and you need to understand your data in terms of mathematical relevance (e.g. how many unique visitors to sales conversions will it take based on traffic patterns to make a relevant sample size?).
Ongoing—Make user testing an ongoing part of your budget, design, and development.
The best A/B testing occurs when you’re constantly learning from your users. As you grow your user base or expand your offerings new aspects of interaction will be introduced, and as web technologies change users will change predictable patterns.
Both large and small companies can A/B test successfully, but the same rule applies for making mistakes. Executing A/B testing isn’t rocket science, and it’s certainly not supposed to be harder than finding a great love in this life—e’hem eHarmony.