William Reynolds, User Engagement Associate @ Peace, Love & Firefox Event #SFPride
Photo by Melissa Shapiro @shappy, Mozilla Firefox PR & so much more!

Spin the Wheel!

Sometimes user experience design is the result of taking chances and creating experiences that are unexpected, catching users’ attention and hopefully not causing an annoyance, what follows is one such instance!

Looking for Low-Hanging Fruit

My team and I do a lot of work on fine-tuning user experiences around the Firefox download funnel. Sometimes we do big redesign projects, starting from scratch to test out new waters, but often we look for low-hanging fruit, making the tiniest tweaks to learn (and share) what gets big results. There’s magic in making small adjustments and testing out the simplest thing that could possibly work.

In this case, we were working on ways to encourage Firefox users using older versions to download the latest version. Our goals:

• increase downloads of the latest version of FX
• target users of 3.6 and below (let’s get these users onto a modern & safe web!)

We want to provide users with the best, latest and greatest that the web has to offer, but more importantly, there are also security vulnerabilities that we want our users to be protected from and increasing issues around privacy that we want our users to benefit from too. It’s about helping out our users and not being overly intrusive in the process.

2 Weeks Post-Tweak

We recently released an upgrade campaign with a persistent message (here’s an example of a persistent message on support.mozilla.com) that appears only to users on FX 3.6 and below. We A/B test everything possible, so we released two versions of this messaging:

• Version A, appears to the user slowly dropping down from the top of the screen
• Version B, appears to the user slowing rising up from the bottom of the screen

Version A. Top

Version B. Bottom

Results

Results from the first 12 days of the campaign:

• Total # of Visits, 83,797
• Total # of Page Views, 203, 269
• Total # of Downloads, 62,906

• Version A. Upgrade Top, 26,262 downloads, converting at 76.36%
• Version B. Upgrade Bottom, 36,644 downloads, converting at 74.17%

Next Steps

We’ll be consulting with our metrics team to determine which experience wins based on statistical relevance and then we’ll probably test out an alternate design (learning what is most effective, from color to iconography), and then we’ll hone in on the nuances of the copy.

Credits: Steven Garrity (silverorange), Chris Howse (@no_wait_yes), Laura Forrest (@lauraforrest) & Mozilla Firefox Product Website Team

Moar!

Read Laura Forrest’s post on increasing newsletter conversions

Share on Facebook

Test 1:

A: Header, Mozilla Tab, & Full Navigation

v.

B: No Header, No Tab, No Navigation (WINNER)

Results: New version “B” = 2.24% conversion increase with 99.47% statistical significance. Bounce rate is down 2.25%.

(Note: a 2% increase is a HUGE win! but, wait, it gets better…)

Test 2:

A: No Header, No Tab, Reduced Footer Copy, Plus Level 1 Navigation at the Bottom of the Page

v.

B: No Header, No Tab, No Promo Space, Plus Level 1 Navigation at the Bottom of the Page

>>> added edits<<<

Results: After 24k assignments, download conversions are way up. Currently, the “winning” variation improves download conversions by 6.7%! Bounce rates are down across the board as well :)

Share on Facebook

Some user behaviors make clear sense, while others can confound your expectations. Overall, however, designing a complete experience around a conversion page is best understood taking these 3 “Ss” into consideration:

1. Somewhere—Users get to conversion pages from somewhere, so go there first!
2. Superficial—Good looks matter. Users respond to a particular aesthetic, so try different designs!
3. So Much More—Users engage deeper through community, so get them connected to you and each other! (FYI a Facebook Fan Page is a great ecosystem & it’s free)

Okay, so few weeks ago I posted about a redesign I’m working on for a lead generation page. That project began with a single page, but preliminary outcomes further demonstrated that we needed to step backwards and design for an entire experience, which is where Somewhere, Superficial, and So Much More came into play.

I’ll explain!

First, our initial A/B test is over:

• Good news—we didn’t break the existing conversion rate.
• Better news—we raised the conversion rate a little bit ;-)
• Great news—we blew up our Facebook Fan page by designing a thoughtful “Thank You”! (*We increased fan acquisition from 3 fans per day to 30+ fans per day)

Second, we’re embarking on Phase 2. of our test plan, so let me share what that plan looks like, because A/B testing can be delivered in a number of ways. Here is the 3-phase method I recommend and that we’re currently using:

Phase 1.) Update, Don’t Break

(This phase is optional, in the project I’m working on we did need to update an older design before we could effectively move forward with testing.)

• Update the design to a higher standard that better expresses the brand
• Don’t break existing conversion (50 v. 50 split test, confirmed by a 5 v. 95 follow-up)
• Phase original design out completely

(Current design in the wild, original design)

Phase 2.) Design Different Concepts

• Create 2 more well-informed aesthetically different designs
• Conduct user testing (DIY-style works well!)
• Release new designs into testing cycle in increments (10 v. 90% to start)

(Potential design to test)

Phase 3.) Optimize One

• Select the “winning” design from the 3 that have been tested
• Optimize the winning design with A/B testing of the smaller elements on page (copy, image, steps, etc)

(Potential design to test)

Third, as we prepare for our next test, which will involve 3 very different designs, here are things we considered about the current experience to make sure we covered “Somewhere, Superficial, and So Much More”:

Traffic Sources—

It’s important to learn about where your users are coming from; this is the first step to take in order to reduce your bounce rate.

Are your users coming to your site after clicking an ad? Are users coming to your site to learn something, or have they accidentally gotten there, is the message seductive, on target?

User Testing—

User testing doesn’t have to be expensive, drawn out, or complicated. Do informal (or formal if you can) user testing to gather feedback on the different designs before you release them into the wild.

Form Friction—

At the heart of every conversion page is a form. Determine what variables can be tested, and what variable make sense to test. Some form friction is good!

Let me share an example, for us, it seems like “phone number” is an easy variable to test out. But, it turns out it’s not! While we might get more conversions by taking that element out of the form, we would also open ourselves up to less qualified leads, which we don’t want and also aren’t equipped to scale for. (*Consider your community support team, if you put their phone number on your conversion page can they handle the incoming calls?)

The Thank You—

Never underestimate the opportunity for a second level of conversion that your “Thank You” messaging provides for you to create and facilitate community and further engagement.

Our Thank You experience in our first round of testing, totally changed our approach to how we were looking at this conversion page.

To wrap it all up, throughout this process of redesigning and optimizing a single conversion page (we’ve planned on about 3 months of testing), we’ve really come to understand that the best and most effective lead gen page designs involve an entire experience that goes well beyond a single page. Suddenly, our little conversion page isn’t so lonely or so little.

Take a look at your app’s conversion page, rally your team, and consider the impact of Somewhere, Superficial, and So Much More.

Share on Facebook

As a user experience designer, A/B testing is not only something I design for, but something I advocate that all my clients implement. It’s one of the best ways we can both provide users with the best and most effective experience and provide businesses with ongoing opportunities to optimize.

The reality is that like almost anything in design, some do it really well, while others fail in appalling and reasonably smile-inducing ways. Getting caught doing A/B testing by your most unassuming and uninformed user is like getting caught with your pants down—embarrassing.

One of the great examples of this occurred in 2000 when Amazon customers found themselves paying different prices for the same DVDs. Ouch!

This brings me to eHarmony, an online dating site that advertises some well-known commercials on television. If you’ve watched TV in the past year on a TV chances are you know the Dr. Neil Clark Warren mantra I’m talking about;)

Throughout 2009 I found myself in a position where I experimented with online dating for two reasons:

• the experience of how dating companies message and market to people
• the dates (er, yes, I do mean the actual dates themselves)
  

Both taught me quite a bit.

Match, Chemistry, and a handful of others never shocked me with their emails (some of the usernames that men choose did). However, eHarmony stood out of the crowd for a few reasons:

• rapid-fire morning messages, usually between 4 and 8 in a row (like I was under fire from the “matchmaking tool”)
• radically different branded email templates
• really bad subject lines, all different, but with the same purpose

It’s the really bad subject lines that I’d like to focus on, and they relate to the rapid-firing in a row, because together they produced an experience that killed the romance.

Here’s a typical morning from 2009:

All of these messages, though different, say the same thing: “Get to know so-and-so.” They would have been more scanable and less distracting if they used the same subject line emphasizing the users’ names.

I never wanted to share these observations with anyone in my profession, because it’s inevitably tied to a much larger issue of my love life, and was ready to let this go on January 4 at the same time as my subscription ran out, except then eHarmony took thing a step further.

Beginning January 4, 2010, my inbox filled with an even more challenging set of differing subject lines. More challenging because the messaging was all over the place. Was I a user more interested in “activities,” “spark,” “unique,” “common,” or greatness? I felt like it was eHarmony having the problem ordering off a menu of men and not me. I wanted simple and given to me straight up.

Here’s a typical morning in 2010 (*pardon that they’re all in my trash now):

Distracting and disruptive, right?

This kind of A/B testing made me feel more like an experiment and less like a client. Moreover, my perception of eHarmony became characterized by the realization that this was not a thoughtful process that took pride in emotionally connecting, but that it was more of a massive warehouse churning out widgets.

A/B Testing Should Not Be:

• Transparent
• Radical
• Isolated
• Hurried
• Fleeting

A/B Testing Should Be:

Subtle—User shouldn’t be aware that they are being tested.

She should feel like she’s experiencing the very best your company has to offer from personalization to copy to look and feel. You’re learning from her actions, so make it impossibly easy for her to accomplish tasks that teach your team without drawing attention to your team.

Incremental—Don’t break things that are currently working well.

Change the text on the button v. changing the button entirely simultaneously altering its shape, color, size, and text.

Aware—Web apps don’t exist alone the world.

News events, seasons, weather, traffic sources, search engine patterns, and even the economy shape the ways in which users discover, interpret, and engage with information.

As you A/B test you should make decisions based on data that’s measured and tracked over time and interpreted with a sensitivity to the world-at-large.

Paced—It’s not a sprint, and when developing apps there is no finish line online.

A/B testing takes time and you need to understand your data in terms of mathematical relevance (e.g. how many unique visitors to sales conversions will it take based on traffic patterns to make a relevant sample size?).

Ongoing—Make user testing an ongoing part of your budget, design, and development.

The best A/B testing occurs when you’re constantly learning from your users. As you grow your user base or expand your offerings new aspects of interaction will be introduced, and as web technologies change users will change predictable patterns.

Both large and small companies can A/B test successfully, but the same rule applies for making mistakes. Executing A/B testing isn’t rocket science, and it’s certainly not supposed to be harder than finding a great love in this life—e’hem eHarmony.

Share on Facebook