Today, I want to respond to a very interesting video posted by YK Sugishita about A/B Testing. Before reading the rest of this blog post, please enjoy this 4 1/2 minute video.
At JC Digital, we are big fans of A/B Testing. But as the video points out, if you don’t have a clear strategy of where you want to be, it can quickly devolve into a cluttered mess of testing inane things.
The problem is you depend on employees of your company, or worse, executives for opinions on what to test next. Sure, employees should have ideas and opinions on how to improve your site. Nothing wrong with that. And dealing with the “Highest Paid Person’s Opinion” (HiPPO) needs be handled gracefully. But as employees, we are way too familiar with our product. We have been drinking our own Koolaid for way too long.
If you are not careful, you end up testing the wrong things to that don’t make up much impact on the overall goal. You waste your precious time. In a humorous way, the video points this out very well.
So after watching this you might want to give up A/B Testing completely….. Don’t do it.
A more systematic approach to A/B testing is necessary.
We recommend these steps:
So now armed with actual meaningful data, you can now make some intelligent decisions on how to conduct your A/B tests.
You might find one of the following issues.
and so on.
These are the types of realistic things you can and should test with A/B testing.
The best part about steps 3 & 4, is you can get the user comments in their own words. The words they use can then be used in the copy (or text) of your A/B Test.
Now getting back to the video, there are certain cases where testing the size or the placement of a button might give you some lift in conversion rates. That might be helpful if you are the size of the Google Search or Amazon store, where millions of users view the page each day. For those companies that have already optimized that funnel so much, then go ahead and test the button size or color. For the rest of us, we have much bigger fish to catch.
In summary, it pays to be intelligent about what to choose next for your A/B test. It’s OK to listen to employees opinions. But way better is to systematically locate your pain points using analytics, then choose your next A/B test by aggregating the feedback of your users. And above all, as the video points out, it’s important to have a goal.
Happy A/B Testing.
Jeff Crawford is a Digital Marketing expert, Engineer and Manager. He has worked for technology companies in Silicon Valley such as Apple, WebTV and Microsoft. He has lived in Tokyo Japan since 2004, working for companies such as Microsoft KK and Adobe Systems Japan. He is the founder of Connect-Local.com, a company that specializes in Local Advertising in the USA. He is also the founder of JC Digital Consulting based in Tokyo Japan. Jeff is also the founder of the Tokyo Digital Marketers Meetup.