ONA Weekly #365: Ask The Right Questions For Your A/B Tests

A/B tests and surveys are extremely useful for collecting user feedback. Many newsrooms use A/B testing to gather data on headlines that work-for the Boston Globe, the removal of the word 'bad' in one headline led to 12% more clicks. The Huffington Post has used A/B testing to determine where to place social share buttons and Vox used A/B testing on their StoryStream feature, which collects articles on a topic, to distinguish read and unread articles as a method of promoting engagement.

There's plenty to test. But first, it's important to make sure you're asking the right questions and optimizing for the right thing. (In other words: it's not always about click-through rate.)

Sophie Ho and Angela Wong, who both worked at the Washington Post, led an ONA17 session about how to make the most of A/B testing:

• Determine the type of data you're looking for: Are you looking for qualitative (comments, etc) or quantitative feedback (time spent, return visits)? What information is actually useful?
• Be able to answer the question of who your ideal audience is, what the existing behavior is and what new behavior you're trying to incentivize.
• Make sure you have a hypothesis and are testing that hypothesis, not just collecting information for the sake of it. Make a prediction and see how the data bears that out.
• Try the 'five whys' method-asking 'why' five times will help drill down to a problem's root cause. For example, 'Why would you want to use email?' and then keep asking 'why?'
• Use Optimizely's sample size calculator to help you figure out how long you'll need to run your test

Dig deeper: For more on the joys of testing, check out How To Grow Your Reader Revenue by Testing Faster and Better: Lessons from the Facebook Accelerator - one-hour recording from ONA20
Featured experts: Duc Luu, Sonya Quick
Takeaway: Case studies from the Facebook Accelerator's Membership community show that small publishers can use nimble testing to achieve steady growth without expensive technology.

We're always on the lookout for helpful resources and tips. If you have other examples to share, please reply directly to this email.