06/15/2022 | Press release | Distributed by Public on 06/15/2022 16:50
We've seen candidates ace A/B testing interviews as much as we've seen candidates answer interview questions in ways that made them look less qualified than they truly were. So we decided to write this piece to help you do it right, whether you are in product management or data science.
If you're the hiring manager, you'll find inspiration for interview questions that'll help you uncover how good the candidate truly is. If you're the interviewee, you'll find what you need to ace the interview.
A/B testing, also called split testing, is a method of determining which of two versions of a product feature, web page, campaign element, or other asset performs better for a certain goal. You can use A/B testing to improve the product development workflow, user interface (UI), or conversion rates.
One classic example is to A/B test a button at the bottom of a screen, with two different colors or verbiage, shown randomly between the treatment and the control group. The A/B test would track which button got more clicks. Another example would be to A/B test two flows in a product where the treatment group sees a different feature than the control group. The goal could be to determine which feature leads to a higher in-app spend for two weeks after using the feature.
Do you need to know more about how they would design a test, or how they would analyze the results? Do you have a lot of data issues to clean up, or do you need someone comfortable with a specific A/B testing software so they can hit the ground running?
Probe about how the candidate makes data-driven decisions, combine and use the data while analyzing, or how they apply intuition. For instance, talk about how the candidate would define a minimum detectable effect (MDE), the smallest effect you could have that's still important for you as a business.
The data scientist on the team will need to be there to ensure that the experiments are set up right. The questions should be about hypothesis testing, confidence intervals, or p-values.
Experimentation and analysis are not one-size-fits-all. Each company has different challenges. Focusing on those specific to your own testing context will help you find candidates who understand your industry. For instance, these specific challenges could be that your company:
Pair methodology questions with business case questions. The cases should be closely related to your own day-to-day. This further helps in the effort to find out how the candidate would be able to apply general A/B testing knowledge on the specific context of your company.
Letting the candidate speak freely while answering the question shows you their thought process. In addition, making the candidate think about what information they're missing is a way to learn more about how they solve problems in the imperfect world. If you mostly asked questions with one correct answer only, you would not learn enough about the interviewee.
Ask questions in the same logical order as the order used in A/B testing. Start with designing the A/B tests before moving into measurement, analyzing results, and making decisions. This approach helps both interviewer and interviewee stay on track when the interview gets longer.
Finding a candidate who can adapt and learn fast will help you when you struggle finding someone who can hit the ground running. With the competitive job market, this is more and more important. So ask about:
Assess your candidates' awareness of how A/B testing evolves for your industry. You'll get to hear their unique insights while also seeing how they connect their work with the bigger picture. This helps uncover the candidate's personality, too. You might even have a conversation that you'll both get passionate about and connect over.
Below is a representative sample of questions we have asked or have been asked in A/B testing interviews. As a digital optimization company, our questions have a slight bias towards the context of using A/B testing to build and grow a product.
You'll find questions and answering tips for both product management and data scientist roles, and questions that assume that the hiring process already verified basic knowledge of A/B testing. You'll also find that the questions apply the tips from the previous section.
You can start your interview by foundational questions about A/B best practices such as:
Questions like these will make a capable candidate warm up and feel more comfortable. The answers should then demonstrate that the candidate has both theoretical and practical knowledge, and confirm that you speak the same language.
The next batch of questions can focus on the tools of trade:
Great answers will demonstrate that the interviewee has first-hand experience with common tools. If they have a clear answer about how they usually learn a new tool, you'll know that they've done it before. Finally, if they ask a question back and wonder about your own tools, that could indicate they are ready to hit the ground running.
Next, consider asking questions about how to deal with problems that come up during experiments:
Answers to these questions should come from essential data science knowledge and demonstrate that the candidate understands concepts such as sample size calculators, test timing, or setting up hypotheses that are clear. The answers should also demonstrate that the candidate thinks about mitigating the problems in a structured and proactive manner.
In the final batch of our questions about experiment design and setup, we recommend that you focus on scenarios common in your own organization. For instance:
Analysis yields conclusions that vary wildly, depending on the thought process. So ask questions that will help you understand the candidate's thought process:
Great answers will showcase the candidate's ability to make decisions about rolling out tests, setting up new treatment and control groups, processing conflicting evidence, or trade-off's between metrics. One viable approach we like to see in interviews is to think about it like finding bugs in engineering, except from a data scientist's perspective.
Our list of recommended questions will conclude with a batch about how the candidate manages the workflow and resources:
The answers to these should prove relevant experience of a product manager, and show that the candidate is organized.
If you've read this far, you're familiar with how to design A/B testing interview questions, and you have a list of sample questions that will give you a head-start. If you also avoid the following mistakes we commonly see in interviews, you'll be on your way to make the hiring or getting hired successful:
Common mistakes by hiring managers when asking the questions:
Common mistakes by candidates when answering the questions:
A new hire or job will empower you to do great work, if you also have the best software. We invite you to keep going and learn how to analyze A/B test results in Amplitude Analytics, or how to run tests in Amplitude Experiment. You can also review our list of 11 top A/B testing tools out today.