A/B Testing
What is it? An experiment where two versions, A and B, of a creative treatment (e.g. email, concept, web page or app) are shown to different users randomly, and then statistically analyzed (compared) to determine which version performs better.
When is it best used? When there is more than one option to test - usually toward the end of the development cycle, to determine which of the options is most likely to have the greatest success (however success is determined for that particular initiative.) It can be used independently, or as a series of tests over time, to inform development/design decisions.
What does it entail? Version A and Version B are tested randomly with a live audience. A and B are shown to the same number of (different) respondents in the test, with the intent to find out which performs most strongly against the goal of the study. Analysis of output is used to determine whether A or B performs better, (for a website, this might be conversion rate or bounce rate, for a concept, it might be appeal/relevance scores.) When there are more than two versions to be tested, it is called a “multivariate test” rather than an A/B test.
Interchangeable terms: Split testing
Use in a sentence: Good marketers A/B test their ad copy to learn which version attracts more clicks.
Related Terms: analytics, bucket testing, conversion rate, design, multivariate testing, user behavior, multivariate test
Visual: Yes
Additional Resources
Books
Articles
Videos
AB Testing At Pinterest: Building A Culture Of Experimentation from HyperightData Talks
Design Testing at Netflix from Netflix
AB Testing Intro/ Why, What, Where, & How to AB Test from Testing Theory
Share this page
