A/B testing is a methodology of using randomized experiments with two variants, A and B, which are the Control and Treatment in the controlled experiment. Such experiments are commonly used in web development and marketing, as well as in more traditional forms of advertising. Other names include randomized controlled experiments, online controlled experiments, and split testing. In online settings, such as web design (especially user experience design), the goal is to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. Version A might be the currently used version (Control), while Version B is modified in some respect (Treatment). For instance, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales. Significant improvements can be seen through testing elements like copy text, layouts, images and colors.^[1] Multivariate testing or bucket testing is similar to A/B testing, but tests more than two different versions at the same time.

While the approach is identical to a between-subjects design, which is commonly used in a variety of research traditions, A/B testing is seen as a significant change in philosophy and business strategy in Silicon Valley.^[2][3][4] A/B testing as a philosophy of web development brings the field into line with a broader movement toward evidence-based practice.

An emailing campaign example [edit]

A company with a customer database of 2000 people decides to create an email campaign with a discount code in order to generate sales through its website. It creates an email and then modifies the Call To Action (the part of the copy which encourages customers to do something — in the case of a sales campaign, make a purchase).

• To 1000 people it sends the email with the Call To Action stating "Offer ends this Saturday! Use code A1",
• and to another 1000 people it sends the email with the Call To Action stating "Limited time offer! Use code B1".

All other elements of the email's copy and layout are identical. The company then monitors which campaign has the highest success rate by analysing the use of the promotional codes. The email using the code A1 has a 5% response rate (50 of the 1000 people emailed used the code to buy a product), and the email using the code B1 has a 3% response rate (30 of the recipients used the code to buy a product). The company therefore determines that in this instance, the first Call To Action is more effective and will use it in future sales.

In the example above, the purpose of the test is to determine which is the most effective way to impel customers into making a purchase. If, however, the aim of the test was to see which would generate the highest click-rate – that is, the number of people who actually click onto the website after receiving the email — then the results may have been different.

More of the customers receiving the code B1 may have accessed the website after receiving the email, but because the Call To Action didn't state the end-date of the promotion, there was less incentive for them to make an immediate purchase. If the purpose of the test was simply to see which would bring more traffic to the website, then the email containing code B1 may have been more successful. An A/B test should have a defined outcome that is measurable, e.g. number of sales made, click-rate conversion, number of people signing up/registering etc.^[5]

Companies well-known for using A/B testing [edit]

Many companies use the "designed experiment" approach to making marketing decisions. It is an increasingly common practice as the tools and expertise grows in this area. There are many A/B testing case studies which show that the practice of testing is increasingly becoming popular with small and medium-sized businesses as well.^[6]

While it is widely used behind the scenes to maximize profits, the practice occasionally makes it into the spotlight:

• Amazon.com pioneered its use within the web e-commerce space.^[7]
• BBC^[8]
• eBay^{[citation needed]}
• Google^[9]
• Walmart^[10]
• LogMeIn^[11]
• Microsoft^[12]
• Netflix^[13]
• Playdom (Disney Interactive) ^[14]
• Vegas.com^[15]
• Zynga^[16]

A/B testing tools [edit]

Many A/B testing tools are actively developed. Some are available under an open source license or free

• Google Analytics Content Experiments (formerly Google Website Optimizer) (server-side tagging required)
• Easy Website Optimizer
• ABinator
• Amazon A/B Testing

Other solutions are commercially supported, generally offering a broader range of features:

• Innometrics
• Myna
• GlobalMaxer
• Arise Mobile Framework
• Optimimo Mobile A/B Testing
• Artisan App Testing
• SiteSpect
• Webtrends Optimize
• Optimizely^[17]
• Visual Website Optimizer
• Convert
• Unbounce
• Monetate
• Autonomy Optimost
• Adobe Test & Target
• Personyze
• Metrica Mobile

A detailed overview of such tools is available at WhichMVT.

See also [edit]

• Choice modelling
• Multivariate testing

References [edit]