A/B Testing Quick-Start Guide, Part 1: What is A/B Testing?

Business.com / Marketing Strategy / Last Modified: February 22, 2017

An overview of key metrics that can be improved, the types of tests available to you, and discuss how to select pages and page elements...

As a small business, you have a ton of competing priorities and finding time for A/B testing and optimization of your website can be easy to put off, especially if you haven't done it before. Small changes to key pages on your site, however, can have a truly significantly impact on your business, and A/B testing is a worthwhile use of your time. In part one of this two-part article, I'll provide an overview of key metrics that can be improved, the types of tests available to you, and discuss how to select pages and page elements for testing. In part two, I'll lay out software, tools, and processes that will allow you to get started with A/B testing with a minimal investment of time and expense.

Related: Considerations your business must take when using Javascript.

Why Test?

Small increases in conversion rates on key pages can make a huge difference to your marketing efforts' ROI. Improving ROI will, in turn, allow you to increase your marketing spend, visitors, and revenue. Even a relatively modest 5- 10% lift in a page's conversion rate can result in thousands of dollars of incremental revenue. The good news is that, if you haven't optimized your pages before, it's very likely that you will be able to achieve a sizable lift.

Sites can be tested and optimized for a variety of reasons. The most common reason for testing is to increase the number of visitors who fill out a lead form or contact us form. Other common testing goals are clicks on a button or link, increased user engagement, or reduced bounce rate on a page. Whatever your goal, optimization software can be set up to tests and track improvements for nearly any metric.

A/B vs. Multivariate Testing

The most common test methodologies are A/B testing and multivariate testing.

A/B testing, also known as split testing, involves setting up two versions of a page: the "A," or control, version, and the "B," or test, version. Once the test is live, 50% of visitors are shown the A version of the page, while the other 50% are shown the B version. If the B version outperforms the A version, the test is a winner. You can then set up a new test with the winner as the new A version, and a completely new variation as the B version. (Note: you can set up as many variations as you like, e.g., an A/B/C/D test.)

  • Pros: Simpler, faster results, easier to interpret results, arguably more satisfying
  • Cons: Iterative, tendency to not test smaller elements that may actually make an impact

Multivariate testing involves the creation of multiple variations of one or more elements on a page and then using an algorithm to test all combinations of the variations. The theory is that non-obvious combinations of elements could lift performance, whereas variations of each element on their own could have no effect. For example a button that says "sign up now" may perform better when the accompanying image on the page is a picture of a person, but may underperform when the image shows your product.

  • Pros: non-obvious combinations of elements cannot be accomplished with A/B testing
  • Cons: longer test cycle, results can be difficult to interpret

Related: Get help making your site more dynamic from an online marketing services provider.

What to Test?

The first step in setting up your first test is identifying the page or pages on your site that have the highest potential to impact the metric you are trying to improve. For example, if you are trying to increase the number of visitors that fill out your "contact us" form, select a page or pages with enough traffic to reach statistical significance (see explanation below) and run your test there. Many times, it is best to start with your homepage, since that is often where the majority of your visitors land on your site.

Any element on a page can impact the performance of the page. Common elements to test are copy, images, buttons and links, types of offers, and even the entire design of a page. If you haven't done much testing on a page previously, you may be able to produce significant improvements with seemingly very simple changes. Some commonly tested elements are the:

  • Headline: straightforward, catchy, promotional, casual, formal, etc.
  • Offer: test a variety of offers to determine what resonates best with your audience.
  • Main Image: people of varying appearance, race, and sex, or pictures of your products
  • Button Copy: experiment with variations of call to action copy, e.g., "Sign up now", "Sign up Today", "Learn More", "Get the First Month Free", "Start Your Free Trial", etc.
  • Button Color: surprisingly, changing the color of a button can increase page performance
  • Short Page vs. Long Page

Another aspect to consider when selecting a page to test is whether there will be enough visitors to the page to determine a winner for your test in a reasonable amount of time. You'll need to test on pages with enough traffic to reach what is referred to in testing parlance as statistical significance. Statistical significance uses standard deviations and confidence levels to determine whether your test reflects what will happen when a winning test is rolled out to 100% of your visitors. There are many handy tools out there for calculating statistical significance, such as this one from Visual Website Optimizer.

RelatedImprove Online Conversions: Avoid 5 Common Mistakes

Coming Up Next: Testing software, setting up your first test, and iterating on your A/B tests

Now that we've covered the basics and started to identify the pages you'd like to test and which elements can be tested on each, the next step is... actually setting up your first test! Part two of this post, coming up next month, will cover the various testing software on the market as well as recommended software for A/B testing newcomers. It will also cover how to go about setting up the software (don't worry, it's easy), running your first test, and how to build testing into your routine on an ongoing basis.

(Image: KROMKRATHOG via freedigitalphotos.net)

Login to Business.com

Login with Your Account
Forgot Password?
New to Business.com? Join for Free

Join Business.com

Sign Up with Your Social Account
Create an Account
Sign In

Use of this website constitutes acceptance of the Terms of Use, Community Guidelines, and Privacy Policy.

Reset Your Password

Enter your email address and we'll send you an email with a link to reset your password.