A/B Testing Quick-Start Guide, Part 2: Setting Up Your First A/B Test

marketing site optimizationAs a small business, you have a ton of competing priorities, and finding time for A/B testing and optimization of your website can be challenging.  Part 1 was an overview of A/B testing, its value, and how to select pages for testing.  Part 2 builds upon the basics by explaining how to quickly and easily get started with A/B testing on your site.

RelatedA/B Testing Quick-Start Guide, Part 1: What is A/B Testing?

A/B Testing Software

The number of Testing and Optimization software options has increased significantly over the past several years.  In the mid-2000’s, the well known and widely used testing software cost around $10,000 per month and required at least half of a full time employee’s time to administer and operate.  As testing has evolved and become more commoditized, these days it’s not uncommon to pay as little as $49 per month for a full featured and easy to use testing platform.

Two great platforms to try out are Wingify’s Visual Website Optimizer (VWO) and Optimizely.  Both are very easy to use, and both offer a full set of features for a low monthly price.  After initial setup, these platforms require little or no IT involvement to set up new tests.  Even initial setup is relatively easy for anyone with basic HTML skills: a single code snippet needs to be pasted into your website’s header so that it will render on every page of your site.  Both VWO and Optimizely have free 30 day trials. If you’d like to explore a comprehensive list of testing software available today, this site has done a great job of compiling an unbiased list: http://www.conversion-rate-experts.com/split-testing-software/

A note on Google Analytics’ Content Experiments: Google used to have a free A/B testing tool called Google Website Optimizer that has since been retired.  Google now offers testing and optimization functionality through its (also free) Analytics software.  The new version of their software requires that you create multiple variations of pages, and then it splits traffic between the pages and tracks the results.

Setting Up Your First Test

You’re ready to begin testing when:

  • The testing software code is in place (see above)
  • You’ve selected a page or pages for your test (see Part 1)
  • You’ve chosen elements that you want to test (see Part 1)


  1. The first step is to use the testing software interface to create a test version of the page that you want to test.  VWO and Optimizely both have drag and drop interfaces that allow you to easily modify your page.
  2. Next, set the traffic split to 50% control (your original layout) and 50% test
  3. Turn on the test through the testing interface, and monitor it regularly to get an understanding of early performance and whether statistical significance has been reached (see Part 1 for more information on statistical significance)

Related: The Top 4 Tips for Dealing with Marketing Complexity


It is important to keep testing the same page until you’ve exhausted ideas and your gains begin to plateau.  Inevitably, some tests that you’re 100% convinced will win (i.e. provide a boost to your metrics) will actually perform worse than your existing page.  This happens often in A/B testing.  They key when this happens is not to abandon the test but to stay the course and iterate on the failed tactic or try something else.

Staying the Course

Lastly, the biggest mistakes with any new or existing page on your site are a) not optimizing with A/B testing, and b) getting discouraged by early underperformance and shifting focus to other projects.  Don’t get discouraged if your early A/B testing efforts fail!  It’s very typical if the first couple tests either lead to a decrease or have no effect.  With a bit of time and practice, you’ll be on your way to increasing your key metrics in no time.

(Image: nokhoog_buchachon via freedigitalphotos.net)

Rob Canning

Rob Canning

Author's Website: http://www.business.com

Author's Social Links: Author Google Plus Profile Link Author Twitter Profile Link Author LinkedIn Profile Link

Rob Canning is the VP of Product Management and Content at Business.com.

View Comments

One Response to A/B Testing Quick-Start Guide, Part 2: Setting Up Your First A/B Test

  1. K Butler says:

    I found this a useful article that covers the topic of multi-variate testing well and provides helpful resources. From my experience A/B resting soon becomes very complex and you need great discipline just to vary one element at at time and run comparisons and then use an evolutionary elimination of the weakest result. Be ruthless.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>