knowledge base

We're here to help.


A&B Testing

What is A&B Testing?

  • A/B Test (Split Test) is the most simple way to analyze the effectiveness of a particular element in your marketing campaign
  • This type of test consists of 4 key components
  • All key components must be present before the test goes live

What is good to test?

  • Success of A2 Features (video, carousel, colors, etc)
  • Success of Campaign Type (Slingshot, DPP, Cart Sweeper, etc)
  • Discounts (amount, urgency level, reciprocation, etc)
  • Comparison between in house creative and customer creative
  • Segmentation and Sectioning (geotarget, multiscreen, site pages, etc)
  • Product Focus vs Category Focus

Best Practices & Tips

  • Test one element at a time
    • Testing one element at a time will limit the contamination of other effects on the performance data for better, clearer results
  • Always have a Hypothesis
  • Test both variations simultaneously 
    • Timing affects performance. Both variations need to run at the same time, day and for the entire length of the testing period.
  • Dig deep when analyzing results
    • Be open to alternative hypotheses that may arise. Surprising results might require more testing and minor revisions to your hypothesis and testing variables.
    • Consider all variables that could affect the data. For example, was it a holiday weekend? Was the testing period too short? Were there other marketing campaigns running at the same time that you did not know of?
  • Pick a sensible length for the testing period
    • Ending the test too soon means you may miss out on valuable performance data
    • Dragging the test out too long and you run the risk of marginalizing and diluting the data
  • Use a minimum split of 80/20 for best results
  • An easy test can be conducted with 'feature' vs. 'no feature' (ie, carousel vs. no carousel)
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Powered by Zendesk