knowledge base

We're here to help.


A&B Testing

What is A&B Testing?

  • A/B Test (Split test) is simplest way to analyze the effectiveness of a particular element in your marketing campaign
  • This type of test consists of 4 key components
  • All key components must be present before the test goes live

What is good to test?

  • Success of A2 Features (video, carousel, colors, etc)
  • Success of Campaign Type (Slingshot, DPP, Cart Sweeper, etc)
  • Discounts (amount, urgency level, reciprocation, etc)
  • Comparison between in house creative and customer creative
  • Segmentation and Sectioning (geotarget, multiscreen, site pages, etc)
  • Product Focus vs Category Focus

Best Practices & Tips

  1. Test ONE ELEMENT at a time! 
    1. Less  is more. Testing one element at a time will limit the contamination of other effects on the performance data for better, clearer results. 
  2. ALWAYS have a Hypothesis.
  3. Test both variations SIMULTANEOUSLY
    1. Timing affects performance. Both variations need to run at the same time, day and for the entire length of the testing period.
  4. DIG as deep as you need when analyzing results
    1. Be open to alternative hypotheses that may arise. Surprising results might require more testing and minor revisions to your hypothesis and testing variables.
    2. Consider all variables that could affect the data. For example, was it a holiday weekend? Was the testing period too short? Were there other marketing campaigns running at the same time that you did not know of?
  5. Pick a sensible length for the testing period
    1. Ending the test too soon means you may miss out on valuable performance data
    2. Dragging the test out too long and you run the risk of marginalizing and diluting the data
  6. Use a minimum split of 80/20 for best results
  7. Always be testing, but apply COMMON SENSE
    1. While everything is testable, not everything should be tested. Think about what makes the most sense.
  8. When testing with Revenue Per Visitor (RPV) as the main metric, a RPV lift of 4% is not conclusive. Keep testing!
  9. An easy test can be conducted with 'feature' vs. 'no feature' (ie, carousel vs. no carousel)
Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request


Powered by Zendesk