What is A&B Testing?
- A/B Test (Split test) is the simplest way to analyze the effectiveness of a particular element in your marketing campaign
- This type of test consists of 4 key components
- All key components must be present before the test goes live
What is good to test?
- Success of A2 Features (video, carousel, colors, etc.)
- Success of Campaign Type (Slingshot, DPP, Cart Sweeper, etc.)
- Discounts (amount, urgency level, reciprocation, etc.)
- Comparison between in house creative and customer creative
- Segmentation and Sectioning (geotarget, multiscreen, site pages, etc.)
- Product Focus vs Category Focus
Best Practices & Tips
- Test one element at a time
- Less is more. Testing one element at a time will limit the contamination of other effects on the performance data for better, clearer results.
- Always have a hypothesis.
- Test both variations simultaneously
- Timing affects performance. Both variations need to run at the same time, day and for the entire length of the testing period.
- Dig deep when analyzing results
- Be open to alternative hypotheses that may arise. Surprising results might require more testing and minor revisions to your hypothesis and testing variables.
- Consider all variables that could affect the data. For example, was it a holiday weekend? Was the testing period too short? Were there other marketing campaigns running at the same time that you did not know of?
- Pick a sensible length for the testing period
- Ending the test too soon means you may miss out on valuable performance data
- Dragging the test out too long and you run the risk of marginalizing and diluting the data
- Use a minimum split of 80/20 for best results
- An easy test can be conducted with 'feature' vs. 'no feature' (i.e. carousel vs. no carousel)