What is A&B Testing?
- A/B Test (Split test) is simplest way to analyze the effectiveness of a particular element in your marketing campaign
- This type of test consists of 4 key components
- All key components must be present before the test goes live
What is good to test?
- Success of A2 Features (video, carousel, colors, etc)
- Success of Campaign Type (Slingshot, DPP, Cart Sweeper, etc)
- Discounts (amount, urgency level, reciprocation, etc)
- Comparison between in house creative and customer creative
- Segmentation and Sectioning (geotarget, multiscreen, site pages, etc)
- Product Focus vs Category Focus
Best Practices & Tips
- Test ONE ELEMENT at a time!
- Less is more. Testing one element at a time will limit the contamination of other effects on the performance data for better, clearer results.
- ALWAYS have a Hypothesis.
- Test both variations SIMULTANEOUSLY
- Timing affects performance. Both variations need to run at the same time, day and for the entire length of the testing period.
- DIG as deep as you need when analyzing results
- Be open to alternative hypotheses that may arise. Surprising results might require more testing and minor revisions to your hypothesis and testing variables.
- Consider all variables that could affect the data. For example, was it a holiday weekend? Was the testing period too short? Were there other marketing campaigns running at the same time that you did not know of?
- Pick a sensible length for the testing period
- Ending the test too soon means you may miss out on valuable performance data
- Dragging the test out too long and you run the risk of marginalizing and diluting the data
- Use a minimum split of 80/20 for best results
- Always be testing, but apply COMMON SENSE
- While everything is testable, not everything should be tested. Think about what makes the most sense.
- When testing with Revenue Per Visitor (RPV) as the main metric, a RPV lift of 4% is not conclusive. Keep testing!
- An easy test can be conducted with 'feature' vs. 'no feature' (ie, carousel vs. no carousel)