Launch Optimiser Test
Enables you to test your hypothesis and ensures it is statistically conclusive so that you are confident to roll it out.
An Optimiser test is a champion/challenger test where the champion is your current campaign and the challenger is another campaign that you build to test your hypothesis and it runs side-by-side with the champion.
Create Optimiser test
You can Build a next challenger from the Overview screen or you can Start experiment on the Campaigns screen you'd like to challenge from the Campaigns screen. This will open up a wizard that guides you through building a challenger. This challenger can either have different message content, communication channels or just have a different frequency of communication days with different features. It depends on your hypothesis and what exactly are you testing as a differentiator in collections uplift.
While launching the test, you can decide what percentage of customers you'd want to distribute in the champion and challenger. Typically, a 50-50 split ensures there are enough customers going through both variants to make a timely and statistically conclusive decision.
Once you launch the test, we strongly recommend not peeking for at least 2 weeks from the start of the test. This gives enough time for customers to go through the campaign and avoids any knee-jerk reactions on campaign performance given it's too early to get a statistically conclusive result.
Monitor test metrics
After 2 weeks, you're encouraged to view the test performance in the Optimiser 'Metrics' screen. This screen shows you the Conversion and NPS metrics and uplift percentage. It also shows if the test is statistically conclusive or not. If the test reaches a statistically conclusive result, it can either recommend to Promote challenger on Continue with champion. If the test is not yet statistically conclusive, it will say Wait.
If you go with the Promote challenger recommendation, this will automatically stop the champion and promote your challenger as the new champion (i.e., all customers qualifying for the campaign target audience will now go through this new champion). On the other hand, the Continue with champion option allows you to continue with your current champion and stops the challenger.
Overtime, as you continue to promote the challenger (when successful), you are optimising your campaign resulting a better version iteratively.
💡 Pro Tip:
- Avoid peek into newly created tests for at least first 2 weeks to avoid making any preliminary conclusions while the test is still not statistically conclusive.
- Avoid changing too many variables for a single test to ensure you can identify the change that causes the uplift.
Updated 5 days ago