Setting Up Cancel Survey A/B Testing
A/B testing lets you run two or more cancel surveys simultaneously and measure which one saves more subscribers. Instead of guessing which offer sequence, tone, or flow structure performs better, you can split live traffic between survey variants, collect real performance data, and promote the winner, all without leaving Stay.
Follow the steps below to get started.
Before You Begin
You need at least two published cancel surveys to create an A/B test. If you haven't built your surveys yet, see Setting Up Your Cancellation Survey.
Only one A/B test can be active at a time.
While a test is active, the individual active toggle on each survey included in the test is disabled. Changes to a live test survey can skew results — lock in your surveys before enabling the A/B test.
1. Create an A/B Test
To create a new A/B test:
Navigate to Optimization → Cancellation Surveys in the Merchant Portal.
Click A/B Test in the top right corner of the survey list page.
Click Create New A/B Test.
Give the test a clear internal name (e.g., Q3 Offer Sequencing Test).
2. Configure Your Test Branches
Each branch in the test maps to one of your existing cancel surveys. When a subscriber initiates a cancellation, Stay routes them into one of the branches based on the traffic weights or conditions you set.
First, determine if you'd like to segment by weight, or segments. If you select "Manually A/B Test", you will determine the percentage of customers that should see each survey. If you choose to segment, the conditions that you define will determine which cancel survey the customer goes through.
Once you determine which A/B test type you want to use, configure the branches:
Under Branch A, click Select Survey and choose the first survey.
Under Branch B, click Select Survey and choose the second survey.
For manual A/B tests, set the traffic split for each branch.
📌 Note: You can add more than two branches if you want to test multiple survey variants at once. Click Add Branch to include a third or fourth survey.
✅ Best Practice: For statistically meaningful results, run the test long enough to collect a substantial sample before declaring a winner. A common minimum is 2–4 weeks, depending on your cancellation volume.
3. Activate the Test
Once your branches are configured:
Review the branch summary and confirm each branch points to the correct survey and weights are correct.
Click Activate Test.
📌 Note: Ensure all of your branching is correctly configured in each survey prior to activating the A/B test. If any of the surveys have errors at the time of activation attempt, you will not be able activate your test.
Stay immediately begins routing subscribers into branches according to your configured weights. The active toggle on each survey included in the test will be disabled in the survey list — this is expected behavior while the test is running.
📌 Note: Activating a test locks the participating surveys from being deactivated individually. To stop a test early, you must end the test from the A/B Test tab, not from the survey list.
4. Monitor Performance
While the test is running, you can track how each branch is performing in real time.
To view in-flow analytics per branch:
Navigate to Optimization → Cancellation Surveys.
Click A/B Test to open the test dashboard.
Select the A/B Test tab to see per-branch performance, including save rate, offer acceptance, and cancellation completion for each survey variant.
💡 Tip: The per-branch view mirrors the in-flow analytics available on individual surveys. If you're familiar with reading the Cancel Survey In-Flow Analytics, the A/B Test tab works the same way, just split by survey.
5. Declare a Winner
When you have enough data to make a call:
In the A/B Test tab, disable your A/B test.
Navigate to the Cancel Survey list, and activate the winning survey
The winning survey becomes the active survey and begins receiving 100% of traffic.