Your onboarding programme has a 45% open rate and the customers who receive it upgrade more than average. Is it working? Possibly. But those customers might have upgraded anyway. Open rates, click rates and "customers who received it" comparisons can't tell you what a programme actually changed.
The only way to know is to measure incremental uplift: the difference between what happened with the programme and what would have happened without it. The tool for that is the holdout group.
What is a holdout group?
A holdout group is a randomly selected portion of an audience that deliberately doesn't receive a programme. Because the selection is random, the holdout and the targeted group are otherwise alike, so any difference in their results can be attributed to the programme itself.
It's the same principle as a controlled experiment. Without a control group, you're measuring correlation. With one, you're measuring cause and effect.
Two levels of holdout
A robust measurement framework uses holdouts at two levels:
- Global holdout: a small, permanent group that receives no lifecycle programmes at all. Comparing it with everyone else shows the total incremental impact of lifecycle marketing on business metrics like revenue, lifetime value and churn.
- Programme holdouts: a holdout for each individual programme, measured against that programme's specific goal, such as activation rate for onboarding or upgrade rate for an upsell journey.
The global holdout answers "is lifecycle marketing worth the investment?". Programme holdouts answer "which programmes are working, and by how much?".
How big should a holdout be?
Holdout size is a trade-off. A bigger holdout gives you more reliable results, faster, but it means more people miss out on a programme that probably helps them. As a rule of thumb:
- B2C: around 5%. Audiences are large, so even a small percentage gives you a statistically useful group.
- B2B: up to around 25%. Lead volumes are much lower, so you need a larger share to reach meaningful results in a reasonable time.
Check that the holdout will be big enough to detect the size of uplift you care about within the time you're willing to wait. If it won't be, either run the test longer or increase the holdout.
Worked examples
The figures below are illustrative, but they show how holdout results are read. First, a B2B lead nurture programme with a 25% holdout:
| B2B lead nurture | Leads | Lead → MQL (primary) | MQL → SQL (secondary) |
|---|---|---|---|
| Nurtured (75%) | 620 | 34.0% | 41.0% |
| Holdout (25%) | 205 | 27.0% | 38.0% |
| Incremental uplift | +7.0pp | +3.0pp |
And a B2C upgrade programme with a 5% holdout:
| B2C upgrade programme | Customers | Upgrade rate (primary) | New product activated in 30 days (secondary) |
|---|---|---|---|
| Targeted (95%) | 85,500 | 4.2% | 61.0% |
| Holdout (5%) | 4,500 | 3.1% | 58.0% |
| Incremental uplift | +1.1pp | +3.0pp |
Notice that the holdout in the upgrade example still upgrades at 3.1%. Without a holdout, you'd credit the programme with the full 4.2%. The real impact is the 1.1 percentage points on top, and that's the number to multiply by customer value when you estimate incremental revenue.
Testing improvements without losing the holdout
When you want to improve an existing programme, test a new challenger version against the current control, while keeping the holdout in place. With a 10% holdout, for example, the control and challenger each receive 45% of the audience. That tells you both whether the challenger beats the control, and how much each beats doing nothing.
Always-on programme: challenger test with a holdout
One-off campaign: the 10/10/80 test
Start every test with a hypothesis
Tests are most useful when they're written down before they start: what you're changing, why you expect it to work, and how you'll judge it. Two examples from an onboarding programme:
Writing the hypothesis first stops results being reinterpreted after the fact, and makes the learning reusable: if a reminder strategy works for one step, it's a strong candidate to test at the next.
One-off campaigns: the 10/10/80 test
Holdouts suit always-on programmes. For one-off campaigns, such as a launch email or a seasonal promotion, a 10/10/80 test is a simple way to improve results in a single send:
- Send version A to a random 10% of the audience and version B to another 10%.
- Wait long enough to see a clear winner on your primary metric, often a few hours for opens and clicks, longer for conversions.
- Send the winning version to the remaining 80%.
Pick the metric that matches the campaign's goal: click rate or conversion for a promotion rather than opens, which privacy features make unreliable. Record what won and why, so each campaign teaches you something for the next.
Multivariate testing, which tests several elements at once, is only worth it where volume supports it. That usually means high-volume B2C flows like activation and upgrades. B2B lead volumes rarely sustain it.
Choose the right metrics for each stage
Each lifecycle stage should have performance (primary), engagement (secondary) and commercial metrics, all compared against the holdouts:
| Stage | Performance | Engagement | Commercial |
|---|---|---|---|
| Acquire (B2B) | Lead → MQL rate | MQL → SQL rate | Pipeline created |
| Activate | Activation rate | Key feature rate | Free-to-paid conversion |
| Adopt | Feature adoption rate | Weekly active users | Average revenue per user |
| Expand | Upgrade rate | Cross-sell rate | Average revenue per user |
| Retain | Monthly active users | Renewal rate | Churn rate, lifetime value |
Group KPIs by what they tell you
It also helps to group KPIs into three families, so reports answer three different questions:
- Activation KPIs: are new customers reaching value? Step completion rates, activation rate and time to first value.
- Retention KPIs: are customers sticking around? Day 7 and day 30 retention, repeat usage rate, usage frequency and stickiness (daily active users divided by monthly active users).
- Efficiency KPIs: is the programme working as hard as it can? Push opt-in rate, email and in-app engagement, core feature adoption and the value of customers activated within 30 days.
Each programme also needs a written specification of the events, attributes and signals it depends on, such as signups, plan changes and product usage events, so the data is in place before it launches.
Report it in the right places
Different audiences need different views of the same data. Four dashboards cover most needs:
- Executive dashboard: weekly, monthly and quarterly business reviews, covering total customers, lifetime value, churn, activation and estimated incremental revenue against the global holdout.
- Audience dashboard: segments and cohorts, such as users per product and single versus multi-product customers, compared with the global holdout.
- Funnel dashboard: the shared B2B view with sales, covering conversion at each stage, SLA compliance and rejection reasons.
- Programme dashboard: each programme's recipients, engagement and primary metric against its own holdout.
Make it a habit
- 1Prioritise
- 2Hypothesise
- 3Test
- 4Analyse
- 5Validate
- 6Scale
↺ Validated learnings feed the next round
Holdouts work best as a rule, not an occasional experiment: every programme ships with a hypothesis and a holdout, results are analysed against the primary metric, and only validated learnings are scaled. Over time that gives you something most marketing teams don't have: proof of exactly what lifecycle marketing adds to the business.
Measurement underpins every stage of our lifecycle marketing framework. If you'd like help setting up incremental measurement for your programmes, book a free consultation.