The Hidden 65%: How Holdout Groups Reveal Your True Direct Mail ROI
Matchback reports only capture 20-35% of direct mail responses. Learn how holdout groups help HVAC, plumbing, and electrical contractors measure true incremental revenue.
Your matchback report says the spring tune-up campaign drove $216,000 in revenue. Your CFO asks: "How much of that would have come in anyway?"
You don't have a good answer.
Matchback attribution tells you who received mail and later booked a job. It doesn't tell you whether the mail caused them to call. For contractors running maintenance reminders or win-back campaigns, this distinction is everything.
The Attribution Gap
According to SeQuel Response, directly trackable elements like QR codes and call tracking only capture 20-35% of direct mail responses. The rest? Customers who saw your postcard, stuck it on the fridge, and called your main line three weeks later when the AC quit.
Matchback recovers some of those jobs by matching your ServiceTitan data back to your mail file. But it can't answer the real question: would they have called anyway?
The Holdout Test
A holdout group is a small slice of your audience (5-10%) that you intentionally don't mail. They're your control group.
After 60-90 days, compare revenue between the two groups. Here's what that looks like for an HVAC contractor running a spring AC tune-up campaign to 10,000 past customers:
| Mailed Group | Holdout Group | |
|---|---|---|
| Customers | 9,000 | 1,000 |
| Jobs Booked | 540 (6.0%) | 38 (3.8%) |
| Total Revenue | $216,000 | $15,200 |
| Avg Ticket | $400 | $400 |
| Revenue per Customer Mailed | $24.00 | $15.20 |
The holdout group generated $15.20 per customer without any mailer. That's your baseline: the natural seasonal demand as temps rise and homeowners start thinking about their AC.
Your true incremental lift: $8.80 per customer.
Campaign value: $8.80 × 9,000 = $79,200 in incremental revenue, not $216,000.
That's still strong ROI on a direct mail campaign. But now you know what the mail actually moved, and you can make smarter decisions about targeting, offers, and spend.
A note on sample size: A 1,000-person holdout gives you directional insight, but statistical confidence improves with larger lists or by aggregating results across multiple campaign cycles. If you're mailing 50,000+, a 5% holdout becomes much more reliable.
Why This Matters for Contractors
Seasonality drives demand in the trades. Furnaces fail in January. AC units die in July. Drains back up after Thanksgiving. Customers on your maintenance list may call regardless of whether they got a postcard.
Without a holdout test, you might be offering a $50 discount on jobs that were coming anyway, eroding margin on revenue you didn't need to buy.
Research from Measured found one brand's direct mail was delivering just 14% incremental lift, far below the 40% their vendor reported.
What You'll Learn
The real power comes from comparing holdout performance by segment:
| Segment | Mailed Response | Holdout Response | Incremental Lift |
|---|---|---|---|
| Active maintenance agreement | 12% | 11% | Low |
| Last service 1-2 years ago | 6% | 3% | High |
| Lapsed 3+ years | 2% | 1.5% | Moderate |
This tells you where your marketing dollars actually work:
Maintenance agreement customers are already loyal. They'll book their tune-up without a reminder. Reduce frequency or skip them entirely.
1-2 year inactive customers show strong incremental lift. Double down here. Test stronger offers or earlier timing.
Lapsed 3+ years respond poorly overall. They may have moved or switched contractors. Consider reallocating that spend to conquest campaigns targeting new homeowners in your territory.
Making It Operational
Holdout testing requires discipline: clean data, consistent execution, and tracking who was held out and why. Most shops don't have the bandwidth to manage this manually alongside day-to-day operations.
Purpose-built software can automate the heavy lifting. At Arch, holdout measurement is built into every campaign. Contractors see true incremental revenue alongside matchback results without managing the test themselves, and the system continuously learns which segments need the nudge and which don't.