What To Do After A Bad Incrementality Test Result | Open Haus Playbook

What “bummer news” means in incrementality testing

A poor incrementality result is framed as a business learning opportunity, not a failure—because it reveals what your business actually gets from a channel beyond what attribution dashboards imply.

Common “bummer” symptoms

• CPIA (cost per incremental acquisition) is much higher than expected
• Incrementality rows (incremental outcome “rows”) are low
• Lift % is low
• Incremental ROAS is very low
• Sometimes channels perform much worse than the team’s efficiency benchmark (example cited: up to ~700% worse)

Framework: Reframe the poor result into decisions + next-step actions

The presenters emphasize that the critical work is what you do after the test—financial, operational, and creative.

Three buckets of “bummer news”

1. Baseline channel / undervalued-attribution channel

   • Example: YouTube, CTV, and other channels thought to be undervalued by click-based attribution
   • Result: Can come back closer to “normal” than expected (not always disastrous)

2. Hype new channel

   • Example: Apple “11”, Snapchat, Reddit-style “everyone is piling in” channels
   • Result: Can come back far worse than expected (example cited: ~700% worse)

3. Sacred cow

   • A channel with massive historic spend
   • Result: Low lift and inefficient incremental economics
   • Implication: Your historical operation may have been misleading due to attribution/measurement blind spots

KPI / metrics explicitly used (and how they drive action)

• Lift %: Low lift triggers re-evaluation of channel efficiency and spend allocation
• CPIA: High CPIA vs expectations indicates poor incremental economics
• Incremental ROAS: Very low implies weak incremental returns
• Incremental contribution to revenue: (example provided below)

Concrete example: YouTube test result and what it changes

YouTube experiment (e-commerce brand)

• Design: Treatment = YouTube Ads; Holdout = no YouTube Ads
• Duration: 6-week test + 3-week lag window (total 9 weeks maturity)
• Observed outcome: YouTube drove only 0.28% of contribution in revenue
• Interpretation:
  • If you removed YouTube, you’d lose only 0.28% of contribution (per the test framing)

Management decision implication

Either:

• Improve YouTube using evidence, or
• Reallocate budget (and free marketing time) toward incremental channels

How to communicate “bummer news” to finance / leadership

Key messaging tactic

• “We now have a data point we didn’t have before.”
• This reshapes how media investment should be deployed—return to profit, R&D/creative iteration, or other incremental channels.
• Finance should be positioned as learning the true incremental economics, so budget can be corrected or repurposed rather than justified by dashboards.

Quote used as a rationale

• If media dollars were truly delivering business outcomes, you’d “go buy lottery tickets,” implying attribution/assumptions shouldn’t be treated as truth without incrementality.

Post-test playbook: the decision tree (care vs don’t care)

If you don’t care about the channel (cut / redeploy)

Triggered when results show low lift/efficiency (e.g., low incremental contribution, high CPIA).

Actions:

• Redeploy budget to a more incremental channel (already tested or proven)
• Use freed budget for activation (e.g., launching in a major city) or for creative/marketing initiatives previously constrained by budget
• Reduce operational complexity and free bandwidth/time for higher-impact marketing work

If you do care about the channel (make it work)

When leadership buy-in exists that the channel can earn share of wallet, improvement paths include:

• One-to-one approach (methodical)
• Fire hose approach (rapid, larger bets)

Also noted:

• Maintain a short-term vs long-term balance—tests run for weeks (6–12), but channel effects may have longer-term implications.

Two operational strategies to follow the baseline test

1) One-to-one approach (“sandwich” of tactical tests)

Goal: improve the channel while keeping causal clarity.

Process (baseline → 2 tactical tests → re-baseline)

• Baseline = your low-lift test result
• Next, run two tactical tests you have high conviction in (in a “sandwich” pattern)
  • Creative optimization
  • Media buying optimization
• After improvements (e.g., 1–2 months later), re-baseline

Example target result format:

• Lift improves (e.g., 1% → 4%)
• CPIA drops (e.g., $1500 → $500)

Operational requirements

• Build and document hypotheses (why lift was low), then test them cleanly
• Don’t pile too many changes at once, or you lose clarity on what caused improvement
• Maintain a historical backlog of what changed over time (ad tech/market shifts can alter performance)

Concrete hypothesis examples

• Creative
  • Test brand story vs direct response
  • Test mobile-heavy vs other creative story angles
• Media buying
  • YouTube: test CTV screen vs computer screens targeting
  • Paid social: attribution settings (e.g., 1-day click vs 7-day click, 1-day view)
  • Persona expansion: broader spending toward a high-value customer persona not previously built for

2) Fire hose approach (“machine gun”)

Goal: unlock a breakthrough fast when you believe the channel is strategically important.

Process

• Run rapid succession of two-cell tests with big swings (creative and/or audience targeting)
• Change faster than A/B-style incremental tuning
• Continue testing until one of two outcomes:
  • Efficiency skyrocks after a later swing (sometimes after 1 more test)
  • After 3–5 tests, conclude the channel fit isn’t right and move on

Speed-to-win example

• A brand fell ~6% below target
• Identified a creative theme and oriented the entire account around it
• Retested and improved immediately after the second baseline (showing potential rapid upside)

When to choose which

• Fire hose: best for net-new channel expansion and new ad product experimentation
• One-to-one: best when the channel has been working historically, and you’re refining incrementally

Additional case studies / examples cited

Javy Coffee (creative shift case)

• Shift: bottom-of-funnel direct-response (pricing/promos) → top-of-funnel value proposition / awareness
• Result: 13x lift (between the exploratory awareness creative and the direct-response creative)

Consumer electronics brand (YouTube ROAS improvement)

• Improved YouTube ROAS by 400%
• Path:
  • Baseline → one-to-one iteration
  • Then a three-cell test comparing creative styles
    • Brand story creative vs product creative
  • Winner: brand story creative
  • Next: audience targeting after establishing the creative “breakthrough”

Playbook: tactical experiment “menu”

The team notes there are dozens of tactical experiments beyond what’s shown—often hundreds tailored to specific brands. The discussion highlights that the “sandwich” generally covers:

• Creative optimization
• Media buying optimization
• Then re-baseline

Key takeaways / management principles

• A poor incrementality result is a learning gift because it uncovers profitability-relevant truths you can’t infer from traditional attribution.
• The goal isn’t to “win the test”; it’s to drive behavioral change in media allocation, creative resourcing, and operational focus.
• Even perfect results won’t change the business unless you act on them—doing nothing means nothing changes.

Presenters / sources

• Dean Gordon, Measurement Strategist, House Analytics (Open Haus Playbook)
• Ike Armstrong, Measurement Strategist, House Analytics (Open Haus Playbook)