Coming next: The Second Factory →

Back to Library
Metrics & Evidence⚠️ Pitfall

The Rain Dance That Worked (When You Credit the Wrong Thing)

The Problem

Sales team starts a new outreach campaign in January. Revenue spikes in February. VP of Sales declares victory: "Our campaign worked! Do more of it!" Except revenue spikes every February—it's tax refund season. The campaign did nothing. But the team doubles down, wastes budget, and wonders why results don't repeat. It's the rain dance fallacy: you do a rain dance, it rains, you assume the dance worked. Ignore that it was already cloudy. Humans are wired to find patterns and assign credit, even when it's coincidence. In business, this leads to investing in things that don't work and ignoring things that do. It's expensive and dumb.

The Principle

Before you credit success to your action, ask: "What would have happened anyway?" Use control groups, historical baselines, and holdout tests to separate your impact from external factors. Think manufacturing: if defect rates drop after you change suppliers, is it the supplier? Or is it the new QA process you also implemented? Isolate variables. The magic happens when you know what actually drives results so you can do more of it and less of everything else. Less guessing, more isolating. That's how you spend budget wisely.

🎯

3 Steps to Take Action

  1. 1

    Check Historical Baselines: Before crediting your action, look at what happened in prior years. If sales spike in December, check last December. Separate your impact from seasonality.

  2. 2

    Use Control Groups: For big initiatives, test with a control group. Half get the treatment, half don't. Measure the difference. That's your true incremental impact.

  3. 3

    Build an "Impact Checklist": Before celebrating success, ask: What would have happened anyway? Did we isolate our impact? Do we have a control? No shortcuts on causality.

In Practice

A retail chain launched a loyalty program in November. December sales were up 30%. CMO presented to the board: "Our loyalty program is a massive success!" Allocated $5M to expand it nationally. But someone checked: December sales were up 30% last year too—holiday season. They ran a test: half the stores got the loyalty program, half didn't. Results? Stores with loyalty: +31%. Stores without: +29%. The program added 2%, not 30%. Holiday season did the heavy lifting. They scaled back the rollout, optimized the program, and actually measured incremental lift. Next iteration added 8%. That was worth scaling. Saved $3M by not over-investing in something that barely worked. All because they isolated causality instead of celebrating coincidence.

More on Metrics & Evidence

From the book

In the book this comes to a head in a board meeting, where growth theater meets its limit. The fix is honest metrics, not better slides.

This card is one of 57 drawn from The Auction Block by Brian Cariveau — the story of an analytics leader with 90 days to fix what the cards describe.

Each leadership pattern falls into a few pitfalls most often. Find your pattern.The Analytics Leadership Mirror: free, 5 minutes, no account