Coming next: The Second Factory →

Back to Library
Metrics & Evidence⚠️ Pitfall

Finding What You Wanted to Find (Even When It Wasn't There)

The Problem

CEO wants to prove the new pricing strategy is working. Asks analytics for a report. Analyst finds data showing revenue is up in two regions—ignores the three regions where it's flat or down. Presents the good news. CEO is thrilled. Strategy continues. Six months later, overall revenue is down 8%. What happened? Confirmation bias. We see what we want to see. We cherry-pick data that supports our hypothesis and ignore data that doesn't. It's human nature. But it's deadly in analytics. You end up making decisions based on a partial, distorted view of reality. It's like a factory manager only checking the machines that are working and ignoring the broken ones. Eventually, production grinds to a halt.

The Principle

Seek to be proven wrong, not right. Before you analyze, write down what you expect to find. Then actively hunt for data that contradicts it. Pre-commit to metrics before you see results. Involve people who disagree with you in the analysis—they'll catch your blind spots. Think manufacturing: quality control doesn't just test products they expect to pass. They stress-test everything, especially edge cases. Same with analysis. The magic happens when you build a culture where being wrong fast is celebrated because it prevents being wrong expensive later. Less ego, more truth. That's how you make decisions that hold up.

🎯

3 Steps to Take Action

  1. 1

    Pre-Register Your Hypothesis: Before you start analysis, write down what you expect to find. Then actively hunt for data that contradicts it. Force yourself to seek disconfirmation.

  2. 2

    Red Team Your Analysis: Assign someone to poke holes in every analysis before it goes to leadership. Their job: find what you missed, challenge your assumptions, be skeptical.

  3. 3

    Pre-Commit to Metrics: Before you run an experiment or launch, agree on the success metrics. Write them down. No changing them after you see results. This prevents cherry-picking.

In Practice

A marketing VP believed their Super Bowl ad drove a huge sales spike. Asked analytics to prove it. Team found sales were up 22% the week after the ad. Report confirmed the VP's belief. Victory! Except one skeptical analyst checked: sales were up 22% every year the week after the Super Bowl, ad or no ad. It was seasonality. The ad did nothing. But nobody wanted to hear it. Company ran the ad again next year—$8M spend, zero incremental sales. New CMO came in, instituted "red team reviews"—every analysis has someone assigned to poke holes in it. Next ad campaign, they ran a proper holdout test. Ad drove 6% lift—worth it, but not the 22% they thought. Spent smarter, optimized creative, ROI improved. All because they stopped confirming beliefs and started seeking truth.

More on Metrics & Evidence

From the book

In the book this comes to a head in a board meeting, where growth theater meets its limit. The fix is honest metrics, not better slides.

This card is one of 57 drawn from The Auction Block by Brian Cariveau — the story of an analytics leader with 90 days to fix what the cards describe.

Each leadership pattern falls into a few pitfalls most often. Find your pattern.The Analytics Leadership Mirror: free, 5 minutes, no account