---
name: analytics-pitfalls-before-the-readout
description: "Before the readout: 13 cards from Analytics Pitfalls & Principles, by Brian Cariveau, to check a number, an analysis or a deck before it goes to leadership."
license: CC-BY-4.0
---

# Before the readout

From Analytics Pitfalls & Principles, by Brian Cariveau: 13 cards and 1 framework for a number, an analysis or a deck before it goes to leadership. Every card, free: https://www.analyticspitfalls.com/library

## How to use it

Use it when someone shares or describes a number, an analysis or a deck before it goes to leadership, before it goes any further.

1. Take Brian Cariveau's questions below in order, against their actual work, and ask them in his words rather than your own. For each, open the files it names and answer from them.
2. Skip a question the work already answers well. Say so in a line and move on.
3. Then pick the one or two cards that matter most for this work, say why in a sentence each, and give their principles in the cards' own words. Never hand someone the whole list.
4. Say the work looks like a pitfall; never tell someone they have it. If nothing fits, say so.
5. Finish with the links of the cards you used, and say which skill comes next (Where this fits, below).

## Brian Cariveau's questions

1. If this number goes up, does the business actually win? A million signups or 41,000 hours saved can both go nowhere. From: The Million User Mirage (`cards/million-user-mirage.md`), The 41,000 Hours (`cards/41000-hours.md`).
2. What else could explain it? Correlation is a clue, not proof. From: Ice Cream Causes Drowning (`cards/ice-cream-causes-drowning.md`), The Rain Dance That Worked (`cards/rain-dance-that-worked.md`).
3. Is the change bigger than the noise, or did we just flip a coin a few times? From: The Coin Flip Fallacy (`cards/coin-flip-fallacy.md`).
4. What’s the average hiding? Remember the statistician who drowned in a river three feet deep, on average. From: The Statistician Who Drowned (`cards/statistician-who-drowned.md`).
5. Did we set out to prove what somebody already wanted to hear? Write down what you expect before you look. From: Finding What You Wanted to Find (`cards/finding-what-you-wanted-to-find.md`).
6. What should someone do on Monday because of this? If it only looks in the rearview mirror, nobody will do anything. From: Rearview vs Windshield (`cards/rearview-vs-windshield.md`).
7. Does it need to be certain? Seventy percent on time beats ninety-five percent too late. Would a quick test settle it? From: The 95% Confidence Trap (`cards/95-confidence-trap.md`), Hope is Not a Strategy (`cards/hope-is-not-a-strategy.md`), The 70% Confidence Principle (`frameworks/seventy-percent-confidence.md`).
8. If AI wrote any of it, did someone actually read it? And is it inside the tolerance you set, or are you still asking whether it’s right? From: Confident, With Formatting (`cards/confident-with-formatting.md`), Is It Right? (`cards/is-it-right.md`).
9. Say the number’s right. What has to change in how people work for it to move? That’s the part a readout can’t do for you. From: We're in the Behavior Change Business (`cards/were-in-the-behavior-change-business.md`).

## Where this fits

These run in order: the strategy, then the work, then the results.

1. **The strategy**: Before the AI strategy (`analytics-pitfalls-before-the-ai-strategy`), Before the data strategy (`analytics-pitfalls-before-the-data-strategy`), Before the BI strategy (`analytics-pitfalls-before-the-bi-strategy`). Pick the one you’re writing. They sit side by side; there’s no order between them.
2. **The work**: Before the dashboard (`analytics-pitfalls-before-the-dashboard`). Once for each dashboard the strategy calls for.
3. **The results**: Before the readout (`analytics-pitfalls-before-the-readout`). When the first numbers go to leadership. This one is here.

This is the last of them. When you're done, say so: the next round starts again with the strategy.

## The cards

- **The Million User Mirage (Celebrating the Wrong Wins)** (`cards/million-user-mirage.md`): pitfall, Metrics & Evidence. Track what matters, not what looks good on a slide. Ask: "If this metric goes up, does revenue go up? Do customers get happier? Does the business actually win?"
- **The 41,000 Hours (Saved From What, Spent Where?)** (`cards/41000-hours.md`): pitfall, Metrics & Evidence. Never skip the middle. Engagement, experience, effectiveness, efficiency: in that order, every time. At work: "AI saved us 41,000 hours this year."
- **Ice Cream Causes Drowning (When Numbers Lie)** (`cards/ice-cream-causes-drowning.md`): pitfall, Metrics & Evidence. Always ask: "What else could explain this relationship?" Correlation is a clue, not proof.
- **The Rain Dance That Worked (When You Credit the Wrong Thing)** (`cards/rain-dance-that-worked.md`): pitfall, Metrics & Evidence. Before you credit success to your action, ask: "What would have happened anyway?" At work: "Sales went up right after we launched it, so it worked."
- **The Coin Flip Fallacy (When Noise Looks Like Signal)** (`cards/coin-flip-fallacy.md`): pitfall, Metrics & Evidence. Don't confuse noise with signal. Before you react to a change, ask: "Is this statistically significant or just random fluctuation?"
- **The Statistician Who Drowned (in a River with Average Depth 3 Feet)** (`cards/statistician-who-drowned.md`): pitfall, Metrics & Evidence. Dig into the distribution, not just the average. Look at medians, percentiles, outliers.
- **Finding What You Wanted to Find (Even When It Wasn't There)** (`cards/finding-what-you-wanted-to-find.md`): pitfall, Metrics & Evidence. Seek to be proven wrong, not right. Before you analyze, write down what you expect to find.
- **Rearview vs Windshield (Stop Driving While Staring Backwards)** (`cards/rearview-vs-windshield.md`): principle, Metrics & Evidence. Balance rearview (lagging) with windshield (leading) metrics. Lagging metrics tell you what happened (revenue, conversions, churn).
- **The 95% Confidence Trap (Perfection is the Enemy of Progress)** (`cards/95-confidence-trap.md`): pitfall, Metrics & Evidence. Ship at 70% confidence and learn fast. Set a decision deadline and work backward. At work: "Let’s wait until the data is perfect."
- **Hope is Not a Strategy (Why Guessing is Expensive)** (`cards/hope-is-not-a-strategy.md`): pitfall, Metrics & Evidence. Test everything. A/B test product changes, pricing, messaging, features—anything customers interact with.
- **Confident, With Formatting (When Nobody Reads What the AI Wrote)** (`cards/confident-with-formatting.md`): pitfall, Metrics & Evidence. Your name is on it, so read it. Checking is faster than doing, and it is never zero.
- **Is It Right? (The Wrong Question to Ask of AI)** (`cards/is-it-right.md`): pitfall, Metrics & Evidence. Ask whether it's within tolerance, and how you know. Nothing is right every time, in a factory or in AI.
- **We're in the Behavior Change Business (Not the Data Business)** (`cards/were-in-the-behavior-change-business.md`): principle, Adoption & Product. Apply consumer behavior principles to analytics adoption. Make it stupid simple to use—reduce friction, eliminate training requirements where possible.

## The frameworks

- **The 70% Confidence Principle** (`frameworks/seventy-percent-confidence.md`): The 70% Confidence Principle means shipping a directional answer at 70% confidence and learning fast, instead of chasing 95% certainty while the business waits.

---

© 2026 Brian Cariveau, Analytics Pitfalls & Principles. Free to share and adapt under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/): keep this credit, "Brian Cariveau, Analytics Pitfalls & Principles (https://www.analyticspitfalls.com)", and say if you changed it.
The names Analytics Pitfalls & Principles, The Auction Block and The Second Factory are not part of the license. Don't use them for your own version, or to suggest it comes from us.
