Coming next: The Second Factory →

Back to Library
Metrics & Evidence⚠️ Pitfall

The Million User Mirage (Celebrating the Wrong Wins)

The Problem

Board meeting. "We hit 1 million users!" Champagne pops. Everyone celebrates. Three months later you're laying off half the team because revenue is in the toilet. Turns out 97% of those "users" signed up, looked around for 30 seconds, and never came back. You were measuring signups when you should've been measuring engagement. Revenue. Retention. The stuff that actually pays the bills. Or your data team celebrates "We built 47 dashboards this year!" Leadership claps. Until someone asks: "How many have 5+ weekly active users?" Silence. Turns out 28 of those 47 dashboards have 2 or fewer users. Some have zero. It's like a manufacturing plant celebrating "we made 1 million widgets!" while ignoring that 970,000 are sitting unsold in a warehouse. Vanity metrics feel good but they're a trap—they let you lie to yourself about success while the business bleeds out.

The Principle

Track what matters, not what looks good on a slide. Ask: "If this metric goes up, does revenue go up? Do customers get happier? Does the business actually win?" If the answer is no, it's a vanity metric—kill it. Focus on the manufacturing mindset: what's the yield? What percentage of raw materials (users, leads, dashboards) become finished products (paying customers, repeat buyers, dashboards with 5+ weekly users)? Measure the conversion at every step of your production line. Better yet: pick HALO METRICS that force fundamentals. Instead of tracking 20 things, track 3 things that FORCE you to get the other 17 right. You can't game a halo metric—it only improves when fundamentals improve. Examples: "85% of solutions used weekly" forces you to build things people want. "Average time idea→production: 3 weeks" forces you to eliminate waste. The magic happens when your whole team obsesses over the metrics that connect directly to business outcomes. Less is more—pick 3-5 metrics that matter and make them the center of every conversation.

🎯

3 Steps to Take Action

  1. 1

    **Map Your Conversion Funnel** Draw it out - from signup to paying customer to retained customer (or from dashboard built to dashboard with 5+ weekly users). Calculate conversion % at each step. Identify the biggest drop-off. Ask: "What percentage of raw materials become finished products?" If <10%, you have a vanity metrics problem.

  2. 2

    **Pick 3-5 Halo Metrics That Matter** For each potential metric, ask: "If this goes up, does revenue go up? Do customers get happier? Can I game this by cutting corners?" If you can game it or it doesn't drive business outcomes, it's vanity. Kill it. Best metrics: - Force fundamentals (can't improve without fixing root issues) - Measure outcomes, not activity - Are un-gameable Examples: - "85% of solutions used weekly" (forces quality) - "Average idea→production: 3 weeks" (forces efficiency) - "Net revenue retention >110%" (forces customer success)

  3. 3

    **Weekly Metric Review Ritual** Every Monday, review your 3-5 halo metrics as a team. Ask: - "What changed?" - "Why?" - "What are we doing about it?" Make the uncomfortable visible. Best metrics reveal problems, not hide them. If a metric consistently shows green with no effort, it's probably vanity. Replace it with something harder that forces improvement.

In Practice

A SaaS startup celebrated hitting 500K signups. Investors loved it. Then burn rate caught up—they had 3 months of runway left. New CEO asked one question: "How many of those 500K are paying us?" Answer: 4,200. That's a 0.8% conversion rate. They pivoted hard. Stopped measuring signups. Started measuring: Free trial → paid conversion, Monthly active users, Customer LTV, Churn rate, Net revenue retention. Turned out their onboarding sucked—87% of free trials never completed setup. They fixed onboarding, conversion jumped to 12%. Revenue tripled in 6 months with the same traffic. The vanity metric almost killed them. The real metrics saved them. Meanwhile, a data team loved their metrics: "We built 47 dashboards this year!" Team got bonuses. New VP asked: "How many of those 47 have 5+ weekly active users?" They ran the numbers. Brutal: 19 dashboards had 5+ users, 28 had 2 or fewer. VP changed the metric: "Dashboards with 5+ weekly active users for 3+ consecutive months." First month: 40% pass rate. Team panicked. "This is too hard!" VP: "Good. It should be hard. It forces us to build things people actually want." Team shifted focus: spent time understanding user needs, simpler designs, better data quality, actual user testing. Month 6: 82% pass rate. Bonus: they built 60% fewer dashboards but delivered 3x the business value. Quality over quantity. Usage over activity. The halo metric forced everything else to improve.

More on Metrics & Evidence

From the book

In the book this comes to a head in a board meeting, where growth theater meets its limit. The fix is honest metrics, not better slides.

This card is one of 57 drawn from The Auction Block by Brian Cariveau — the story of an analytics leader with 90 days to fix what the cards describe.

Each leadership pattern falls into a few pitfalls most often. Find your pattern.The Analytics Leadership Mirror: free, 5 minutes, no account