Coming next: The Second Factory →

Back to Library
Metrics & Evidence⚠️ Pitfall

Hope is Not a Strategy (Why Guessing is Expensive)

The Problem

Product team has two ideas for the homepage redesign. They debate for 3 weeks. Everyone has an opinion. VP picks the one they like. Launch it to everyone. Traffic drops 15%. Oops. Meanwhile, a competitor runs 5 A/B tests a week, learns what works, and iterates constantly. Guessing is expensive. Hoping your idea works is not a strategy—it's gambling with the business. Yet companies do this constantly: launch features with no tests, change pricing without experiments, redesign apps based on HiPPO (Highest Paid Person's Opinion) instead of data. It's like a manufacturing plant changing the production process without testing—sure, maybe it works, but probably you just bricked the line.

The Principle

Test everything. A/B test product changes, pricing, messaging, features—anything customers interact with. Set up experiments with control groups so you know what actually caused the change. Make experimentation a reflex, not a special event. Think manufacturing: before you change the production line, you test it on a small batch. Same here. The magic happens when you build a culture of rapid experimentation—ship fast, learn fast, kill what doesn't work, double down on what does. Less debating in conference rooms, more testing with real customers. That's how you win.

🎯

3 Steps to Take Action

  1. 1

    Make A/B Testing the Default: Nothing ships to 100% without a test. Start with 10% of traffic, monitor 5 key metrics, scale winners, kill losers fast.

  2. 2

    Build Kill Switches: Every new feature needs an easy way to turn it off. If a test fails, you should be able to revert in minutes, not days.

  3. 3

    Track Experiment Velocity: Count how many A/B tests you run per quarter. If it's <10, you're not learning fast enough. Speed up the experimentation cycle.

In Practice

An e-commerce site wanted to redesign their checkout flow. Designer made it "cleaner and more modern." Team loved it. Exec said "ship it." They did—to 100% of traffic. Conversion rate dropped 18%. Revenue tanked. Panic. Rolled back. Lost $4M in a week. New CPO instituted a rule: "Nothing ships to 100% without a test." Next redesign, they A/B tested to 10% of traffic. Conversion dropped 12%. Killed it immediately, lost only $50K instead of $4M. Tried a different design. This one lifted conversion 6%. Rolled to 100%. Revenue up $8M that quarter. Ten more A/B tests that quarter: 3 winners, 7 losers. But they caught losers early and scaled winners fast. That's the power of experimentation.

More on Metrics & Evidence

From the book

In the book this comes to a head in a board meeting, where growth theater meets its limit. The fix is honest metrics, not better slides.

This card is one of 57 drawn from The Auction Block by Brian Cariveau — the story of an analytics leader with 90 days to fix what the cards describe.

Each leadership pattern falls into a few pitfalls most often. Find your pattern.The Analytics Leadership Mirror: free, 5 minutes, no account