Hope is Not a Strategy (Why Guessing is Expensive)
The Problem
Product team has two ideas for the homepage redesign. They debate for 3 weeks. Everyone has an opinion. VP picks the one they like. Launch it to everyone. Traffic drops 15%. Oops. Meanwhile, a competitor runs 5 A/B tests a week, learns what works, and iterates constantly. Guessing is expensive. Hoping your idea works is not a strategy—it's gambling with the business. Yet companies do this constantly: launch features with no tests, change pricing without experiments, redesign apps based on HiPPO (Highest Paid Person's Opinion) instead of data. It's like a manufacturing plant changing the production process without testing—sure, maybe it works, but probably you just bricked the line.
The Principle
Test everything. A/B test product changes, pricing, messaging, features—anything customers interact with. Set up experiments with control groups so you know what actually caused the change. Make experimentation a reflex, not a special event. Think manufacturing: before you change the production line, you test it on a small batch. Same here. The magic happens when you build a culture of rapid experimentation—ship fast, learn fast, kill what doesn't work, double down on what does. Less debating in conference rooms, more testing with real customers. That's how you win.
3 Steps to Take Action
- 1
Make A/B Testing the Default: Nothing ships to 100% without a test. Start with 10% of traffic, monitor 5 key metrics, scale winners, kill losers fast.
- 2
Build Kill Switches: Every new feature needs an easy way to turn it off. If a test fails, you should be able to revert in minutes, not days.
- 3
Track Experiment Velocity: Count how many A/B tests you run per quarter. If it's <10, you're not learning fast enough. Speed up the experimentation cycle.
In Practice
An e-commerce site wanted to redesign their checkout flow. Designer made it "cleaner and more modern." Team loved it. Exec said "ship it." They did—to 100% of traffic. Conversion rate dropped 18%. Revenue tanked. Panic. Rolled back. Lost $4M in a week. New CPO instituted a rule: "Nothing ships to 100% without a test." Next redesign, they A/B tested to 10% of traffic. Conversion dropped 12%. Killed it immediately, lost only $50K instead of $4M. Tried a different design. This one lifted conversion 6%. Rolled to 100%. Revenue up $8M that quarter. Ten more A/B tests that quarter: 3 winners, 7 losers. But they caught losers early and scaled winners fast. That's the power of experimentation.
