Ashley Stirrup

Most experimentation programs are confidently wrong - unless they have a plan. The webinar "Designing Experiments for Long-Term Growth" with Ron Kohavi and Luke Sonnet was a hit yesterday. My 4 takeaways: Naive metrics can point you in exactly the wrong direction. Bing's ranking bug increased queries and revenue by showing bad results — a naive OEC would have told you to fire the relevance team. Shipping "flat" results is organizational self-deception. It adds maintenance costs, code complexity, and path dependence for zero proven value. A dashboard full of metrics without a decision framework is just expensive noise. If you haven't formalized how conflicting signals translate to ship or don't ship, you're making million-dollar decisions on vibes. The fix is unglamorous: write down your long-term goal, write down what you can measure, and draw the line between them before you see results. Revisit it quarterly because your model was probably wrong the first time. Check out the comments for a link to the recording!

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.