Ashley Stirrup

Getting numbers is easy. Getting numbers you can trust is hard. We hosted Ron Kohavi (co-author of Trustworthy Online Controlled Experiments, ex-Amazon, Microsoft, Airbnb) and Luke Sonnet, our head of experimentation, to pull apart a real published experiment that claimed a 44.8% lift with "99% confidence." It failed four independent checks at once. The 54,000 "users per variant" were really about 300 once you counted who actually saw the change. That left the test with roughly 7% power, a false positive risk near 87%, and a sample ratio mismatch more significant than the "win" itself. The uncomfortable part: this wasn't a uniquely bad experiment. It looks like a large share of the A/B test results published today. We turned the session into two checklists you can steal: what to do during setup (power, MDE, alpha, sizing for the triggered population) and what to check before you ship (sample ratio mismatch, false positive risk, A/A tests, Twyman's Law). See the link in the comments to the full write-up.