Graham McNicoll

Three years into building our own experimentation platform at my last company, we found a bug in the statistical engine. The question that followed was not a small one. Did we just invalidate every product decision we had made for the past three years? Had we been making calls on broken math? We were a team of engineers and data experts. Statistical analysis at the level required for rigorous experimentation is a different discipline. That is where silent errors live. The kind you do not find until years of decisions are already made on top of them. We did a deep retroactive analysis. Fortunately for us the bug was minor and the conclusions held when reanalyzed. But the potential was real, and it forced a question we had been avoiding: were we actually qualified to be building this? This is what most engineering teams do not price in when they choose to build their own. The initial build is the easy part. The statistical layer is where the real cost hides. Validating the math and maintaining an audit trail that you can actually trust takes years, and you may not know something was wrong until the damage is done. I built GrowthBook because I did not want another team to sit with that question. GrowthBook's statistical library is open source and has been reviewed by data science teams across the industry. You can read the code and reproduce every result against your own data. Take a look at how we handle the stats at growthbook.io