Graham McNicoll

Building your own experimentation stack is a fun engineering challenge until the day it isn't. When I was CTO at education.com, we evaluated every tool on the market. Most could only answer about half our data questions, when we really used them. We wanted to use our own data, and to be able to interrogate the data and results we were seeing. None of the platforms we looked at gave us that. So we built our own. We had something functional within a few months. It had missing features, but it worked well enough to use. What we didn't anticipate was what came next. Every time the data schema changed, someone had to fix the platform. Every team that started using it had new questions, new slicing requirements, new things they wanted to see. That list never ends. Three years in, some of our best engineers were spending a significant portion of their time maintaining infrastructure that wasn't our product. That is the real cost of building your own experimentation stack. It's not the initial build. It's the compounding technical debt of owning something that was never your core focus. There is also an inherent risk that you'll discover a bug in your home grown experimentation platform that invalidates every decision you've made with it. We built GrowthBook because we wanted teams to have a in-house experimentation platform without having to build it or maintain it. Providing full transparency and working with your existing data (warehouse-native). What is one internal tool your team built that eventually became a drain on your best engineers?

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.