Graham McNicoll

I see experimentation leads run two or three tests a month and feel like they have figured this out. They haven't. Running an experiment and running an experimentation program are two different jobs. The first thing I see that breaks is having the experimentation team become a bottleneck (a "bottleneck of excellence", as one experimentation leader told me). If you are running a lot of experiments, you cannot be in the critical path of every one of them. Your team has to launch, analyze, and ship without you. The moment that opens up, the harder problem shows up. Different people on my team had different standards for a high-quality experiment, a good goal metric, and a valid readout. Quality control at scale is the harder half, and most teams I talk to do not realize they have this problem until they already do. By then, trust is already eroded, and trust takes a long time to rebuild. If your team is past the first few tests and starting to feel this, DM me. Happy to talk through what tends to break first.