Graham McNicoll

I talk to a lot of teams who are proud of how much they experiment. A company you would recognize once told me they were doing it well. When I asked what that meant, the number was three tests a quarter. Three! Their product team ships far more than three changes a quarter. Most of what they release never gets tested at all. Believing you experiment a lot is exactly what stops you from experimenting more. If you think you are already know what "good" looks like, and you're doing it, there is nothing to fix. The teams at the frontier look nothing like this. The largest consumer platforms run thousands of experiments at once, testing things most people would never question, down to a few pixels of spacing on a single screen. Three tests a quarter while shipping constantly means deciding almost everything by gut feel, and if you're familiar with experimentation at all, you'll know how bad we can be at this. How many experiments did your team actually run last quarter, and how many things did you ship without one?