Graham McNicoll

One very large financial company, a name you would recognize, told me they were doing experimentation well. They were running three tests a quarter. The next conversation I had was with a leader running 60,000 tests at a time. Both told me they were doing experimentation well, yet only one was. When we were building GrowthBook, we asked hundreds of companies the same question, and the answers were wildly inconsistent. Velocity is an important benchmark for the health of an experimentation program. Without a number, "doing it well" is just a feeling. How many experiments did your team run last quarter, and how does that compare to the companies you are competing with for the same users? Do you have a good learning rate? Did you choose a good mix of ambitious and simple tests? If your competitors have more at-bats than you, they are compounding their learning rate while you are still debating whether to ship the next test. They are more likely to win in the long term. How many tests did your team run last quarter?