Graham McNicoll

At education.com, we ran thousands of experiments. Roughly a third won. The ones that lost taught us more than the ones that won, and it took me longer than I would like to admit to see the value in those tests. A losing experiment tells you that something you had assumed that was true is not. That is incredibly valuable information- but only if your team feels safe enough to run experiments that might lose, and humble enough to want to admit it. Most do not. The test loses and someone notes it as a failure. The team moves on and tries to avoid this in the future. Nobody asks what the loss actually meant. If your team is optimizing for win rate, they are playing it safe. They are running experiments they think will win, which means they are not pushing into the areas where the real learning happens. Industry-wide, only 20 to 30% of experiments move the metric they were built to move. That number is the reality of building product. The teams who compound learning fastest are the ones who run more experiments and treat every result as a data point worth examining, regardless of the outcome. When a test loses, the important question worth asking are: what did we assume that turned out not to be true? Was our hypothesis wrong or the implementation? Use the discussion to bring up new ideas to test. What does your team do with a lost experiment?