Graham McNicoll

When an experiment fails, there should be a discussion about if the idea was bad or if it was just the implementation that was bad or even if it was a bad test. Sometimes this turns into an argument. Sorting through all the possible ways to end up with a failed experiment is hard, and usually takes a great deal of humility and experience to come up with a correct next steps. If you lack this judgement and experience, here's what I would recommend: think about it from probability. Take an idea that has already failed once. There is some probability the test was the problem and some probability the idea was, or the implementation itself. Compared with a greenfield idea nobody has tried, the failed one carries a chance of being genuinely bad that the new idea has not yet earned. Its expected value of the old idea is lower. That holds even when your instinct says the test was unfair. So the default should be to test something new. Go back to a failed idea only when the potential impact is high enough to justify spending at-bats on worse odds, and when you do go back, run a few variants rather than the same version again. The uncomfortable part is how badly this cuts against the way teams feel about their own ideas. The one you fought for is the one you want to re-run. There is a good write-up on reframing a loss as a result at https://lnkd.in/gx9whwt8