Graham McNicoll

The return you report from experimentation is higher than the return you actually got. This is the part nobody enjoys hearing, and it is worth sitting with, because most programs are sending the wrong number upward every quarter. You run a quarter of tests. Some win, some lose, some come back flat. You take the winners, add up their measured lifts, and present that sum as what the program delivered. It looks like a reasonable calculation and leadership accepts it. Then the winner's curse arrives. An experiment wins partly because the change was good and partly because the noise broke in its favor that particular week. When you filter for winners, you are also filtering for favorable noise. Every win in the list carries some of it, and the sum carries all of it. So the aggregate comes in lower than you expect. That is a property of selecting on the outcome, and it happens to everyone who reports this way. By the way, what compounds reliably is the learnings. They are harder to put in a slide because they do not demonstrate ROI cleanly. Yet they have tremendous value as they can apply broadly across everything the team builds next. So keep track of your learning rate. That is the number worth defending, even though it is the harder one to show. The full breakdown of why summing your wins overstates the impact is at https://lnkd.in/g3Hgkcue