Ashley Stirrup

A Microsoft team once presented this finding internally about Office 365: "Users who use our feature are half as likely to churn." The implication was obvious — the feature was working. Ship more of it. Double down. There was just one problem. The finding was completely wrong. Ron Kohavi explains this in his A/B testing course and it's a great example of how smart teams fool themselves with data. The real explanation? Heavy users of Office 365 both use more features and churn less, not because the features cause retention, but because heavier engagement predicts both behaviors. The feature team was measuring their own users against the lightest, most at-risk cohort and calling it causation. He gives another example from the same product: users who see more error messages in Office 365 actually have lower churn rates. Not because error messages help. Because the people who see error messages are power users who live in the product all day. They churn less for the same reason, they're deeply embedded. Strip away the causation, and you're left with a correlation that points exactly the wrong direction for decision-making. This is why Kohavi is so direct about it: "In some domains, like new features in software, there's rarely a good excuse for not running a controlled experiment to get trustworthy results." Observational data tells you what happened. A controlled experiment tells you why. Confusing the two doesn't just lead to bad decisions, it leads to confident bad decisions, which are the worst kind. If your team is using engagement metrics to "prove" that features work without ever running an experiment, you're not measuring outcomes. You're measuring selection bias. Have you ever shipped something based on correlation data that turned out to be wrong when you actually ran the experiment? What did you learn?

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.