Ashley Stirrup

A slide made it all the way to a VP all-hands at Microsoft. It said: "New users who use our new feature are half as likely to churn compared to new users who don't use it in the next 30 days." The conclusion: the feature reduces churn and is critical for retention. It was wrong. Here's what was actually happening: the users who adopted the new feature were already the heaviest, most engaged users — people who were naturally exploring the product, using more of it, and already liked it. They were less likely to churn regardless. Not because of the feature. Because of who they already were. The feature itself? It might improve retention. It might worsen it. It might do nothing. The observational data alone couldn't say. What looks like causality is just selection bias in disguise. To make this concrete, Ron Kohavi offered a parallel example: users who see error messages in Office are also less likely to churn. Not because errors help retention, but because heavy users hit more edge cases and see more errors. They're power users. They were staying anyway. The lesson isn't "don't track feature adoption." It's that "users who use X churn less" is almost never a causal statement. Your heaviest users use everything. They also churn less. Conflating those two facts leads product teams in the wrong direction...confidently. Without a randomized controlled experiment, you cannot separate the effect of the feature from the effect of who chose to use it.

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.