Graham McNicoll

Most product teams I talk to know how to run an experiment. They have the tooling, a primary metric, and a process for calling a result. What I see less of is a clear answer to: what are we not measuring? A primary metric tells you whether the thing you were watching moved. It does not tell you what happened to everything adjacent to it. Teams get very good at finding wins in the data they track, and gradually lose visibility into the data they don't. That gap compounds quietly. A test wins. The feature ships. Six months later something is off, and nothing in the experiment results pointed there. It's very common for a new feature to "win" and actually cannibalize other parts of your product. The more useful question is whether the metrics you are using would surface the problems your primary goal metric will never show. Did your downstream metrics change? (ie, support rate, or return rate), Did you damage other well functioning funnels? If you want to talk through what this looks like in your stack, send me a DM.

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.