Graham McNicoll

Think about how many layers sit between you and an actual user. User research tells you what people want. Personas abstract real user segments. Dashboards average behavior. And, increasingly, evals where one model grades the output of another. Each of these is useful. Each is also a proxy, and every proxy introduces a bias. The danger is treating the output of a layer as the truth instead of as an estimate. Skipping the check before you trust it is the actual risk. I come from a physics background, so the framing I keep returning to is error bars. In physics, a value with no sense of its uncertainty is incomplete. The same is true of your data and abstractions. You need to know how much it could be lying to you. The only way to find out is to validate each layer against reality. Run the experiment. Compare what the proxy predicted against what real users actually did. When they disagree, you have learned something about the layer, and that is often more valuable than the original answer. You may also find that the abstraction into a proxy is flattening key user segments. Which of the proxies your team relies on have been checked against real user behavior, and which ones do you just trust? That gap is what experimentation is great at. See more at growthbook.io.