Graham McNicoll

Most product teams use evals to figure out if their AI tools or products are working correctly. And quite often, those evals can lie to you. Evals show you how your models perform under preset conditions, usually based on real user behavior, but it's synthetic. It's canned. You can tune for that one set of users and actually be making it worse for another set, or the evals you created originally don't apply anymore because users are using it differently now. So they can silently cause your product to be suboptimal, because you're tuning for the evals and not for real-world performance. Use evals as a sanity check, but also sanity check your evals. If your evals and your user data disagree, where it passes the eval and looks better but then performs worse with real users, you have a real problem. Keep testing with real users to figure out if your evals are actually lying to you.