Ashley Stirrup

Intuition is the most dangerous tool in gen AI. It could have cost Khan Academy their AI tutor's math accuracy. Disabling their AI tutor's math verification tool seemed reasonable. Latency was a real problem and the model had gotten better. Math errors doubled. Fortunately, Khan Academy was rigoursly A/B testing and iterating. Dr. Kelli Hill, Head of Data at Khan Academy, shared that story on The Experimentation Edge as the moment that crystallized something her team now treats as non-negotiable: with gen AI, you cannot predict what a change will do until you measure it. A sentence in a prompt, a system instruction tweak, a model swap. All of it gets tested. The teams moving fastest on gen AI aren't the ones skipping experimentation. They're the ones who figured out it's the only way to move with any confidence. #experimentation #abtesting #productmanagement

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.