Graham McNicoll

Win rates across the industry sit around 20%, and I do not think AI is going to improve on that. A lot of teams have quietly started using AI to examine experiment ideas, or even as synthetic users to simulate experiment results. On the idea side, you put 100 ideas in front of a model and it hands back 10 that look sensible. This is letting the AI apply judgement. AI can probably tell you an obviously bad idea from a not obviously bad idea. But ask it to separate a good idea from a great one, or a flat idea from one that actually moves a number, and it will guess. It guesses using historical user behavior and roughly the reasoning you would have used yourself. Which is the whole problem. Historically we are bad at guessing what users want. Assigning this judgement to an AI doesn't solve this problem, it just moves it. A model trained on our data inherits our biases along with it. There is no reason to expect it to beat the historical precedents on best practices we already know. There is a version of this that is worse than no help at all. Artificial intelligence helps you feel artificially intelligent. If you are not a real expert in the area, the output reads like genius. Someone who is good at it looks at the same thing and says that is not quite right. Getting 'statistical' results from synthetic experiments makes subjective things feel objective. Where in reality, we have just offloaded the guessing to AI. It's still guessing. So use it where it is strong. If you have too many ideas you can't test, using AI to whittle down 100 ideas to 10 you can actually run is valuable. Use it to find bugs in an implementation by actually trying to use the product. But use large grains of salt when using AI to simulate experiments. (Some more conversation about trust worthy experiments: https://lnkd.in/gfT6Yi3D )