Graham McNicoll

PMs ship experiment winners too early. A metric moves 3%, the test gets called, and the PM moves to the next item on the roadmap. On paper, it looks like a win. In practice, a lot of learning just got left on the table. When I dig into results like this, the 3% average is often hiding a more interesting picture. One segment had a strong positive response. Another segment was hurt badly enough to almost cancel the first one out. The net came out positive, so the test got called. That not completely bad, its just incomplete. The next move is a follow-up test on the segment that responded negatively. Adjust the experience for that group and rerun. I have seen that the second test takes a 3% win to 7 or 8% once both segments are optimized. Slice your next experiment result by segment. Start at growthbook.io.