Ashley Stirrup

Amazon's team was confident in their data. Then someone noticed Oprah had called Kindle her favorite thing. Between the two test periods — Website A and Website B — Oprah had told her national TV audience that the Amazon Kindle was her new favorite product. Sales spiked. The before-and-after analysis attributed the change to the website design. Oprah had driven all of it. This is the fundamental problem with before-and-after studies: the world doesn't pause while you measure. Competitors launch. Seasons change. A public figure endorses your product. An algorithm shifts. Your marketing team runs a promo. None of those show up in a "we had X, we changed Y, now we have Z" analysis. They just look like noise, or worse, they look like signal. Controlled experiments solve this by running treatment and control simultaneously. The external world affects both groups equally. What's left is the effect of your change. Ronny Kohavi (@ronnyk) makes this point plainly: in software, there's almost never a good reason not to run a controlled experiment. The infrastructure is accessible. The cost is low. The cost of trusting a before/after study is high. You're not measuring in a vacuum. You're measuring in a world that keeps moving. Build accordingly. Tomorrow: the trap that catches even experienced data teams — when correlation looks exactly like causation. Ron Kohavi. GrowthBook

Only one snapshot so far — the engagement-over-time curve appears once the daily scrape has captured this post at least twice.