Graham McNicoll

A Hacker News thread a while ago argued that A/B testing is unethical. I believe their framing was wrong, but I am curious what the product leaders on my feed think. The objection was around being included in an A/B test without consent. They invoked the language of medical research: drug trials and clinical studies. Quite often, people opt in to be included in the trial, even if they don't know what they are getting. Product A/B testing is done differently. When a product team runs a test, they are comparing variants of a feature. Someone has a hypothesis that variant B will improve performance, or improve the user experience, or make the product better in some measurable way. The key difference is that the alternative is not "do not experiment." The alternative is to ship untested changes to 100 percent of users and hope it works. This is how products were built before A/B testing. Which of those two is more respectful of the people using your product? The opt-in argument has an edge case worth taking seriously. You could test only on users who agreed to be tested, but you would immediately bias your sample. The people who opt in are not representative of the people who will use the feature. Your result tells you something about a self-selected group, not about your actual user base. Maybe we just need a better word. Product improvement testing. Controlled rollouts. Validated release. Something that does not carry the weight of medical research. The underlying practice is not unethical. The word "experimentation" is borrowed from another field and comes with baggage. I am genuinely curious how the people on my feed think about this. If you have a better word for what we are doing, drop it in the comments. And if you think the ethical framing is correct, I want to hear that argument too.