A or B, not A/B
A/B testing can turn into a way to skip the thinking. Sometimes you have to pick A or B.
I'm currently looking for a new "home" for my knowledge base, and then I came across anytype. Which is what triggered this thought.
I like opinionated software. It attracts the right people and repels the wrong ones, so you (as an engineer) can focus on the group of people that actually matters. When a product has a clear perspective you don't have to wonder what it's trying to be, you just need to figure out if your opinion matches.
A/B testing is the opposite of that. And it rubs me the wrong way.
# It is great, in theory...
A/B testing sounds rigorous. Data-driven! Scientific! You're testing hypotheses instead of guessing. Which sometimes is true.
But something about it has always bothered me, or at least about how I've seen it implemented and abused. It changes the question from "what should we build?" to "let's see what performs better", and those are different questions. The first one takes deep thought, conviction and a real sense of what's right for your users. The second one just takes two variants and enough traffic.
When A/B testing becomes the default the incentives change. Instead of thinking a feature through before shipping it, you ship two half-baked versions and see which one "wins". Thinking is slow and A/B tests are fast. And stakeholders ain't got time, you know, time is money...
So you end up with a product shaped by whatever happened to perform better in a two-week experiment, instead of a coherent idea of what it should be.
(And btw, some portion of your users got the shitty experience along the way. Where's the care in that?)
# What the data doesn't tell you
A/B tests optimize for what you can measure: clicks, conversions, time on page. They can't measure trust, craft, or whether a feature should exist at all. So you optimize for the numbers, and the numbers tell you what people clicked. They don't tell you what people respected.
But look... I'm not saying never experiment. Trying stuff is good, learning from users is good. I mean "we believe X is the right approach, let's validate it" is a different thing from "we have two ideas, let's see which gets more clicks". The first is thoughtful, the second is outsourcing your decisions.
Pick A. Or pick B. But pick it because you thought about it, instead of hoping the data will think for you. Opinionated software comes from opinionated people making actual decisions, and that's the kind I want to use and build.