Skip to content

Why does one anecdote beat a meta-analysis in an argument?

5 replies·

Recurring pattern. Somebody posts a careful summary of the evidence. Somebody replies "my mate did it and his knee is fine now". The second one wins the room.

Why is that so reliable, and is there anything to do about it?

Because a meta-analysis is a claim about a distribution and an anecdote is a story about a person, and human reasoning was not built for the first thing.

A story has an actor, a sequence, and a consequence. It is memorable, it is transmissible, it is easy to picture. A pooled effect size with a confidence interval has none of those properties and cannot acquire them.

The uncomfortable part is that the anecdote is not worthless. It is a genuine observation. It is just one draw from a distribution, presented with all the vividness of a conclusion about the distribution.

And there is a selection effect that makes it worse than a random draw.

You only ever hear the anecdotes that are worth telling. Nobody says "I took it for eight weeks and honestly I could not tell". So the anecdotes reaching you are pre-filtered for being surprising, which means they are pre-filtered for being unrepresentative.

The distribution you are sampling from in a pub is not the distribution the meta-analysis measured. It is the tail of it.

That is the cleanest statement of it I have seen. The anecdote is not a bad sample, it is a good sample of a different population — "things people bothered to say".

On what to do about it: not much, in an argument. But the one move that sometimes works is asking for the anecdotes that go the other way. "How many people do you know who tried it and nothing happened?"

Usually the answer is that they have no idea, and occasionally that lands.

It also explains why the pinned thread in Experiences spends a paragraph begging for negative reports. Not politeness — it is trying to unfilter the sample.

Sign in or create an account to reply.