@thane16
Excellent question.
It means that one needs to be very, very cautious of generalising from personal experience. That doesn’t in any way negate those experiences, but underlines that it is from a specific point of view in a specific set of circumstances. It is, in essence, storytelling. Now, nobody (least of all here!) is going to denigrate the importance of storytelling, but personal anecdote is subject to a lot of thinking errors.
The most common of these is confirmation bias (you are thinking of a tune, turn the radio on and that tune is playing and we think: “Whoa…freaky”. Not telepathy, confirmation bias because we give zero weight to the things that don’t happen, the 1000s of times we have turned the radio on and it wasn’t playing a tune we were thinking about).
So to count as evidence we need to eliminate these flaws as best we can.
We need, say:
- multiple examples,
- that are representative of the population and chosen at random
- in which the experimenter is blind to the selection
- And in the case where the sample is people, the people are also blind (so don’t know if they are getting the medicine or a placebo) – these two together is what is meant by double blind.
- where the sample is big enough to be meaningful
- and using methods that someone else can replicate (with their own sample) to see if they get the same results.
Even then there are still traps (there’s a thing call p-hacking where the experimenter keeps sampling until they get the result they expect (or want: 8 of to 10 cats prefer whiskas (in 1 test, ignoring the 50 where the cat just scoffed on whatever they were offered first)). Usually the dishonest ones are obvious (but not always) but the honest errors are far harder to spot.
But the tl;dr is one example tells us nothing of general applicability, no matter how compelling. Some time ago the poster formally known as puro told me that there is a Crohns severity scale on which the average is 22, and she is a 90. To which:
a) OUCH!
and
b) we cannot generalise from her experience because it is such an outlier;
c) that knowledge was gathered the methods outlined (VERY briefly) above. So we know immediately why she is a walking medical emergency, and also something of a miracle.
(See what I did there?)
