When sharing an article on social media that is critical of a particular “alternative” treatment, I often get responses from friends and family that appeal to anecdotes in order to argue that the treatment is effective. These may include stories they have heard from others or a story of their own personal experience with the treatment.
For example, if I point out that a thorough review of over 3,000 scientific studies shows that acupuncture is an elaborate placebo, someone may chime in and say “well, it worked for me” or “it worked for a friend of mine,” or “it has worked for many people. How can all of those people be wrong?”
In some cases, the objections are defensive in tone. After all, who are scientists to dismiss the personal experiences of others? Do scientists think these people are untrustworthy? Do they think that these people are lying? Do they think that these people are stupid? Are scientists just arrogant and pretentious jerks who dismiss these claims because they don’t want to be told they’re wrong?
These defensive objections are certainly understandable. The experience of feeling better after a treatment can be incredibly compelling – to the point that it feels completely obvious that the treatment was effective. Hence, it seems only natural to take offense when scientists refer to these anecdotes in a pejorative manner.
Why do scientists react this way?
To understand the reason for this, it helps to first understand the logical flaws in the typical arguments such as “it worked for me.”
I admit, the formal logic and arguments that I’m about to discuss may seem dry and unpersuasive to the reader who is convinced they benefited from a treatment. After all, why should one be persuaded by such dispassionate arguments to change a belief that emotionally feels very real and true? To these individuals, I would ask the following questions:
- Do you want to believe things that are most likely to be true?
- If you currently hold a belief that isn’t true, would you want to discover that?
If your answers to either of these questions is “no”, then stop reading right now.
Logically Fallacious
There are two problems with the logic used in the above arguments. First, consider the argument “how could so many people be wrong?” This argument appeals to the popularity of a belief, arguing that if a large number of individuals hold the belief, then that belief is more likely to be true. In fact, this is untrue. The popularity of a belief is not a justifiable reason to conclude that the belief is true. This is a logical fallacy known as the appeal to popularity. There are countless examples in history where a large number of people believed something that turned out to be untrue.
More importantly, the bigger problem has to do with the logical fallacy known as post hoc ergo propter hoc, which I discussed in a previous post. In short, the anecdotes claiming that alternative treatments are effective are largely based on the fact that a positive outcome was observed following the administration of the treatment. As such, the fallacy here is the conclusion that the positive outcome must have been caused by the treatment because it followed the treatment.
Apart from logical fallacies, there are a number of reasons why scientists consider anecdotes to be unreliable.
Regression to the Mean
Regression to the mean is the scientific way of saying that when things move toward one extreme or another, that over time, they tend to trend back toward the average. For example, imagine you are sick with a cold but you feel better again after 10 days without taking any medication. You moved toward a state of being sick, and then trended back toward the average of feeling normal. Consider an alternate scenario: you are sick for the same amount of time, but you decide to take some cold medicine on the 9th day and you feel better the next day. Did the cold medicine cause you to feel better, or would you have gotten better anyway? There is simply no way to know based on this evidence alone. Therefore, this anecdote is not a reliable way to believe what is true.
Coincidence
To the person making the claim that an alternative treatment helped them, it may seem like a cop-out to say that it was a coincidence. However, a basic consideration of statistics and probability tells us that with large numbers of random events, some coincidences are expected.
Humans are generally terrible at comprehending large numbers and intuitively understanding randomness. For example, think of a random sequence of ten digits between 1 and 10. Chances are that the sequence you thought of is not truly random. Experiments of this sort have shown that we tend to underestimate how often two or more identical numbers will follow each other in a truly random sequence such as this one.
A good example is the case of vaccines and autism. Given the many millions of vaccine doses administered and the high frequency of autism, statistics tells us that we would expect a pretty significant number of children to begin showing signs of autism around the time a vaccine is given. It would be incredibly odd if there were no coincidences. Based on this information alone, there is no reliable way to determine whether or not there is a causal connection. This is why scientists are not persuaded by the anecdotes of parents who claim vaccines caused their kids’ autism. Unfortunately, many parents lack an understanding of statistics and they come to the erroneous conclusion that scientists are heartless shills.
Selection Bias
Humans tend to focus on unusual, interesting events while ignoring mundane ones. This is why we don’t see headlines such as “Man drives to work without getting into a car accident” and “Local couple flies across the country without incident.” Thus, anecdotes tend to focus more on these unusual events and the result is that studies have shown that we tend to overestimate how common these events actually are.
Let’s say that 100,000 people have a particular type of terminal cancer and someone invents an untested alternative treatment for this cancer. All 100,000 people try the treatment, 4,000 people go into remission, and 96,000 pass away. Following these events, we now have 4,000 people providing testimonials that this alternative treatment cured their cancer. Should we accept these testimonials? Well, consider that diagnostic tests for cancer are not perfect, and some percentage of patients will be a “false positive,” i.e. diagnosed with cancer even though they don’t have cancer. In this example, if the false positive rate for this cancer was 4%, we would expect 4,000 people to not have cancer in the first place – the same number of people who are now providing testimonials.
The lesson here is not that the claims of anecdotes are necessarily wrong, but simply that anecdotes alone are not a reliable way to determine whether or not a treatment is effective.
Confounding Factors
If someone takes multiple treatments at the same time, it is not possible to know which treatment (or combination thereof), if any, caused the positive outcome based solely on an anecdote.
For example, let’s say someone has a condition and they begin taking three different supplements at once. Then, their condition gets better and they become convinced that a particular pill caused the improvement. They then begin to tell their friends, so convinced that this one pill made them better that they omit the fact that they also took other supplements. Based on their anecdote, there is no way to be sure that other factors weren’t involved.
This phenomenon has been observed in some cancer patients who claimed that an alternative treatment cured them. Upon further investigation it was then revealed that the patient was taking other conventional treatments at the same time, any of which may have been the cause of the remission.
The Placebo Effect
Experiments have shown that believing or not believing in the effectiveness of a treatment can have real physiological effects in the body and that the more invasive a treatment is perceived to be, the stronger this effect. For example, placebo experiments have shown that two placebo pills are more effective than a single placebo pill, that placebo injections are more effective than placebo pills, and that placebo (sham) surgeries are more effective than injections or pills. This may seem completely counter intuitive, but it is a real, demonstrated effect that scientists are attempting to understand.
Therefore, when you feel like an alternative treatment worked for you or a friend, it may very well be due to the placebo effect, and not a product of the treatment itself. Hence, there is no way to know if the treatment itself actually worked based solely on an anecdote.
Summary
Understanding the limitations of anecdotal evidence is one of the greatest challenges that our society faces today. The only way to overcome this challenge is to think more critically by asking questions and considering alternative explanations. When faced with anecdotes, we need to critically evaluate their reliability in order to determine what is most likely to be true. In most cases, anecdotes alone are insufficient to determine what is true with any reliability, and in these cases we need to come to the sobering conclusion of “I don’t know.”
When a scientist dismisses your anecdote, it’s not because they think you are stupid, untrustworthy, or uneducated. It is because they are astutely aware of the limitations of anecdotal evidence, and they know that more rigorous, high quality studies that control for confounding factors and biases are required in order to verify a claim. They are simply doing what they have been trained to do, and it is likely not a personal judgment about you.
If you want to believe true things, be skeptical. The next time you’re tempted to come to a firm conclusion based on a personal experience or a friend’s anecdote, ask yourself how reliable that anecdote really is.
The Skeptics' College 