Dismissing Anecdotes – Are Scientists Jerks?

DSC01628

When sharing an article on social media that is critical of a particular “alternative” treatment, I often get responses from friends and family that appeal to anecdotes in order to argue that the treatment is effective. These may include stories they have heard from others or a story of their own personal experience with the treatment.

For example, if I point out that a thorough review of over 3,000 scientific studies shows that acupuncture is an elaborate placebo, someone may chime in and say “well, it worked for me” or “it worked for a friend of mine,” or “it has worked for many people. How can all of those people be wrong?”

In some cases, the objections are defensive in tone. After all, who are scientists to dismiss the personal experiences of others? Do scientists think these people are untrustworthy? Do they think that these people are lying? Do they think that these people are stupid? Are scientists just arrogant and pretentious jerks who dismiss these claims because they don’t want to be told they’re wrong?

These defensive objections are certainly understandable. The experience of feeling better after a treatment can be incredibly compelling – to the point that it feels completely obvious that the treatment was effective. Hence, it seems only natural to take offense when scientists refer to these anecdotes in a pejorative manner.

Why do scientists react this way?

To understand the reason for this, it helps to first understand the logical flaws in the typical arguments such as “it worked for me.”

I admit, the formal logic and arguments that I’m about to discuss may seem dry and unpersuasive to the reader who is convinced they benefited from a treatment. After all, why should one be persuaded by such dispassionate arguments to change a belief that emotionally feels very real and true? To these individuals, I would ask the following questions:

  • Do you want to believe things that are most likely to be true?
  • If you currently hold a belief that isn’t true, would you want to discover that?

If your answers to either of these questions is “no”, then stop reading right now.

Logically Fallacious

There are two problems with the logic used in the above arguments. First, consider the argument “how could so many people be wrong?” This argument appeals to the popularity of a belief, arguing that if a large number of individuals hold the belief, then that belief is more likely to be true. In fact, this is untrue. The popularity of a belief is not a justifiable reason to conclude that the belief is true. This is a logical fallacy known as the appeal to popularity. There are countless examples in history where a large number of people believed something that turned out to be untrue.

More importantly, the bigger problem has to do with the logical fallacy known as post hoc ergo propter hoc, which I discussed in a previous post. In short, the anecdotes claiming that alternative treatments are effective are largely based on the fact that a positive outcome was observed following the administration of the treatment. As such, the fallacy here is the conclusion that the positive outcome must have been caused by the treatment because it followed the treatment.

Apart from logical fallacies, there are a number of reasons why scientists consider anecdotes to be unreliable.

Regression to the Mean

Regression to the mean is the scientific way of saying that when things move toward one extreme or another, that over time, they tend to trend back toward the average. For example, imagine you are sick with a cold but you feel better again after 10 days without taking any medication. You moved toward a state of being sick, and then trended back toward the average of feeling normal. Consider an alternate scenario: you are sick for the same amount of time, but you decide to take some cold medicine on the 9th day and you feel better the next day. Did the cold medicine cause you to feel better, or would you have gotten better anyway? There is simply no way to know based on this evidence alone. Therefore, this anecdote is not a reliable way to believe what is true.

Coincidence

To the person making the claim that an alternative treatment helped them, it may seem like a cop-out to say that it was a coincidence. However, a basic consideration of statistics and probability tells us that with large numbers of random events, some coincidences are expected.

Humans are generally terrible at comprehending large numbers and intuitively understanding randomness. For example, think of a random sequence of ten digits between 1 and 10. Chances are that the sequence you thought of is not truly random. Experiments of this sort have shown that we tend to underestimate how often two or more identical numbers will follow each other in a truly random sequence such as this one.

A good example is the case of vaccines and autism. Given the many millions of vaccine doses administered and the high frequency of autism, statistics tells us that we would expect a pretty significant number of children to begin showing signs of autism around the time a vaccine is given. It would be incredibly odd if there were no coincidences. Based on this information alone, there is no reliable way to determine whether or not there is a causal connection. This is why scientists are not persuaded by the anecdotes of parents who claim vaccines caused their kids’ autism. Unfortunately, many parents lack an understanding of statistics and they come to the erroneous conclusion that scientists are heartless shills.

Selection Bias

Humans tend to focus on unusual, interesting events while ignoring mundane ones. This is why we don’t see headlines such as “Man drives to work without getting into a car accident” and “Local couple flies across the country without incident.” Thus, anecdotes tend to focus more on these unusual events and the result is that studies have shown that we tend to overestimate how common these events actually are.

Let’s say that 100,000 people have a particular type of terminal cancer and someone invents an untested alternative treatment for this cancer. All 100,000 people try the treatment, 4,000 people go into remission, and 96,000 pass away. Following these events, we now have 4,000 people providing testimonials that this alternative treatment cured their cancer. Should we accept these testimonials? Well, consider that diagnostic tests for cancer are not perfect, and some percentage of patients will be a “false positive,” i.e. diagnosed with cancer even though they don’t have cancer. In this example, if the false positive rate for this cancer was 4%, we would expect 4,000 people to not have cancer in the first place – the same number of people who are now providing testimonials.

The lesson here is not that the claims of anecdotes are necessarily wrong, but simply that anecdotes alone are not a reliable way to determine whether or not a treatment is effective.

Confounding Factors

If someone takes multiple treatments at the same time, it is not possible to know which treatment (or combination thereof), if any, caused the positive outcome based solely on an anecdote.

For example, let’s say someone has a condition and they begin taking three different supplements at once. Then, their condition gets better and they become convinced that a particular pill caused the improvement. They then begin to tell their friends, so convinced that this one pill made them better that they omit the fact that they also took other supplements. Based on their anecdote, there is no way to be sure that other factors weren’t involved.

This phenomenon has been observed in some cancer patients who claimed that an alternative treatment cured them. Upon further investigation it was then revealed that the patient was taking other conventional treatments at the same time, any of which may have been the cause of the remission.

The Placebo Effect

Experiments have shown that believing or not believing in the effectiveness of a treatment can have real physiological effects in the body and that the more invasive a treatment is perceived to be, the stronger this effect. For example, placebo experiments have shown that two placebo pills are more effective than a single placebo pill, that placebo injections are more effective than placebo pills, and that placebo (sham) surgeries are more effective than injections or pills. This may seem completely counter intuitive, but it is a real, demonstrated effect that scientists are attempting to understand.

Therefore, when you feel like an alternative treatment worked for you or a friend, it may very well be due to the placebo effect, and not a product of the treatment itself. Hence, there is no way to know if the treatment itself actually worked based solely on an anecdote.

Summary

Understanding the limitations of anecdotal evidence is one of the greatest challenges that our society faces today. The only way to overcome this challenge is to think more critically by asking questions and considering alternative explanations. When faced with anecdotes, we need to critically evaluate their reliability in order to determine what is most likely to be true. In most cases, anecdotes alone are insufficient to determine what is true with any reliability, and in these cases we need to come to the sobering conclusion of “I don’t know.”

When a scientist dismisses your anecdote, it’s not because they think you are stupid, untrustworthy, or uneducated. It is because they are astutely aware of the limitations of anecdotal evidence, and they know that more rigorous, high quality studies that control for confounding factors and biases are required in order to verify a claim. They are simply doing what they have been trained to do, and it is likely not a personal judgment about you.

If you want to believe true things, be skeptical. The next time you’re tempted to come to a firm conclusion based on a personal experience or a friend’s anecdote, ask yourself how reliable that anecdote really is.

Mercury in Vaccines: Are We Really STILL Talking About This?

 

One basic principle of toxicology is that the dose makes the poison. This means that a substance has observable adverse effects in the body only once a certain dose is achieved. This not only applies to substances that are widely considered “harmful”, but also to substances that are generally considered “safe.”

For example, water is a molecule that is vital to the existence of life, yet it can cause serious problems if large quantities are consumed within a short period of time. While it is generally considered to be the least toxic chemical compound, it is possible to drink enough water such that the normal balance of electrolytes in the body is thrown off, which in rare cases can lead to death. Cases of water intoxication have been recorded in marathon runners as well as in participants of water drinking contests.

This may seem like an extreme example, because it is. However, there are plenty of other examples to illustrate this concept:

  • Snake venom can be toxic and lethal, however there is a dose below which there is no detectable toxic effect
  • Drugs such as acetaminophen and ibuprofen are usually benign when taken as directed but can cause serious harm in large quantities
  • Acetic acid can cause severe burns, but we regularly consume smaller, harmless doses in vinegar
  • Oxalic acid is naturally found in vegetables, including broccoli, garlic, beets, onions, carrots, celery, cucumbers, peas, tomatoes, and potatoes, yet in larger doses it causes a variety of problems
  • Nicotine, a drug found in cigarettes, is found in small quantities in plants such as tomatoes, eggplants and peppers
  • Vitamin A is an essential nutrient, yet large quantities of it can cause liver damage
  • Our bodies require magnesium, however large quantities can lead to diarrhea and even death
  • Consumption of alcohol in large quantities can lead to alcohol poisoning, while much smaller amounts cannot
  • Solanine is a compound found naturally in potatoes, apples, blueberries, bell peppers and tomatoes, yet in large quantities it can cause gastrointestinal symptoms, hallucinations, paralysis, and death

Anti-vaxxers claim that certain substances have no safe level of exposure, even though this violates basic toxicology. It is not enough to say that a substance is toxic. The question that matters is “at what dose are adverse events observed?”

Thankfully, scientific studies can answer this question. Animal studies can be performed to measure the lethal dose and toxicity of substances. Researchers can measure the LD50 (a measure of acute toxicity) which is also known as the “median lethal dose.” This is the dose of a given substance that is required to kill 50% of the test population. The lower the LD50, the higher the toxicity.

Another quantity is the “no-observed-adverse-effect-level,” or the NOAEL, which is the highest studied dose at which no adverse effects were observed in the test population. For example, if researchers were studying substance X and divided animals into four groups each receiving different doses, say 25, 50, 100 and 200 milligrams (mg), and adverse effects were observed only in the 200 mg group, the NOAEL for substance X would be 100 mg.

Scientists can also study large human populations to determine whether people exposed to a given substance experience more adverse effects compared to those that have not been exposed to it, or they can look for correlations that may be indicative of adverse events.

Chemistry is Important

Elemental (pure) sodium will explode when it comes into contact with water. Chlorine can be used to create a biological weapon and it can also be used to clean your swimming pool. Intuitively, we might be tempted to demonize any compound that contains one of these molecules, but basic chemistry shows that this fear is unfounded. Sodium and chlorine combine to form sodium chloride, otherwise known as table salt – a substance that does not produce a volatile explosion in water, does not serve as a chemical weapon or effective pool water cleaner, and has different effects on the body compared to pure sodium or chlorine.

Basic principles of chemistry tell us that it is erroneous to characterize a compound as dangerous based on the properties of its constituent molecules.

Thimerosal in Vaccines

In the late 19th and early 20th centuries, bacterial contamination in vaccines was a serious problem for health professionals. Vaccines that were delivered in multi-dose vials were sporadically prone to bacterial contamination resulting from multiple uses of the same vial. This contamination was less of a problem with single-dose vials, however the disadvantage of single-dose vials is that they are more costly to produce compared to multi dose vials.

In 1916, a tainted batch of typhoid vaccines caused 68 severe reactions and 4 deaths in South Carolina. In 1928, 12 children died after being administered a diphtheria vaccine that was contaminated.

Health professionals and researchers sought to fix this problem of bacterial contamination by adding a substance called a germicide to the vaccines. I won’t go into the finer details here, but the end result was that they discovered that ethylmercury (the principle ingredient of thimerosal) was a very effective germicide and that it could be administered in high doses (up to 20 mg per kg of body weight) without any adverse effects. Thus, beginning in the 1930’s, thimerosal was added to vaccines to prevent this bacterial contamination and save lives.

Why Fear Mercury?

Mercury is widely considered to be dangerous for humans – and with good reason. The Minamata Bay incident in Japan in the 1950’s is a good example of how mercury poisoning can be frightening and even deadly. The Chisso Corporation had a factory in the town of Minamata that produced acetaldehyde, a chemical used to make plastics. For many years, the company dumped its waste water into Minamata Bay. Eventually, there were stories of cats “dancing” and dying in the street, and strange symptoms soon appeared in humans – which in some cases proved to be fatal. Scientists eventually traced the cause of these symptoms to methylmercury poisoning. It turns out that significant quantities of methylmercury were being dumped into the bay by the Chisso Corporation. Unfortunately, at that time, seafood from the bay was the community’s primary source of protein.

Today, human exposure to methylmercury is achieved via consumption of fish and other aquatic species. Methylmercury is not efficiently excreted (eliminated) from the body and it therefore tends to accumulate up the food chain because it remains absorbed by tissues for a significant period of time.

Mercury and Autism?

Concern about thimerosal in vaccines began to take hold in the late 90’s, around the time Andrew Wakefield published his study raising concerns about the MMR vaccine. (Ironically, the MMR vaccine does not and has never contained thimerosal.) After many scientific studies showed that there was no link between autism and the MMR vaccine, anti-vaxxers then committed the informal fallacy of moving the goal posts by turning their attention to thimerosal.

In the anti-vaxxers’ eyes, thimerosal in vaccines was causing autism. This idea was championed by journalist David Kirby in his book Evidence of Harm, and by celebrities such as Robert F.Kennedy Jr. and Jim Carrey.

Capture

Problems with their Argument

Our intuition might tell us that it is reasonable to be fearful of thimerosal in vaccines. Is this true? No, here’s why.

First, we must go back to our discussion above which explained that chemicals with different structures behave differently. Readers with a keen eye (or those already familiar with this topic) will have noticed that in discussing mercury toxicity, I made reference to methylmercury, while I described thimerosal as containing ethylmercury. The difference is important and cannot be overstated. Somewhat similar to my example of sodium and chlorine behaving differently from sodium chloride, it is true that methylmercury and ethylmercury behave quite differently. Methylmercury, found in seafood, is absorbed by the body’s tissues for quite some time, while ethylmercury is efficiently eliminated from the body over a short period of time. Therefore, it is wrong to speak of methylmercury (found in seafood) and assume that its effects will be the same as that of ethylmercury (found in vaccines). Anyone who does this is demonstrating a lack of knowledge of basic chemistry. (This is also similar to methanol vs. ethanol: methanol is unfit for consumption, while ethanol in wine is enjoyed safely by many.)

Second, consider the other basic principle I covered earlier: the dose makes the poison. Anti-vaxxers may argue that the differences between methyl and ethyl mercury don’t matter, and that exposure to any dose of mercury is toxic. Of course, we know this is not true as this argument violates a basic principle of toxicology.

But what if ethylmercury is toxic at the dose given in vaccines? To support this assertion, someone once sent me a study showing that thimerosal administered to mice can cause autoimmune problems. Now that we know that the dose makes the poison, a perfectly reasonable and rational question is: what dose of thimerosal was administered to the mice? How many vaccines must a human receive to achieve a dose equivalent to that given in the study?

The lowest dose given to the mice that produced adverse effects was 147 micrograms per kg of body weight, per day. Doing some math, we can determine that to get the equivalent dose in a 150 lb human, we would need to administer approximately 10 mg of thimerosal. The vaccine with the highest dose of thimerosal is currently the flu shot, which comes in at 0.025 mg of thimerosal per vaccine dose. A 150 lb human would need to receive 400 flu shots to achieve this dose.

Further, we must account for the fact that ethylmercury is excreted by the body. Therefore, the span of time over which the vaccines are administered matters as well. Given that this dose of thimerosal was administered to mice over 70 days, the 400 vaccines for a 150 lb human would also have to be administered within a span of 70 days. In reality, childhood vaccines are spread out over many years. For example, the publicly funded vaccine schedule in the province of Ontario recommends 18 vaccines staggered over several years of childhood.

The two paragraphs above don’t even take into account two important points: many vaccines don’t contain thimerosal, and observations made in animals are not directly translatable to humans. These points aside, using this mouse study to argue that thimerosal is toxic is akin to arguing that a cup of water is toxic because drinking 24 cups of water at once can kill you. (The LD50 for water is roughly 6 litres)

What About Humans?

If thimerosal was somehow connected to the rise in incidence of autism, then we would expect autism rates to go down if the dose of thimerosal given to children were reduced. This occurred in 2001 in the United States when the CDC removed thimerosal from the childhood vaccination schedule. Proponents of the thimerosal-autism connection, such as David Kirby, predicted that the incidence of autism would plummet. They were wrong, as autism rates continued to rise even after the removal of thimerosal. California passed a law in 2006 removing thimerosal from all of their vaccines, yet the state’s autism rates continued to rise.

Epidemiological studies in humans have extensively studied the alleged connection between thimerosal and autism. A review of the scientific evidence published in 2010 concluded:

[…] studies have consistently failed to identify a cause-effect relationship between thimerosal and autism. In addition, the prevalence of autism has increased despite a decrease in the thimerosal content of vaccines; […] Despite failure to demonstrate an association, certain states continue to mandate that vaccines given to children contain no more than trace amounts of thimerosal. Epidemiologic studies continue to provide evidence that there is no association between thimerosal exposure and autism.

Of course, these findings should not come as a surprise given all of the science that has been done to investigate the toxicity of thimerosal.

Summary

Despite the wealth of epidemiological studies and thimerosal’s removal from most vaccines having no effect on autism rates, anti vaccine proponents and celebrities continue to stoke a fire that should have been extinguished over 10 years ago.

The case of thimerosal is a good example of why basic tenets and evidence from epidemiology, toxicology and chemistry need to be effectively communicated to the public.