How not to compare statistics

I’ve spent the last five years learning how to interpret research, I know it isn’t easy.  I didn’t always know this.  I used to think I could read the conclusions of a paper, check out a few things and either incorporate it or ignore it.  The problem was, just like all humans, my trust in a paper was more related to how closely it resembled what I already believed than anything within the design of the study.

Now when I read a paper I go through a long process of making decisions;

  • What exactly is the research question?
  • What theory does this study build from?
  • What assumptions does the underlying theory make?
  • Is this design appropriate for this framework?
  • Is this population appropriate for this question?
  • Have they designated outcomes a priori?
  • Are they using appropriate statistical tests?
  • What are the sources of bias?
  • Will the biases move the estimate away from the null?
  • Do the results answer the research question?
  • Do the results indicate a clinically relevant difference?
  • Do the conclusions make sense given the framework and underlying theory?
  • What does this study add to what is known on this topic?

No, this list is not complete.  No, I can’t teach you to do this…remember this is a process of learning I’ve been working on for over five years.  I’ve learned to read as a clinician, as an epidemiologist, as a policy maker, and as a designer of research.

But there are some simple mistakes you can learn to recognize when you see them on your social media feed or in blogs.  I want to show you one I saw this week. The specific numbers do not matter, we will be looking at the concept of an unequal comparison.

The comparison I saw was this:

The number of deaths/injuries from a specific vaccine in the # years:  X

The number of deaths/injuries from the illness in the # years: 0

The comments on this post were evidence that people thought the appropriate comparison should be: because X > 0, people should not get the vaccine.

Why is this an unequal comparison?  Because this pretends to be comparing the risks of being vaccinated to the risks of not being vaccinated; but it only shows the total risks to being vaccinated.

What do I mean?

To compare vaccination to no vaccination you need to compare the total number of deaths/injuries in a vaccinated population (which the post in question included); to the total number of deaths/injuries in the same population if they were not vaccinated (which the post in question did not include).

So the comparison should read:

The total number deaths/injuries in a vaccinated population in # years: X

The total number deaths/injuries in a non-vaccinated population in # years: Y

It is only from this comparison that you can determine the risks of vaccination vs. no vaccination. You cannot have both the protection of being vaccinated (0 injuries/death), and avoid the risks of being vaccinated (X) because you can only be vaccinated or not vaccinated.  You either accept the full risk and benefit of vaccination, or you accept the full risk and benefit of not being vaccinated.

Now you are armed with one more technique for decision making, and you can apply it to all types of scientific comparisons.  Next time you read a social media post or a blog that gives a comparison, take a step back and figure out if it is unequal. If it is unequal, find the numbers to make it equal and then decide if you would make the same conclusions about that decision.


Jennifer Vanderlaan (Author)