Assessing Study Quality

We are nearly done with our discussion of statistics, so I wanted to take a day to discuss study quality. When researchers talk about the quality of a study they are considering the quality of the total package, not only the statistical significance of the findings. In fact, the quality of the study will affect the value the findings.

There are two documents you should be familiar with before you begin to assess the quality of a study.  The first is the CONSORT Statement and the second is the STROBE statement. 

CONSORT and STROBE represent the consensus of researchers on what information must be provided when reporting a study. CONSORT is written specifically for controlled trials and STROBE is for observational studies. Look through the checklists available on their websites and you will quickly see assessing the quality of a study requires a thorough reading of each section of an article.

If a study fails to report any of the information listed in these checklists without a rationale for why it does not apply, the study is generally considered to be low quality and this puts doubt into the value of the findings. Why? Because without this information you cannot accurately judge the validity of the research and its applicability to your situation.  For example, if the researchers say they randomly selected groups but do not tell you how the randomization was performed you may wonder if findings are due to a failed randomization protocol.

Quality Assessment Tools

There are some online tools you can use to help you learn to evaluate the quality of a study.  You can find tools at the Effective Public Health Practice Project, Duke University’s Introduction to Evidence-Based Practice, and the Center for Evidence-Based Medicine. Spend some time getting familiar with these websites and the tools they offer.  Then choose a study you like and review it using the most appropriate tool from one of these programs.  The more studies you review, the better you become at appraising study quality.

The Birth Worker Survey

Because we only talk about how the different aspects of a study play out in the Birth Worker Survey, we don’t actually have an article to assess for quality.  But we do know some things that can help us determine if the results discussed in any of our previous posts are valid.

The most important thing we know is that the sample used in the survey is biased, which makes all the results invalid. If we had further doubts, we know that in our controlled study the control group was contaminated making those results invalid. Finally, we know the sample is too small which prevents us from finding any significant results even if they existed.

Next week we can talk about just a few more types of statistics which might be used.