Why Sample Size Matters

On Monday we talked about regression techniques and I explained that finding a statistically significant result would be difficult because of our sample size.  Today we’ll explore how sample size affects the results you will get.

To understand the importance of a sample size, you need to think back to our discussion about p-values and confidence intervals. If you haven’t read those posts yet, go back and look over them now.

Remember, the larger your sample, the smaller the variation from the mean.  The problem with sample size is that if your sample is too small, the normal variation within the sample will be so large that your confidence intervals will always overlap. If your confidence intervals overlap, you do not have a significant finding — even if the result would be significant with a large enough sample.

The problem with large samples is the opposite, your confidence intervals may be so small overlap is nearly impossible. With too large a sample, you will find significance in things that are irrelevant. For example, with a large enough sample size you might be able to find that a certain labor technique reduces first stage by 15 minutes — and this could be a statistically significant finding. The question to ask yourself about that finding would be — is it clinically significant?  Is there anything about a 15 minute difference in first stage for normal healthy women that would make me choose this technique (or position, or medication) over other techniques?

Literature Example

Check out this publication from the CDC, Recent Declines in Induction of Labor by Gestational Age. Look at the graph on the third page, Figure 3. Notice how some of the bars of the histogram indicate the change was not statistically significant? Notice also that this statistical significance is not due to the size of the change in induction rates.  Instead, it is due to the sample sizes.  Even though the largest change in births at 35 weeks was among women 40 or older, there are too few births to determine if this difference is real or just an artifact of the normal variation.

The trick is finding the right sample size — a sample size that allows you to recognize true differences, but doesn’t create statistical significance where no clinical significance lies.

The Birth Worker Survey

As I explained earlier this week, the birth worker survey did not have enough of a sample size to find much statistically significant.  This was compounded with regression techniques because in regression you throw out any observation that doesn’t have all the necessary variables. So if I wanted to look at the relationship between place of birth and doula work while controlling for something childbirth educators answered, I could only include those who answered yes to both childbirth educator and doula.

Next time we will talk about more sampling issues.