I was excited to see a meta-analysis as the assigned reading this week. I was part of a research team that worked on a series of meta-analyses at Purdue University, and I gained a lot of respect for what the process has to offer as far as helping us get closer to the “true” effect in a population. The public is often confused by contradictory studies- how can one researcher find an effect, but then another researcher finds no effect? Is coffee good for us or bad for us? Aack! Well, when you draw a marble out of a bag of mixed colors, even if there are 20 red and 2 blue, you may get a blue one on your first try. So relying on any single “result” can be misleading. A well-done meta-analysis can help us be much more confident by looking at many samples from the marble bag to get a better idea of what is truly in there.
I don’t have a problem with the number of studies overall in the analysis. I would be thrilled to have at least 45. In the meta-analysis I worked on, we only had 33, but still found some meaningful effects to talk about. I am given pause with the idea of making assertions about a sub-group of the studies, though. We’re talking about trying to look at effects across only five to derive generalizations about K-12 education. If the five studies were extremely similar in design and setting, I would be less hesitant. But given the diverse description of the five, including one that was in Taiwan instead of the U.S., that’s just not enough for me as a researcher to be comfortable making assumptions across.
A great resource if you are interested in learning how to do a meta-analysis is Practical Meta-Analysis by Lipsey & Wilson (2001). In their introduction, they mention the wide range of number of studies that can constitute a meta-analysis: “A meta-analysis conducted by one of the authors of this volume, for instance, resulted in a database of more than 150 items of information for each of nearly 500 studies (Lipsey, 1992). We hasten to add, however, that meta-analysis does not require large numbers of studies and, in some circumstances, can be usefully applied to as few as two or three study findings” (p. 7).
So, I must disagree with folks who feel that 45 studies is not enough, because overall in this case I think 45 is reasonable given the inclusion criteria. But when we get down to a comparison of as few as three or five, I would want to see that those studies are all from the same population and had a comparable number of participants. For example, you may have three different studies that each took a sample of roughly 100 students from the same incoming freshman class at a university. Due to error and other reasons, each study finds a slightly different effect. I think that averaging across the three, even though it’s “only” three studies, would be useful in that case.
Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-analysis. Thousand Oaks, CA: Sage.
U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (2010). Evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies. Retrieved from https://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf