Weekly Writing 3

I was excited to see a meta-analysis as the assigned reading this week. I was part of a research team that worked on a series of meta-analyses at Purdue University, and I gained a lot of respect for what the process has to offer as far as helping us get closer to the “true” effect in a population. The public is often confused by contradictory studies- how can one researcher find an effect, but then another researcher finds no effect? Is coffee good for us or bad for us? Aack! Well, when you draw a marble out of a bag of mixed colors, even if there are 20 red and 2 blue, you may get a blue one on your first try. So relying on any single “result” can be misleading. A well-done meta-analysis can help us be much more confident by looking at many samples from the marble bag to get a better idea of what is truly in there.

I don’t have a problem with the number of studies overall in the analysis. I would be thrilled to have at least 45. In the meta-analysis I worked on, we only had 33, but still found some meaningful effects to talk about. I am given pause with the idea of making assertions about a sub-group of the studies, though. We’re talking about trying to look at effects across only five to derive generalizations about K-12 education. If the five studies were extremely similar in design and setting, I would be less hesitant. But given the diverse description of the five, including one that was in Taiwan instead of the U.S., that’s just not enough for me as a researcher to be comfortable making assumptions across.

A great resource if you are interested in learning how to do a meta-analysis is Practical Meta-Analysis by Lipsey & Wilson (2001). In their introduction, they mention the wide range of number of studies that can constitute a meta-analysis: “A meta-analysis conducted by one of the authors of this volume, for instance, resulted in a database of more than 150 items of information for each of nearly 500 studies (Lipsey, 1992). We hasten to add, however, that meta-analysis does not require large numbers of studies and, in some circumstances, can be usefully applied to as few as two or three study findings” (p. 7).

So, I must disagree with folks who feel that 45 studies is not enough, because overall in this case I think 45 is reasonable given the inclusion criteria. But when we get down to a comparison of as few as three or five, I would want to see that those studies are all from the same population and had a comparable number of participants. For example, you may have three different studies that each took a sample of roughly 100 students from the same incoming freshman class at a university. Due to error and other reasons, each study finds a slightly different effect. I think that averaging across the three, even though it’s “only” three studies, would be useful in that case.

References

Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-analysis. Thousand Oaks, CA: Sage.

U.S. Department of Education, Office of Planning, Evaluation, and Policy Development (2010). Evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf

1 thought on “Weekly Writing 3

  1. Owen

    Nice thoughts on this meta-analysis, specifically and on the process of meta-analysis specifically.

    One thing that I always think of in this regard is the amount of variance in online courses. There is a great deal of cohort variance, based on student maturity and preparedness, as well as subject variance, some subjects are much more challenging online than others, and then there are course design and instructor variance issues that can be tremendously diverse. On top of this we have institutional variability… It is hard to ever find a true apples-to-apples situation.

    That being said, I agree that comparing 45 studies does begin to approach a number here some significant results can be reached. Particularly depending on the strength of the results of the involved studies.

    Even comparing 45 courses here at UAF – a very significant challenge.

    -Owen

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *