Meta: meta- combining form. From Greek meta ‘with, across, or after.’ Pertaining to a level above or beyond.
Analysis: analysis |əˈnaləsis| noun. From Greek analuō ‘I unravel, investigate’. Detailed examination of the elements or structure of something,
Often times, researchers and students find themselves going through a dense amount of papers on a certain topic only to find results that don’t really seem to point towards a coherent or homogenous conclusion. Does this treatment work?
Well, research A shows yes, but research B and C are quite similar to research A and show the exact opposite results. Which one should I believe in after all? Is the treatment under focus effective or not? This is where meta-analyses can come as a key element. Huque (1988) defined a meta-analysis as “a statistical analysis that combines or integrates the results of several independent clinical trials considered by the analyst to be ‘combinable’.” Nonetheless, there has still been large debate surrounding its proper classification and even some overlap with the definition of “systematic reviews”; however, a systematic review is considered a broader term that characterizes any type of review, and within which “meta-analyses” are included (Egger & Davey Smith, 1997).
Okay, great, but why should I use meta-analysis?
There are many reasons why a meta-analysis can be a very useful tool in research, particularly when aiming to obtain a more valid methodological approach to a particular hypothesis under which a lot of research has been conducted and you wish to obtain a clear and unidirecional conclusion. These are some reasons that make a meta-analysis such a strong methodological approach (Eggy & Davey Smith, 1997):
- Single studies are often not reliable enough to detect significant differences between two treatments. Type I and type II errors are the risks that every researcher tries to avoid at all costs, but these are nonetheless very common, particularly in clinical trials (Freiman, Chalmers, Smith, & Kuebler, 1992);
- Generalizing the results from a meta-analysis makes more sense than from single studies, considering it integrates different sets of populations into the analysis and, thus, accounts for different variations between different groups which will most likely respond distinctively;
- Due to its statistically reliable approach, it makes the review process of the subject at hand much more transparent and less subject to relativity and subjective appraisal from reviewers who conduct narrative reviews.
You’ve convinced me, what do I need to do to carry my own meta-analysis?
Much like conducting any other type of research, at a very basic level, conducting a meta-analysis runs through three simple steps: (1) formulation of your research question; (2) collection and analysis of the data; (3) reporting the outcomes (Egger, Smith, & Phillips, 1997).
As part of a series around the procedures in conducting meta-analyses, Egger, Smith, & Philips (1997) propose a few essential steps you should be sure to take in order to reach a successful outcome in your future meta-analysis.
Step 1: Define eligibility criteria for the data to be included. The criteria will define how compatible the articles to be selected should be between them to make sure we are assessing a common and reliable outcome. This will be based on: (1) quality of trials, (2) type of trials, (3) patients, (4) outcomes, and (5) lengths of follow-up. However, making a sound assessment of the first factor, the quality of trials, will heavily depend on the researcher’s judgement, and thus a sensitivity analysis should be performed a posteriori.
Step 2: Delineate a strategy for identifying the relevant studies. It is important, in combination with the eligibility criteria, to outline what will be the best strategy, according to your research question and the relevance of your study, to select the studies to be included in your analysis. Particularly, you should consider the inclusion of unpublished studies due to bias effects that are often associated with published studies as will be described further ahead.
Step 3: Create a standardised form for data collection. When extracting the data, the most reliable method would be to use two independent observers. For this step, there are scales that have been designed to assess the quality of, for example, clinical trials (for a review, see Moher, Jadad, Nichol, Penman, Tugwell, & Walsh, 1995). Further, the individuals who will be responsible for rating should be blinded to all factors that could influence their assessment, namely: (1) authors and their institutions, (2) names of journals, (3) sources of funding, and (4) acknowledgements.
Step 4: Standardise individual results for comparison between studies. After you’ve collected the data, in order to compare the results you will need to standardize them to something homogenous. For continuous outcomes, one should extract the mean difference between treatment and control group and present it in units of standard deviation. For binary outcomes, odds ratios or relative risks should be considered.
Step 5: Calculate the overall effect by combining the data. In line with the previous step, it seems obvious from a methodological and statistical point of view that simple arithmetic averages are not a reliable way of comparing outcomes; for example, different sample sizes have different statistical power and, therefore, should be given different weights. Hence, a weighted average of the results should be used, in which the larger trials have more influence than the smaller ones. The statistical techniques to tackle this can be classified into two broad models: “fixed effects” model and “random effects” model. The choice between or the other will depend on the way the variability of the results between the studies we have chosen is treated. The first model tells us that random variation is the sole cause for this variability, meaning that the size of the studies is irrelevant to the type of results they give. The second, “random effects” model considers, and more reasonably, as sources of variation different underlying effects in each study, leading to larger confidence intervals than the fixed effects model.
That all sounds lovely, but what are the risks that a meta-analysis might carry?
Indeed, conducting a meta-analysis sounds like a very attractive approach to combining information into one concrete output that (hopefully) brings us clarity into a topic through a thorough and systematic review of previous research. However, as in any other statistical/methodological approach to research, biases are still there to bug us in the reliabiliy of meta-analyses. Egger & Davey Smith (1998) present us with several biases that are most common in meta-analyses.
Publication bias. A study by Easterbrook et al. (1991) showed that the odds of publishing results were three times greater if the results were significant. Consequently, many authors have often avoided submitting their results for publishing simply due to the fact that they were not significant, making selective submission rather than selective acceptance a frequent issue.
Database bias. Despite being a very good source of scholarly information, the most frequently accessed databases publish predominantly reports from developed counries and tend to neglect existing literature from less developed countries.
Citation bias. In line with what was mentioned above in terms of having a good inclusion criteria of papers for analysis, often times the reference lists of manuscripts will often include studies that are strongly supportive of the published results.
Multiple publication bias. It is quite common for researchers to publish multiple manuscripts based on one single research conducted, and it is often very difficult (if not impossible) to distinguish if two papers are original researches or duplicates from the same trials.
Bias in provision of data. Naturally, conducting a meta-analysis will strongly depend on the willingness of researchers to provide us with their data set for analysis, and this could prove to be a strong bias if researchers are not willing, possibly due to the direction of the results of their research.
Biased inclusion criteria. As abovementioned, an author who is familiar with the topic he wishes to assess might be influenced towards selecting papers that might corroborate what they want to look for and ignore other sets of information that could be relevant but contradicting.
Overall, meta-analysis seems like an excellent systematic review tool, but is there any way around all this biasing, then?
Fighting bias in is one of the most challenging tasks in any research design. However, there are ways of trying to avoid it as much as possible to make your research as reliable as possible. Being aware of all of these confounding factors significantly reduces the lack of statistical reliability of your meta-analysis and will help you in structuring a more efficient strategy. Regarding the selection of the studies, it is important to include all studies that match the basic inclusion criteria, independently of any unexpeted predictions, followed by sensitivity analyses considering the different entry criteria (Egger & Smith, 1998). Another method of tackling the presence or absence of bias is through funnel plots (i.e., simple scatterplots of the trials’ effect estimates against their sample size). (Egger, Davey Smith, Schneider, & Minder, 1997).
References
Easterbrook, P. J., Berlin, J. A., Gopalan, R., & Matthews, D.R. (1991) Publication bias in clinical research. Lancet, 337(8746), 867-872.
Egger, M. & Davey Smith, G. (1997). Meta-analysis: Potentials and promise. British Medical Journal, 315(7119), 1371-1374.
Egger, M. & Davey Smith, G. (1998). Bias in location and selection of studies. British Medical Journal, 316(7124), 61-66.
Egger, M., Davey Smith, G., & Phillips, A. N. (1997). Meta-analysis: Principles and Procedures. British Medical Journal, 315(7121), 1533-1537.
Egger, M., Davey Smith, G., Schneider, M., & Minder, C. E. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315(7129), 629-34
Freiman J. A., Chalmers T. C., Smith H., Kuebler R. R. (1992). The importance of beta, the type II error, and sample size in the design and interpretation of the randomized controlled trial. In Bailar, J. C., Mosteller, F. (Eds.), Medical uses of statistics. (357-373) Boston, MA: NEJM Books.
Huque, M. F. (1998). Experiences with meta-analysis in NDA submissions. Proceedings of the Biopharmaceutical Section of the American Statistical Association. 2, 28-33.
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995). Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Controlled Clinical Trials, 16(1), 62-73.
