Some of you might have asked themselves this question a couple of times when checking out the literature of a specific field. Imagine the following situation: You have completed your research and now you want to compare your results to research done previously. You finally found the suitable article, have the necessary effect sizes/power and can start comparing. But wait: Who actually tells you that the reported results are correct? Would you happen to notice if the results had been influenced by factors which are sometimes not even visible to the authors themselves? The probability of you detecting them, are tiny, especially as you only have certain information of how the study has been done and what elements have been removed. It is hard to digest, but most research findings, even those reported in high quality journals, are incorrect. Try to imagine the impact this situation has on your education and your research such as research in general.
In the following post I want to discuss multiple factors of why and how research results can be false and want to outline some aspects of how the situation might be improved. The main aspects are thereby retrieved from Ioannidis essay (2005).Ioannidis (2005) claims that the probability of research findings being incorrect is greater than most people might think. He offers multiple aspects that support his point and show the deficits often included in research. When doing research you want to find results. In particular the positive ones. Although we know that negative and even null results are also worth mentioning, we tend to pay less attention to them and keep looking for better or more suitable ones. Most likely, you choose the p-value to represent your study and thereby forget that the study cannot simply be summarized by one single p-value (Ioannidis, 2005).
Impacts on Publication of Research Results
In the following paragraphs I want to outline a few factors that not only have an impact on research results, but also on the publication of these results.
Scientific bias
A well- known phenomenon that influences the significance and thus credibility of research results is the scientific bias. As it increases, the truth of research results decreases. Bias can be evoked by conflicts of interest, inefficient use of data, failure to notice statistically significant relationships such as by large measurements where error relationships might get lost in noise (Ioannidis, 2005). Especially the conflicts of interest can be a serious problem for research in general, as researchers or prestigious investigators often try to prevent contraire research results from getting published. The thought behind it is to publish only studies that support one certain aspect of a theory, because the authors want to support their outcomes or because they feel highly committed to their own findings.
Financial Interests
The financial interests of the results also play a certain role in the mentioned scenario. Research can be quite expensive. So if a specific research is done on behalf of, for example the government or a company, they want the new numbers to support their statement and future plans. Especially if these new results influence the success or loss and the image of the customer. This may even lead to the suppression of peer-review.
Relationships
In your research you are analysing many different relationships. As they play an important role I now want to put a special focus on these relationships: If you choose to test a great number of relationships that are not selected properly, the research findings are less likely to be true than if you take a smaller number to test. Therefore it is very important to make up your mind before you start your research: What do you want to analyse? What is the intention of the study and which relationships do you want to focus on? This selection should take some time as it is framing your whole project.
New hypothesis-generating experiments decrease the post-study probability that a finding is true (PPV, positive predictive value) which depends a lot on the pre-study odds (R, relation of the number of true relationships to no relationships). Ioannidis (2005) puts it all in one formula: , saying that “a research finding is thus more likely true than false if “ (p. 0696-0697). The term represents the pre-study probability of the findings being true, which is influenced by Type I (α) and II (β) error rate. The base for this calculation has its origins in Wacholder, Chanock, Garcia-Closas, Elghormli, and Rothman (2004, as cited in Ioannidis, 2005) and was called the false positive report probability. The impact of the formula for pre-study analyses can be summarized by saying that “the PPV is substantially higher when more research findings are statistically significant” (Moonesinghe Khoury, and Janssen, 2007, p. 0218).
Scientific Field
One important factor that can have an effect on research results lies in the scientific field itself. An example can be given by Social Psychology, a field where results and theories always depend on something. They mostly depend on certain factors which differ between cultures, genders, day times and even temperature. If the research group is flexible enough with the designs, definitions, outcomes, and analytic modes of their study, negative results can easily be transformed into positive ones (Ioannidis, 2005).
How can you detect the false research findings?
But does Ioannidis (2005) really have a point? Or might there be different aspects and points of view as why most research articles are being incorrect? Or might his claim even be wrong? Here comes an example of your daily life illustrating the problem: I guess you all remember the time when you were little and your parents told you to always eat up your spinach as it contains lots of iron and is therefore good for your health and process of growing and developing? From generation to generation parents believed in the power of spinach until researchers discovered that there had been mistakes in their measurement – the comma was set on the wrong spot. How was it possible for researchers not to find out about this for such a long time?
In line with the previous anecdote, the weakness of Ioannidis (2005) essay has to be mentioned. As Moonesinghe et al. (2007) state, the more often research results get replicated, the better is the PPV of the results being true. This thought should encourage researchers, young and old, to always engage in science and see if the results can be replicated. Even by repeating the same bias, the value of the replication only gets decreased, not erased (Ioannidis, 2005).
You therefore have to decide for yourself: How high is the possibility that the research findings are false? Do you feel comfortable using Ioannidis’ formula (2005) and base your decision on the PPV? The attempt in research is not to only work with hypotheses that are 100 per cent “true”. “Our acceptability of “truth” depends on how much we care about being wrong” (Djulbegoric & Hozo, 2007, p. 0215). At university you learn that it is better to work with a wrong theory than not having any theory at all. Maybe the same attitude has to be applied to research results: prefer wrong results over no results at all. Science is a field that can be changed by even the simplest facts and smallest discoveries. All research results have to be interpreted with caution and new discoveries need to be monitored to decide if older results have to be seen in a new light.
So, all in all, the new and old conclusion is to always be careful when relying on results and to monitor any changes in the field of research. As a support you can ask yourself the following question: What are the benefits and harms of these results? As long as the beneficial results of the research hypothesis outweigh its harms, the results should be accepted (Djulbegoric & Hozo, 2007).
How can you avoid doing the same mistakes?
So how can you improve the situation or simply make sure that the research results are not being incorrect? A few tiny aspects might help the researcher to find out: First of all, if many teams are working on the same hypothesis, you cannot simply take one of their results and take it for granted. It is important to look out for the totality of the evidence, compare the results and discuss the heterogeneity of research findings. As another point, you should focus more on the importance of the pre-study odds instead of chasing significance. The Type I and II error rates, the power and the R influence the PPV, as mentioned before. Every one of these parts should get your special attention, so you can discuss the pre-study odds and make a well-founded decision. This way, even non-significant results have a good base to build the discussion upon. When trying to come close to the “perfect” study and results, you should focus on large studies or low-bias meta-analyses which have better powered evidence.
So all in all you can never be sure if your own or others’ research results are being correct, since they are influenced by multiple factors. You have to start thinking and decide: How can you contribute good research and methodology to science and how and why do you rely on certain findings and may or may not accept them?
References
Djulbegovic, B., & Hozo, I. (2007). When should potentially false research findings be considered acceptable? PLoS Med 4(2): e26. doi:10.1371/journal.pmed.0040026
Ioannidis, J. P. A. (2005). Why most research articles are false? PLoS Med 2(8): e124 doi:10.1371/journal.pmed.0020124
Moonesinghe, R., Khoury, M. J., & Janssens, A. C. J. W. (2007). Most published research findings are false–But a little replication goes a long way. PLoS Med 4(2): e28. doi:10.1371/journal.pmed.0040028
