What happens to studies that accept the null hypothesis?

Source: Scargle, 2000

“The literature of social sciences contains horror stories of journal editors and others who consider a study worthwhile only if it reaches a statistically significant, positive conclusion; that is, an equally significant rejection of a hypothesis is not considered worthwhile” (Scargle, 2000).

This is a footnote in Jeffrey D. Scargle’s, an astrophysicist working for NASA, article about the publication bias in scientific journals. Usually, the psychologist in me would go all defensive of our precious little social science, but then one discovers this: a couple of researchers trying to publish a paper debunking Bem’s research on ESP (in layman terms, ESP means predicting the future). More precisely, their woes while trying to publish a paper with nonsignificant results. How many papers have you read that have nonsignificant results, that accept the null hypothesis? I have a feeling you have the same answer as me, and it’s frighteningly converging on zero. What happens to those papers? And what’s the implication of such a bias in publishing for science at large?

Before delving deeper into the subject, we should explicitly state what the publication’s bias (or its extreme formulation, the file-drawer problem) is, which was best explained in an influential work by Rosenthal (1979, p. 638; as cited in Scargle, 2000):

…researchers and statisticians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out… The extreme view of this problem, the “file-drawer problem”, is that the journals are filled with the 5% of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95% of the studies that show nonsignificant (e.g., p > .05) results.

This is basically a problem with the untruthfulness of the publication process in science, especially in social sciences where almost all quantitative analysis is based on statistical (non)significance. In an ideal system, a statistically significant result means that the results are significantly different from the expected null hypothesis. But what if we’re fishing for results, which is precisely the case if a bias for publishing only significant results exists? To put it even more metaphorically: if we fish long enough, we will certainly score big game; whatever our research subject might be. Be it personality traits or precognition, as in Bem’s case.

Ritchie’s experience in trying to publish a study offering a proof that no precognition exists when replicating Bem’s research was, as stated in Tom Bartlett’s (2011) article, was rejected out of different reasons. No reviewer will hopefully openly state that a study is rejected specifically because it does not show a significant effect (contrary to Scargle’s horror story from the introduction). Still, the question is–what was the real reason of rejection? Even in such a controversial study as Bem’s, where the scientific community would make a statement by publishing nonsignificant results from a replication study, this study was rejected. By publishing it, the scientific community would back up their disdain of the pseudoscientific ESP research with sound empirically grounded data, and yet they still refuse to publish it?

All this makes it sound like there’s an intricate conspiracy within the scientific community, plotting in the shadows and disallowing the publication of marginal or no effect studies. I think that the absurdity in that is self evident.

My take on it is that this is yet another incarnation of human cognition in an institution, taken into a sophisticated extreme in the most elaborate human epistemological endeavor–science. This artifact of human cognition has been called by psychologists the confirmation bias for decades. What is it precisely?

confirmation bias connotes a less explicit, less consciously one-sided case building process. It refers usually to unwitting selectivity in the acquisition and use of evidence. The line between deliberate selectivity in the use of evidence and unwitting molding of facts to fit hypotheses or beliefs is a difficult one to draw in practice, but the distinction is meaningful conceptually, and confirmation has more to do with latter than the former. The assumption that people can and do engage in case-building unwittingly, without intending to treat evidence in a biased way or even being aware of doing so, is fundamental to the concept. (Nickerson, 1998)

Science, as an organized endeavor of research, the ultimate epistemology of the modern world, just transformed this normal and typical artifact of human cognition into something compatible with the critical methodology it’s based on. The publication bias. Looking for and publishing only evidence that support your theories and hypotheses is ethically questionable at best, or an example of scientific dishonesty and data fishing at worst. But what if the publication process, the mechanism that guards the gates of cited scientific literature has the bias built into it? Then the scientists would remain true to their trade, and still fulfill their basic cognitive need. To publish results that support their theories and leave everything else in the ‘shadow’.

You do the research, you’re an honest researcher, but you only publish the research that supports your ideas. Not because you are dishonest, but because reviewers and journal editors are only interested in cutting edge, top notch and most importantly, significant studies. When you take such a system and couple it with statistical significance as the ultimate demarcation criterion of publishable and unpublishable – you get a science which has the confirmation bias systemically built into it.

On a practical level, you get a system of journals that will not publish Ritchie’s study, even if it proves something the true, proper scientists at large want proven – that ESP is a pseudoscientific research problem that should not be even considered for publication in ‘serious’ journals.

References

Barlett, T. (2011, Dec 6). Wait, maybe you can’t feel the future. The Chronicle of Higher Education. Retrieved from: http://chronicle.com/blogs/percolator/wait-maybe-you-cant-feel-the-future/27984

Flis, I. (2011, Feb 15). The ‘science’ in scientific peer-reviewed journals. JEPS Bulletin. Retrieved from: http://blog.efpsa.org/2011/02/15/the-science-in-scientific-peer-reviewed-journals/

Nickerson, R.S. (1998). Confirmation Bias: A Ubiquitous Phenomenon in Many Guises. Review of General Psychology, 2(2), 175 – 220.
Scargle, J.D. (2000). Publication Bias: The “File-Drawer” Problem in Scientific Inference. Journal of Scientific Exploration, 14(1), 91-106.

The histogram in the introduction is taken from Jeffrey D. Scargle’s cited article, representing the unpublished/published studies ratio.

About the author

Ivan Flis Ivan Flis is a PhD student in History and Philosophy of Science at the Descartes Centre, Utrecht University; and has a degree in psychology from the University of Zagreb, Croatia. His research focuses on quantitative methodology in psychology, its history and application, and its relation to theory construction in psychological research. He had been an editor of JEPS for three years in the previous mandates.

Facebooktwitterrss