Why Would Researchers Skip Peer-Review? Media Reports of Unpublished Findings

‘You love your iPhone. Literally.’ ‘This is your brain on politics.’ ‘Overclock your brain using transcranial Direct Current Stimulation (tDCS).’ There are many other claims in psychology which have been publicised by the media, yet remain unchecked by academic experts. Peer-reviewed publications – papers which have been checked by researchers of similar expertise to the authors – are produced very slowly and only occasionally make instant impacts outside the walls of academia. In contrast, media publications are produced very quickly and provoke immediate reactions from the general public.

Now and again, researchers rush their findings to the media before peer-reviewed journals. This seems to reflect the temptation to prioritise the impact of research over its validity. Although doing so risks one’s academic reputation, is it ever justified to skip peer-review? Is it ever necessary to compromise the validity of research to achieve its applied potential, given practical constraints and entrepreneurial motives? To address this controversial question, let’s turn to some examples of unpublished research.

‘You Love Your iPhone. Literally.’

On 30th September 2011, Martin Lindstrom, a marketing guru, reported his neuroimaging study to readers of the New York Times. Together with the commercial neuroimaging company, Mind Sign Neuromarketing, Lindstrom used functional Magnetic Resonance Imaging (fMRI) to record the brain activity of 8 men and 8 women aged 18 to 25. The participants were asked to watch video clips and listen to sound bites of ringing iPhones, which were found to induce neural activity in the insula. If you were asked to predict the emotion experienced by people with strong activation of the insula, what would you say?

The famous emotional correlate of insula activity is disgust. For example, the anterior insula is activated when people are shown facial expressions of disgust (Phillips et al., 1997) or photos of poor hygiene, pollution, disease and injuries (Wright, Shapira, Goodman, & Liu, 2004). So did Lindstrom infer that ringing iPhones provoke the experience of disgust? Not at all. In fact, to the millions of monthly visitors to the NYTimes.com, Lindstrom concluded the opposite: that his 16 participants experienced the same type of love in response to their ringing iPhones as when in the company of their partners, best friends or parents. Indeed, activity in the medial insula not only correlates positively with disgust but also with love (e.g., Bartels and Zeki, 2000). In Lindstrom’s own words, ‘The man or woman we love most may be seated across from us in a romantic Paris bistro, but his or her 8GB, 16GB or 32GB rival lies in wait inside our pockets and purses.’

Lindstrom’s article received a wave of criticism online: ‘Neuromarketing means never having to say you’re peer-reviewed’ said the Neurocritic on his blog, ‘fMRI shows my bullshit detector going ape shit over iPhone lust’ said David Dobbs (a science writer) at Wired.com and ‘NYT Editorial + fMRI = complete crap’ said Russell Poldrack (Professor of Psychology and Neurobiology at the University of Texas at Austin) on his blog. Of course, if Lindstrom had instead concluded that people experience disgust when their iPhones ring, he would have still been criticised. The problem is not exactly what he inferred but the fact that he dared to make this type of inference in the first place.

In typical neuroimaging studies, researchers manipulate the cognitive activity performed by participants in the experimental and control conditions. The results support the forward inference that any differences in brain activity between the two conditions contributed to performing the differences in cognition between conditions. In contrast, Lindstrom made the reverse inference that activation of the insula demonstrated that his participants were experiencing love. Unfortunately, this does not make logical sense: the EFPSA logo is all orange but not all orange things are EFPSA logos – as disappointing as that may be. Similarly, love may induce activity in the insula but love is not the source of all insula activity. In fact, the anterior insula is activated in one third of neuroimaging studies, given its importance in many cognitive functions (Yarkoni, Poldrack, Nichols, Van Essen, & Wager, 2011).

Before writing this article for the New York Times, Lindstrom published a book called ‘Buyology: How everything we believe about why we buy is wrong’. The book reports many evidence-based findings and claims (including the title!), which have not been published in peer-reviewed journals. Similarly, the company www.foc.us has begun selling devices which supposedly enhance your attentional focus through non-invasive stimulation of prefrontal cortex. The devices ‘make your synapses fire faster’ – a claim which Dr Christian Jarrett (a science writer) at Wired.com highlights is not really possible: only neurons, not chemical synapses, can fire faster. Indeed, there is currently no peer-reviewed evidence that the specific devices sold by www.foc.us work effectively and without exerting side-effects on other cognitive functions – but maybe that will change.

Importantly, the practice of skipping publication via peer-review is not restricted to commercial researchers, such as Lindstrom and the designers at www.foc.us. On 11th November 2007, Professor Marco Iacoboni and colleagues made reverse inferences from the brain responses of 20 undecided voters to photos of candidates and campaign videos relating to the US presidential election of 2008. Again, it was the New York Times which broadcasted the unpublished findings in an article called ‘This is Your Brain on Politics’. Iacoboni is a highly respected neuroscientist at the University of California, Los Angeles and is famous for his work on mirror neurons, such as their function in identifying the intentions of others (Iacoboni et al., 2005). Nevertheless, he and his colleagues decided to skip peer-review this time.

What tempts researchers to skip peer-review?

It would be easy to condemn Lindstrom, Iacoboni, www.foc.us and others who rush their findings to the media before publishing the methodological and statistical details of their research in peer-reviewed journals. Academics readily and rightly criticise sensationalised news reports of psychology. However, it is also important to consider the underrepresented reasons as to why researchers occasionally take the controversial step of communicating research without peer review.

Researchers first face the deterrent of the bias that inflicts peer review. Peters and Ceci (1982) selected 12 psychology studies which had been published in each of 12 greatly respected psychology journals. In the papers, Peters and Ceci changed minor irrelevant details and more importantly, replaced the names of the authors and their highly regarded American institutions with fictitious alternatives. After the papers were resubmitted to the same journals in which they had already been published, just 8% of the editors and reviewers recognised the duplication. Of the nine papers that continued through the review process, eight were rejected and 89% of the referees advised editors against publishing the papers!

The referees and editors attributed their rejection decisions to methodological flaws in the papers. These slight flaws presumably attracted much more attention during the resubmission process, due to the lacking prestige of the institutions from which the papers had been submitted this time round. Therefore, when researchers belong to institutions which lack academic prestige, such bias may deter submissions to non-blind peer-reviewed journals. This is in addition to the bias against publishing null results, which when combined with the ‘publish or perish’ culture of academia, undermines the objectivity of science (Fanelli, 2010a). Fanelli (2010b) found that the odds of a journal article reporting support for the tested hypothesis are higher in psychology than any other physical or social science; for example, the odds are five times in psychology than in space science.

Researchers may also be deterred by the snail slow pace of peer review. The time between submission and acceptance of papers is unpredictable, with far greater variation between articles within a journal than between journals or between subjects (Björk & Solomon, 2013). Entrepreneurs need their evidence-based ideas to reach applied reality quickly, before competitors do the same and the innovation characterising their product fades. Therefore, peer-reviewed support for novel ideas may necessarily follow rather than precede their implementation in the world of business.

Can research change the world, yet also retain validity?

Although the prestige bias and speed of peer review may have deterred Lindstrom, Iacoboni and www.foc.us from submitting their research to peer-reviewed journals before the media, there may also be a more obvious reason for skipping peer review: the authors may have known that their research lacked sufficient validity to pass the stringent checks of peer review. If so, peer review is achieving its goal of quality control. However, does quality control have the side-effect of minimising the potential for innovative applications of research?

By writing their media articles, Lindstrom and Iacoboni intended to capture public curiosity in what the latest neuroimaging technology can tell us about behaviour in everyday and important events: ringing iPhones and national elections. Indeed, people are often inspired by psychology because of its relevance to real-life. Unfortunately, however, when research is oriented towards the goal of identifying real world implications more than academic publication, it often loses validity and so rightly generates academic criticism. Until psychology rewards the practical (in addition to the theoretical) value of research, its applications may continue to lack validity and academia may continue to struggle to make real-life impacts. Until citations are issued to journal papers which engage the opinions of politicians, practitioners, businesses and the public (in addition to fellow academics), Lindstrom may continue to claim that you love your iPhone like it’s a family member and www.foc.us may continue to sell brain stimulation devices which ‘make your synapses fire faster’.

Although there are exceptions, the research community appears divided into two camps: supporters of validity over impact (the average academic) and supporters of impact over validity (the average politician, practitioner, business and member of the public). Whilst each camp could continue to criticise and dismiss each other, it is time to strike a better balance: to design research which has applied value, yet retains as much validity as can be practically achieved in a given time and financial scale. The solution is neither to skip peer-review nor to give up on the sensationalist tendencies of the media. The solution is to reward research which strives to entertain and benefit the theorists and the masses in tandem. So whilst certainly, applied research must hold up to greater academic scrutiny, so must academic research hold up to greater applied scrutiny – and lacking consideration of the latter may be the greatest flaw of peer review as it stands.

Is this blog post valid? Will it have impact? Was it possible to achieve both? Only time will tell…

References

  • Bartels, A., & Zeki, S. (2000). The neural basis of romantic love. Neuroreport, 11(17), 3829-3834.
  • Björk, B. C., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics7(4), 914-923.
  • Dobbs, D. (2011, October 2). fMRI shows my bullshit detector going ape shit over iPhone lust. Wired. Retrieved from http://www.wired.com
  • Fanelli, D. (2010a). Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PloS One5(4), e10271.
  • Fanelli, D. (2010b). “Positive” results increase down the hierarchy of the sciences. PloS one5(4), e10068.
  • Iacoboni, M., Freedman, J., Kaplan, J., Jamieson, K. H., Freedman, T. Knapp, B., & Fitzgerald, K. (2007, November 11). This is your brain on politics. The New York Times. Retrieved from http://www.nytimes.com.
  • Iacoboni, M., Molnar-Szakacs, I., Gallese, V., Buccino, G., Mazziotta, J. C., & Rizzolatti, G. (2005). Grasping the intentions of others with one’s own mirror neuron system. PLoS Biology3(3), e79.
  • Lindstrom, M. (2008). Buyology: How everything we believe about why we buy is wrong. London: Random House.
  • Lindstrom, M. (2011, September 30). You love your iPhone. Literally. The New York Times. Retrieved from http://www.nytimes.com
  • Neurocritic (2011, October 1). Neuromarketing means never having to say you’re peer-reviewed. Neurocritic. Retrieved from http://www.neurocritic.blogspot.com
  • Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences5(02), 187-195.
  • Phillips, M. L., Young, A. W., Senior, C., Brammer, M., Andrew, C., Calder, A. J., … & David, A. S. (1997). A specific neural substrate for perceiving facial expressions of disgust. Nature, 389(6650), 495-498.
  • Poldrack, R. (2011, October 1). NYT Editorial + fMRI = complete crap. Russpoldrack.org. Retrieved from http//:www.russpoldrack.org.
  • Wright, P., He, G., Shapira, N. A., Goodman, W. K., & Liu, Y. (2004). Disgust and the insula: fMRI responses to pictures of mutilation and contamination. Neuroreport, 15(15), 2347-2351.
  • Yarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., & Wager, T. D. (2011). Large-scale automated synthesis of human functional neuroimaging data. Nature Methods, 8(8), 665-670.
Robert Blakey

Robert Blakey

Robert Blakey is a third year undergraduate student of Experimental Psychology at the University of Oxford and was a member of the 2012-2013 cohort of EFPSA's Junior Researcher Programme. He is currently carrying out a research project on the effect of interaction on estimation accuracy and writing a dissertation on consumer neuroscience. He is also interested in social cognition and specifically, public perceptions of influences on behaviour.

More Posts

Facebooktwitterrss