A Psychologist’s Guide to Reading a Neuroimaging Paper

Psychological research is benefiting from advances in neuroimaging techniques. This has been achieved through the validation and falsification of established hypothesis in psychological science (Cacioppo, Berntson, & Nusbaum, 2008). It has also helped nurture links with neuroscience, leading to more comprehensive explanations of established theories. Positron Emission Tomography (PET), functional MRI (fMRI), structural MRI (sMRI), electroencephalography (EEG), diffusion tensor imaging (DTI) and numerous other lesser-known neuroimaging techniques can provide information complimentary to behavioural data (Wager, 2006). With these modalities of research becoming more prevalent, ranging from investigating the neural effects of mindfulness training to neuro-degeneration, it is worth taking a moment to highlight some points to help discern what may be good or poor research. Like any other methodology, neuroimaging is a great tool that can be used poorly. As with all areas of science, one must exercise a good degree of caution when reading neuroimaging papers.

Reading a Neuroimaging Paper

In addition to the more general issues of critically reading a scientific paper, there are some common methodology pit falls that arise especially in neuroimaging papers. While there are physiological limitations to the use of each of these modalities, several others arise due to poor experimental design and analysis. Here I will focus on the latter. For a comprehensive overview of technical and biological limitations in fMRI see Logothetis (2007).

The pre-processing involved and statistical analysis of neuroimaging data can be complex. A lack of understanding of the image processing pipeline and the limitations of the statistical approach used is obviously dangerous. Pressing buttons on a computer isn’t sufficient; a conceptual knowledge of what is being done is really required. Here, a few of the common pitfalls to look out for while reading neuroimaging papers are presented.

Multiple Comparisons

Bennett, Baird, Miller, and George (2009) conducted an fMRI in which a post-mortem salmon was used to determine emotions from images. So what would be the expected result of this study—surely not activity in the brain cavity? You can see for yourself from the image below that indeed, even a dead salmon shows some activation.

Screen Shot 2014-06-29 at 23.28.37

Taken from Bennett et al. (2009), uncorrected (p = 0.001)

This surprising finding is associated with the fact that in any fMRI study, there is going to be noise. Imagine that in the volume of a human brain we have 100,000 voxels (3D pixels)!. In effect, when comparing two conditions we are conducting 100,000 t-tests to determine if there is a change in relative blood flow at each of these voxels. As we know from statistics, there are going to be false positives and by chance, some of these may cluster together. With the significance level of α = .05, there would be 5000 false positives!

There are many simple solutions to this multiple comparison problem. While Bonferroni correction is the first thing that comes to mind, it is generally too conservative for functional data and violates many assumptions. For a Bonferroni correction, the data needs to be independent, however, adjacent voxels are related, especially after the smoothing process during pre-processing. Therefore, various other methods are often used such as Random Field theory, small volume correction, peak, and cluster thresholds (Poldrack, Mumford & Nichols, 2011) .

The standard threshold for corrections may vary in different analysis software, but the more recent programs such as SPM8 (Statistical Parametric Mapping), soon to be SPM12 tend to have more stringent analysis. For any of these, an uncorrected threshold of p = .05 is a red flag in neuroimaging papers (even p =.01 can be a bit suspicious). While less frequent now, this was not an uncommon practice in early imaging papers. There are cases where uncorrected or lower thresholds might be arguably justified. Take, for example a region of interest analysis of solely the amygdala: Due to the reduction in the dimensionality, a less conservative correction needs to be conducted.

The most common form of multiple comparison correction for a whole brain analysis is a family-wise error correction (pFWEcluster < .05) based on cluster extent using a cluster-forming threshold. This cluster-forming threshold tends to be .001 uncorrected, and potentially lower, thus setting an uncorrected threshold for peak activation. The cluster correction then performs a stringent multiple comparisons correction on clusters that reach this peak activation. What is the likelihood of a cluster of adjacent voxels being active by chance alone? Earlier this year, Woo, Krishnan, and Wager (2014) published a paper on the pitfall of reducing this threshold for cluster correction. However, where possible, the current prominent notion in the scientific community is to hold a conservative threshold, which results in confidence of any activations being meaningful. Low thresholds pose the risk of many false positives, thus the results may not be replicable – so the “publish or perish” maxim leads to a far too liberal handling of thresholds. However, a well-powered and controlled experiment should help deter this from happening. On the other hand, having a few subjects may lead to threshold-dropping, so look out for papers with subject numbers less than 15-20 in a group.

Another thing to be suspicious of are unusual threshold limits. Say for example, the study corrected for multiple comparisons at p = .003. While a significance level of p = .05 is an arbitrary value itself, it is not a normal practice for researchers to choose their own level of significance. In relation to this, another questionable reporting method is ‘defining’ significance, for example when instead of conducting a correction, a voxel extent threshold is set for uncorrected data. That is, if more than 10 adjacent voxels are active, it is considered a significant cluster. Defining arbitrary cluster sizes like this is not an appropriate method.

Circularity

The scientific method stipulates that analysis follows a hypothesis. This is especially important for high dimensionality data, like that from neuroimaging. It is easy to accidently fish for results and a problem that arises from this is circularity. The basis of a region of interest (ROI) analysis should not come from the results. This is commonly referred to as double dipping. ROIs need to be selected a priori, independent of the conducted analysis. Let’s say that in conducting a whole brain analysis, you find a cluster of activation around the amygdala. Interesting, you might think, and you explore this further and conduct a ROI analysis based on the signal extracted from this region. Well, of course, the extracted data are going to be strongly correlated! Instead of having a representative sample, only the data that show activation above a selected threshold are being looked at. If the selected activation is representative of the experimental effect, there will be no problem. However, these datasets are inherently noisy due to the nature of the fMRI signal and steps taken during pre-processing, which may distort the results if a selected region is reanalysed without prior evidence from a separate dataset to show plausible recruitment (Kriegeskorte, Simmons, Bellgowan, & Baker, 2009). This can artificially inflate or distort a small or moderate effect size to being large. When correlations greater than r =.8 appear, there is something is fishy. This is considering, at best, personality measures and fMRI have more or less the same reliability (Ioannidis, 2005; Vul, Harris, Winkielman, & Pashler, 2009). This can happen covertly without being reported but it can be spotted in the methods if a priori regions are not specified. sometimes it is written straight out that the ROIs are based on their functional activations. One must be wary of such papers.

  ‘Imager’s Fallacy’

Another common mistake is the issue of Imager’s Fallacy. A difference in significance does NOT imply significant difference. It is difficult to wrap your head around but it is still one of the most common mistakes in analysis still being made (Henson, 2005).

To illustrate, imagine that the striatum is more active during condition 1 compared to baseline. The same region is active but less significantly in condition 2 compared to baseline. This does not indicate whether there is a significant difference between conditions in the striatal region— it only indicates that to varying degrees, there is activation in this region for both conditions compared to baseline.

Further Reading

As a start, I would highly recommend these two guides for writing imaging papers, Poldrack et al. (2008) and Ridgway et al (2008). These will give a good overview of what to expect when reading or writing these types of publication. In addition, for readers lacking an extensive understanding of the physics theory behind magnetic resonance, the image acquisition part of the methods section may be daunting and I recommend this mri-physics introduction. YouTube videos such as these can also be helpful.

And Finally….

The limitations discussed are often legitimate but the argument that they make the modality redundant for measuring behavioural processes is not (Henson, 2005; Logothetis, 2008). MR data acquisition should be viewed as an additional method to complement behavioural measures. It will not solve all theoretical issues in psychology, but it does help provide insights into several cognitive and emotional processes. As with all areas of scientific inquiry, so long as appropriate measures are taken, then reliable data will be produced.

 

References

Bennett, C. M., Baird, A. A., Miller, M. B., & George, L. W. (2009). Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument for Proper Multiple Comparisons Correction. Journal of Serendipitous and Unexpected Results, 1(1), 1–5.

Cacioppo, J. T., Berntson, G. G., & Nusbaum, H. C. (2008). Neuroimaging as a New Tool in the Toolbox of Psychological Science, 17(2), 62–67.

Farah, M. J., & Hook, C. J. (2013). The Seductive Allure of “Seductive Allure.” Perspectives on Psychological Science, 8(1), 88–90. doi:10.1177/1745691612469035

Henson, R. (2005). What can functional neuroimaging tell the experimental psychologist? The Quarterly Journal of Experimental Psychology, 58(2), 193–233. doi:10.1080/02724980443000502

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124

Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009). Circular analysis in systems neuroscience: The dangers of double dipping. Nature Neuroscience, 12(5), 535–540. doi:10.1038/nn.2303

Logothetis, N. K. (2008). What we can do and what we cannot do with fMRI. Nature, 453(7197), 869–78. doi:10.1038/nature06976

McCabe, D. P., & Castel, A. D. (2008). Seeing is believing: The effect of brain images on judgments of scientific reasoning. Cognition, 107(1), 343–52. doi:10.1016/j.cognition.2007.07.017

Poldrack, R. a, Fletcher, P. C., Henson, R. N., Worsley, K. J., Brett, M., & Nichols, T. E. (2008). Guidelines for reporting an fMRI study. NeuroImage, 40(2), 409–14. doi:10.1016/j.neuroimage.2007.11.048

Poldrack, R. Mumford, J. & Nichols, T. (2011) Handbook of Functional MRI Data Analysis. Cambridge University Press. ISBN: 9780521517669

Ridgway, G. R., Henley, S. M. D., Rohrer, J. D., Scahill, R. I., Warren, J. D., & Fox, N. C. (2008). Ten simple rules for reporting voxel-based morphometry studies. NeuroImage, 40(4), 1429–35. doi:10.1016/j.neuroimage.2008.01.003

Schweitzer, N. J., Baker, D. A, & Risko, E. F. (2013). Fooled by the brain: Re-examining the influence of neuroimages. Cognition, 129(3), 501–11. doi:10.1016/j.cognition.2013.08.009

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science, 4(3), 274–290. doi:10.1111/j.1745-6924.2009.01125.x

Wager, T. D (2006). Do We Need to Study the Brain to Understand the Mind. Observer, 19(9). Retrieved from https://www.psychologicalscience.org/index.php/publications/observer/2006/september-06/do-we-need-to-study-the-brain-to-understand-the-mind.html on 15 December 2014

Woo, C.W., Krishnan, A., & Wager, T. D. (2014). Cluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations. NeuroImage, 91, 412–9. doi:10.1016/j.neuroimage.2013.12.058

Interview in Israel: with Prof. Daniel Brom

Prof. Danny Brom is a clinical psychologist, the initiator of the Israel Trauma Coalition, and the Founding Director of the Israel Center for the Treatment of Psychotrauma in Jerusalem. Prof. Brom has published his first controlled outcome study on short-term therapy for Post Traumatic Stress Disorder in 1989, and has since published continuously on the topic. His main effort goes to bridging the gap between scientific data and service provision in the community.   2014-07-05 15.03.48

What is the most fascinating thing about trauma for you?

I would have to say people’s resilience: how human beings can go through so much, experience the most adverse situations, and still manage to cope, to come out of that – and allow you to help them along the way.

 How did you become interested in the field of trauma?

There are two answers to this question. One is very simple: There was an opening for a trauma psychologist in the paper and I thought to myself “that sounds interesting!” and that’s how I landed in this field. The other story is that my father, though originally a musician, started to take care of Jewish orphans after the war. My parents made the decision to help these children and so we moved from one children’s home to another over the years. Later, my father received the training necessary to be a therapist, my brother and sister became social workers, and I – almost inevitably – became a psychologist (laughs).

Do you have any recommendations for people who want to work in this field? Can you tell us of common mistakes/pitfalls?

In the trauma field the most important piece of advice I can give is don’t do things alone. This is why I founded the trauma center in the Netherlands and the Israeli Centre for Psychotrauma here in Jerusalem. Human connection is unmeasurably important for healing. As a therapist, curiosity is vital – together with really wanting to hear and understand what happened. Also, my advice is to stay open to new ideas and ways of thinking – the moment you think you know everything or that there is only one method, you’re in trouble.

Connecting research with practice can often be a challenge in psychology. How do you think we can best translate research into practice?

Fortunately, I was always able to combine both research and practice: they go together. If I only do research I often lose myself in dozens of theories, variables and statistics and miss the human component. However, if I solely focus on doing therapy I start to overgeneralize patients’ problems and don’t see the individual anymore. I know of clinicians who, when encountering a patient with a new problem, sometimes don’t even go to google scholar anymore! This is why I like the combination, i.e. scientist-practitioners or practicing-scientists (or whatever one might call it) – do both and integrate both.

Society and mental health is a big issue. How would you describe the relation between mental health, therapy and Israeli society regarding trauma?

The Piece of Mind Project
‘Peace of Mind’ is a program created by the Israel Center for the Treatment of Pyschotrauma (ICTP). It is aimed at young people who have served for 3 or more years in high risk combat units in the Israel Defense Forces and provides a pathway back to civilian life. Strengthening resilience and helping to cope with the experienced trauma are at the core of the program. ‘Piece of Mind’ brings in soldiers who have been serving together in the same unit or team.  Read more about the Piece of Mind project here.

During the past 20 years there has been a lot of change in Israel. Society has developed from a lack of recognition of symptoms and PTSD to recognizing these issues and people actually wanting to be treated. To illustrate, simply the fact that we could create the “Peace of Mind Project” was in itself an outcome of a societal process. Another example is the change in Holocaust documentation on television: we’ve moved away from heroism stories of individuals to focusing on the great suffering during this time. Generally, Israeli society has become more open and there has been a move from neglect (“we don’t have trauma”) to recognition: Israeli society is slowly coming to terms with the fact that there is a price to pay for this crazy reality that we live in.

Recently there has been a lot of discussion about the DSM-V and it has been proposed that a new system is necessary – what would you think is important for such a system?

Transdiagnostic Treatment
Emerging conceptualizations of the major emotional disorders increasingly focus on emphasizing commonalities (in phenomenology, diathesis, vulnerabilities) rather than differences (Barlow et al., 2010). Barlow and colleagues (2010) aim to move away from numerous individual treatment protocols for specific disorders towards generalization of the treatment response – as a result, they developed the Unified Protocol (UP). UP is a transdiagnostic, emotion-focused cognitive-behavioral treatment across emotional disorders (Ellard, Fairholme, Boisseau, Farchione, & Barlow, 2010).

At the moment a new direction regarding Assessment and Diagnosis is developing: instead of focusing on one DSM-V diagnosis, it has been suggested to go back to the core mechanism that lies beneath, i.e., emotion regulation. We keep expanding and adding more categories, it is almost as if we want to have as many diagnoses as possible – that doesn’t make a lot of sense. Barlow, for example, has developed a unified transdiagnostic treatment for emotional disorders and I think that will be more effective than becoming broader and broader in terms of diagnosis. (click here for our Interview with David Barlow)

Is there something you would like to mention or share that is important to you?

There is an issue that is not being talked a lot about, namely coming to terms with evil. During the course of my career I have encountered and treated patients who have been horribly abused, became victims of ritual abuse or have had their minds controlled. Still, patients often feel like they are overreacting, saying “but that’s all that happened”. The concept of evil is never taught or talked about and in my opinion this needs to get a place.

References

Barlow, D. H., Farchione, T. J., Fairholme, C. P., Ellard, K. K., Boisseau, C. L., Allen, L. B., & May,  J. T. E. (2010). Unified protocol for transdiagnostic
treatment of emotional disorders:  Therapist guide. Oxford University Press.

Ellard, K. K., Fairholme, C. P., Boisseau, C. L., Farchione, T. J., & Barlow, D. H. (2010). Unified  protocol for the transdiagnostic treatment of emotional disorders: Protocol development and initial outcome data. Cognitive and Behavioral Practice, 17(1), 88-101.

“Set the default to ‘Open'” – Impressions from the OpenCon2014

In November 2014, 150 early-career researchers and students met in Washington D.C. for OpenCon, organized by the Right to Research Coalition, to talk about the movement to open science up – be it through Open Access to published literature, Open Data, or Open Educational Resources. The three day event offered lectures and panels on the state of the open today, but also served as an incubator for the future of the whole debate that spans universities, research funders, and publishers. It was an opportunity for the already experienced advocates and academics to interact with the younger generation of students and researchers interested in these issues. Continue reading

Bayesian Statistics: What is it and Why do we Need it?

prlipohellThere is a revolution in statistics happening: The Bayesian revolution. Psychology students who are interested in research methods (which I hope everyone is!) should know what this revolution is about. Gaining this knowledge now instead of later might spare you lots of misconceptions about statistics as it is usually instructed in psychology, and it might help you gain a deeper understanding of the foundations of statistics. To make sure that you can try out everything you learn immediately, I conducted analysis in the free statistics software R (www.r-project.org; click HERE for a tutorial how to get started with R, and install RStudio for an enhanced R-experience) and I provide the syntax for the analysis directly in the article so you can easily try them out. So let’s jump in: What is “Bayesian Statistics”, and why do we need it? Continue reading

Interview with Prof. David Barlow

Prof. Barlow is a professor of Psychology and Psychiatry at Boston University and founder of  Center for Anxiety and Related Disorders. His research focuses on understanding the nature of anxiety and depression and developing new treatments for emotional disorders. He also developed the Transdiagnostic Treatment of Emotional Disorders david.barlow

What I enjoy most about my job as a researcher …  What I learned very early is that there is nothing I do not enjoy about my job! Continue reading

Crowdsource your research with style

Would you like to collect data quick and efficiently? Would you like to have a sample that generalizes beyond western, educated, industrialized, rich and democratic participants? While you acknowledge social media as a powerful means to distribute your studies, you feel that there must be a “better way”? Then this practical introduction to crowdsourcing is exactly what you need. I will show you how to use Crowdflower, a crowdsourcing platform to attract participants from all over the world to take part in your experiments. However, before we get too excited, let’s quickly go through the relevant terminology. Continue reading

Interview with Prof. Nelson Cowan

Nelson Cowan is a Curators’ Professor of Psychology at the University of Missouri. His research focuses on short-term memory, working memory and selective attention in information processing. Amongst other findings, Cowan is well known for bringing the working memory capacity down from Millers magical 7+/-2 items to a more realistic 3-4 items. cowan_new

What I enjoy most about my job as a researcher … I enjoy the ability to decide what aspect of the human mind to investigate, and how to investigate it.  Continue reading

Interview with Dr. David Klemanski

David Klemanski is Director of the Yale Center for Anxiety and Mood Disorders and lecturer of Psychology and Psychiatry. His research interests include mood and anxiety disorders (e.g., social phobia, generalised anxiety disorder, PTSD) in adolescents. His recent research focuses on individual differences in emotion regulation strategies. droppedImage_1

What I enjoy most about my job as a researcher … On a professional level, I most enjoy the opportunity to contribute to a wider area of knowledge in psychological science.  Continue reading

Interview with Prof. Daniel Simons

Daniel Simons is Professor of Psychology at the University of Illinois. His lab does research on visual cognition, attention, perception, memory, change blindness, metacognition and intuition. He is especially well known for his experiments on inattentional blindness, e.g. the famous invisible gorilla experiment.

dsimons

What I enjoy most about my job as a researcher … I get the most enjoyment from analyzing new data to see what we found. That moment when you learn what you found continues to be rewarding no matter how many studies you’ve done. I also enjoy writing and editing. There are few aspects of the research process I don’t like, actually. Continue reading

Interview with Prof. Ralph Hertwig

Ralph Hertwig is director of the Center for Adaptive Rationality at the Max Planck Institute for Human Development in Berlin. He is well known for his interdisciplinary research on cognitive search, judgment, and decision making under risk and uncertainty. To this end, his lab uses a wide array of methods, ranging from experiments, surveys, and computer simulations to neuroscientific tools. 

Ralph Hertwig

What I enjoy most about my job as a researcher … What I most enjoy is the opportunity to team up with people from other fields or schools of thought and produce something I could never have come up with on my own. Continue reading