Tag Archives: Open Science

Are You Registering That? An Interview with Prof. Chris Chambers

There is no panacea for bad science, but if there were, it would certainly resemble Registered Reports. Registered Reports are a novel publishing format in which authors submit only the introduction, methods, and planned analyses without actually having collected the data. Thus, peer-review only focuses on the soundness of the research proposal and is not contingent on the “significance” of the results (Chambers, 2013). In one strike, this simple idea combats publication bias, researchers’ degrees of freedom, makes apparent the distinction between exploratory and confirmatory research, and calms the researcher’s mind. There are a number of journals offering Registered Reports, and this is arguable the most important step journals can take to push psychological science forward (see also King et al., 2016). For a detailed treatment of Registered Reports, see here, here, here, and Chambers (2015).

Picture of Chris Chambers

Chris Chambers is the initiator of the “Registration Revolution”, the man behind the movement. He has introduced Registered Reports into psychology, has written publicly about the issues we currently face in psychology, and has recently published a book called the “7 Deadly Sins of Psychology” in which he masterfully exposes the shortcomings of current academic customs and inspires change. He is somebody who cares deeply about the future of our field, and he is actively changing it for the better.

We are very excited to present you with an interview with Chris Chambers. How did he become a researcher? Where did he get the idea of Registered Reports from? What is his new book about, and what can we learn from hard sciences such as physics? Find out below!


Tell us a bit about your background. How did you get into Psychology and Cognitive Neuroscience? What is the focus of your research?

Since my teenage years I had been interested in psychology (the Star Trek Next Generation episode “Measure of a Man” left me pondering the mind and consciousness for ages!) but I never really imagined myself as a psychologist or a scientist – those seemed like remote and obscure professions, well out of reach. It wasn’t until the final year of my undergraduate degree that I developed a deep interest in the science of psychology and decided to make a run for it as a career. Applying to do a PhD felt like a very long shot. I have this distinct memory, back in 1999, scrolling down the web page of accepted PhD entrants. I searched in vain for my name among the list of those who had been awarded various prestigious scholarships, and as I neared the bottom I began pondering alternative careers. But then, as if by miracle, there was my name at the end. I was last on the list, the entrant with the lowest successful mark out of the entire cohort. For the next two and half years I tried in vain to replicate a famous US psychologist’s results, and then had to face having this famous psychologist as a negative reviewer of every paper we submitted. One day – about two years into my PhD – my supervisor told me about this grant he’d just been awarded to stimulate people’s brains with electromagnetic fields. He asked if I wanted a job and I jumped at the chance. Finally I could escape Famous Negative Reviewer Who Hated Me! Since then, a large part of my research has been in cognitive neuroscience, with specific interests in attention, consciousness and cognitive control.

You have published an intriguing piece on “physics envy” (here). What can psychology learn from physics, and what can psychologists learn from physicists?

Psychology can learn many lessons from physics and other physical sciences. The physics community hinges reputation on transparency and reproducibility – if your results can’t be repeated then they (and you) won’t be believed. They routinely publish their work in the form of pre-prints and have successfully shaped their journals to fit with their working culture. Replication studies are normal practice, and when conducted are seen as a compliment to the importance of the original work rather than (as in psychology) a threat or insult to the original researcher. Physicists I talk to are bemused by our obsession with impact factors, h-indices, and authorship order – they see these as shallow indicators for bureaucrats and the small minded. There are career pressures in physics, no doubt, but at the risk of over-simplifying, it seems to me that the incentives for individual scientists are in broad alignment with the scientific objectives of the community. In psychology, these incentives stand in opposition.

One of your areas of interest is in the public understanding of science. Can you provide a brief primer of the psychological ideas within this field of research?

The way scientists communicate with the public is crucial in so many ways and a large part of my work. In terms of outreach, one of my goals on the Guardian science blog network is to help bridge this gap. We’ve also been exploring science communication in our research. Through the Insciout project we’ve been investigating the extent to which press releases about science and health contribute to hype in news reporting, and the evidence suggests that most exaggeration we see in the news begins life in press releases issued by universities and academic journals. We’ve also been looking at how readers interpret common phrases used in science and health reporting, such as “X can cause Y” or “X increases risk of Y”, to determine whether the wording used in news headlines leads readers to conclude that results are more deterministic (i.e. causal) than the study methods allow. Our hope is that this work can lead to evidence-based guidelines for preparation of science and health PR material by universities and journals.

I’m also very interested in mechanisms for promoting evidence-based policy more generally. Here in the UK I’m working with several colleagues to establish a new Evidence Information Service for connecting research academics and policy makers, with the aim to provide parliamentarians with a rapid source of advice and consultation. We’re currently undertaking a large-scale survey of how the academic community feels about this concept – the survey can be completed here.

You have recently published a book titled “The 7 Deadly Sins of Psychology”. What are the sins and how can psychologists redeem themselves?

The sins, in order, are bias, hidden flexibility, unreliability, data hoarding, corruptibility, internment and bean counting. At the broadest level, the path to redemption will require wide adoption of open research practices such as a study preregistration, open data and open materials, and wholesale revision of the systems we use to determine career progression, such as authorship rank, journal rank, and grant capture. We also need to establish robust provisions for detecting and deterring academic fraud while at the same time instituting genuine protections for whistleblowers.

How did you arrive at the idea of Registered Reports for Psychology? What was the initial response from journals that you have approached? How has the perception of Registered Reports changed over the years?

After many years of being trained in the current system, I basically just had enough of publication bias and the “academic game” in psychology – a game where publishing neat stories in prestigious journals and attracting large amounts of grant funding is more rewarded than being accurate and honest. I reached a breaking point (which I write about in the book) and decided that I was either going to do something else with my life or try to change my environment. I opted for the latter and journal-based preregistration – what later became known as Registered Reports – seemed like the best way to do it. The general concept behind Registered Reports had been suggested, on and off, for about 50 years but nobody had yet managed to implement it. I got extremely lucky in being able to push it into the mainstream at the journal Cortex, thanks in no small part to the support of chief editor Sergio Della Sala.

The initial response from journals was quite cautious. Many were – and still are – concerned about whether Registered Reports will somehow produce lower quality science or reduce their impact factors. In reality, they produce what in my view are among the highest quality empirical papers you will see in their respective fields – they are rigorously reviewed with transparent, high-powered methods, and the evidence also suggests that they are cited well above average. Over the last four years we’ve seen more than 50 journals adopt the format (including in some prominent journals such as Nature Human Behaviour and BMC Biology) and the community has warmed up to them as published examples have begun appearing. Many journals are now seeing them as a strength and a sign that they value reproducible open science. They are realising that adding Registered Reports to their arsenal is a small and simple step for attracting high-quality research, and that having them widely available is potentially a giant leap for science as a whole.

Max Planck, the famous German Physicist, once said that science advances a funeral at a time. Let’s hope that is not true —  we simply don’t have the time for that. What skills, ideas, and practices should the next generation of psychological researchers be familiar and competent with? What further resources can you recommend?

I agree – there is no time to wait for funerals, especially in our unstable political climate. The world is changing quickly and science needs to adapt. I believe young scientists can protect themselves in two ways: first, by learning open science and robust methods now. Journals and funders are becoming increasingly cognisant of the need to ensure greater reproducibility and many of the measures that are currently optional will inevitably become mandatory. So make sure you learn how to archive your data, or preregister your protocol. Learn R and become familiar with the underlying philosophy of frequentist and Bayesian hypothesis testing. Do you understand what a p value is? What power is and isn’t? What a Bayes factor tells you? My second recommendation is to recognise these tumultuous times in science for what they are: a political revolution. It’s easy for more vulnerable members of a community to be crushed during a revolution, especially if isolated, so young scientists need to unionise behind open science to ensure that their voices are heard. Form teams to help shape the reforms that you want to see in the years ahead, whether that’s Registered Reports or open data and materials in peer review, or becoming a COS Ambassador. One day, not long from now, all this will be yours so make sure the system works for you and your community.

Fabian Dablander

Fabian Dablander is currently finishing his thesis in Cognitive Science at the University of Tübingen and Daimler Research & Development on validating driving simulations. He is interested in innovative ways of data collection, Bayesian statistics, open science, and effective altruism. You can find him on Twitter @fdabl.

More Posts - Website

Facebooktwitterrss

Publishing a Registered Report as a Postgraduate Researcher

Registered Reports (RRs) are a new publishing format pioneered by the journal Cortex (Chambers 2013). This publication format emphasises the process of rigorous research, rather than the results, in an attempt to avoid questionable research practices such as p-hacking and HARK-ing, which ultimately reduce the reproducibility of research and contribute to publication bias in cognitive science (Chambers et al. 2014). A recent JEPS post by Dablander (2016) and JEPS’ own editorial for adopting RRs (King et al. 2016) have given a detailed explanation of the RR process. However, you may have thought that publishing a RR is reserved for only senior scientists, and is not a viable option for a postgraduate student. In fact, 5 out of 6 of the first RRs published by Cortex have had post-graduate students as authors, and publishing by RR offers postgraduates and early career researchers many unique benefits.

In the following article you will hear about the experience of Dr. Hannah Hobson, who published a RR in the journal Cortex as a part of her PhD project. I spoke to Hannah about the planning that was involved, the useful reviewer comments she received, and asked her what tips she has for postgraduates interested in publishing a RR. Furthermore, there are some comments from Professor Chris Chambers who is a section editor for Cortex on how postgraduates can benefit from using this publishing format.

Interview with Dr. Hannah Hobson

Hannah completed her PhD project on children’s behavioural imitation skills, and potential neurophysiological measures of the brain systems underlying imitation. Her PhD was based at the University of Oxford, under the supervision of Professor Dorothy Bishop. During her studies, Hannah became interested in mu suppression, an EEG measure purported to reflect the activity of the human mirror neuron system. However, she was concerned that much of research on mu suppression suffered from methodological problems, despite this measure being widely used in social cognitive neuroscience. Hannah and Dorothy thought it would be appropriate to publish a RR to focus on some of these issues. This study was published in the journal Cortex, and investigated whether mu suppression is a good measure of the human mirror neuron system (Hobson and Bishop 2016). I spoke to Hannah about her project and what her experience of publishing a RR was like during her PhD.

As you can hear from Hannah’s experience, publishing a RR was beneficial in ways that would not be possible with standard publishing formats. However, they are not suitable for every study. Drawing from Hannah’s experience and Chris Chambers’ role in promoting RRs, the main strengths and concerns for postgraduate students publishing a RR are summarised below.

Strengths

Reproducible findings

It has been highlighted that the majority of psychological studies suffer from low power. As well as limiting the chances of finding an effect, low-powered studies are more likely to lack reproducibility as they overemphasise the effect size (Button et al. 2013). As a part of the stage one submission, a formal power analysis needs to be performed to identify the number of participants required for a high powered study (>90%). Therefore, PhD studies published as RRs will have greater power and reproducibility in comparison to the average unregistered study (Chambers et al. 2014).

More certainty over publications

The majority of published PhD studies begin to emerge during the final year or during your first post-doctoral position. As the academic job markets becomes ever more competitive, publications are essential. As Professor Chambers notes, RRs “enable PhD students to list provisionally accepted papers on their CVs by the time they submit their PhDs”. Employers will see greater certainty in a RR with stage one approval than the ‘in preparation’ listed next to innumerable papers following the standard publishing format.

Lower rejection rate at stage two submission

Although reaching stage one approval is more difficult due to the strict methodological rigour required, there is greater certainty in the eventual outcome of the paper once you have in-principal acceptance. In Cortex, approximately 90% of unregistered reports are rejected upon submission, but only 10% of RRs which reach stage one review have been rejected, with none being rejected so far with in-principal acceptance.

“This means you are far more likely to get your paper accepted at the first journal you submit to, reducing the tedious and time-wasting exercise of submitting down a chain of journals after your work is finished and you may already be competing on the job market”. – Professor Chris Chambers

As Dorothy Bishop explains in her blog, once you have in-principle acceptance you are in control of the timing of the publication (Bishop 2016). This means that you will have a publication in print during your PhD, as opposed to starting to submit papers towards the end which may only be ‘in preparation’ by the time of your viva voce.

Constructive reviewer comments

As the rationale and methodology is peer-reviewed before the data-collection process, reviewers are able to make suggestions to improve the design of your study. In Hannah’s experience, a reviewer pointed out an issue with her control stimuli. If she had conducted the study following the standard format, reviewers would only be able to point this out retrospectively when there is no option to change it. This experience will also be invaluable during your viva voce. As you defend your work in front of the examiners, you know your study has already gone through several rounds of review, so you can be confident in how robust it is.

Things to consider

Time restraints

Recruiting and testing participants is a lengthy process, and you often encounter a series of setbacks. If you are already in the middle of your PhD, then you may not have time to go through stage one submission before collecting your data. In Hannah’s case, publishing a RR was identified early in the project which provided a sufficient amount of time to complete it during her PhD. If you are interested in RRs, it is advisable to start the submission process as early into your PhD as possible. You may even want to start the discussion during the interview process.

Ethics merry-go-round

During stage one submission, you need to provide evidence that you already have ethical approval. If the reviewers want you to make changes to the methodology, this may necessitate amending your ethics application. In busy periods, this process of going back and forth between the reviewers and your ethics committee can become time-consuming. As time constraints is the pertinent concern for postgraduates publishing a RR, this is an additional hurdle that must be negotiated. Whilst there is no easy solution to this problem, aiming to publish a RR must be identified early in your project to ensure you will have enough time, and have a back-up plan prepared for if things do not work out.

RRs are not available in every journal

Although there has been a surge in journals offering RRs, they are not available in every one. Your research might be highly specialised and the key journal in your area may not offer the option of a RR. If your research does not fit into the scope of a journal that offers RRs, you may not have the option to publish your study as a RR. Whist there is no simple solution for this, there is a regular list of journals offering RRs on the Open Science Framework (OSF).

Supervisor conflict

Although there are a number of prominent researchers behind the initiative (Guardian Open Letter 2013), there is not universal agreement with some researchers voicing concerns (Scott 2013, although see Chambers et al. 2014 for a rebuttal to many common concerns). There have been some vocal critics of RRs, and one of these critics might end up being your supervisor. If you want to conduct a RR as a part of your PhD and your supervisor is against it, there may be some conflict. Again, it is best to identify early on in your PhD if you want to publish a RR, and make sure both you and your supervisor are on the same page.

Conclusion

Publishing a RR as a postgraduate researcher is a feasible option that provides several benefits, both to the individual student and to wider scientific progress. Research published as a RR is more likely to produce reproducible findings, due to the necessary high level of power, reviewers’ critique before data collection, and guards against questionable research practices such as p-hacking or HARK-ing. Providing the work is carried out as agreed, a study that has achieved stage one approval is likely to be published, allowing students the opportunity to publish their hard work, even if the findings are negative. Moreover, going through several rounds of peer-review on the proposed methodology provides an additional layer of rigour (good for science), that aids your defence in your viva voce (good for you). Of course, it is not all plain sailing and there are a several considerations students will need to make before embarking on an RR. Nonetheless, despite these concerns, this publishing format is a step in the right direction for ensuring that robust research is being conducted right down to the level of postgraduate students.

If you like the idea but do not think formal pre-registration with a journal is suitable for your project, perhaps consider using the OSF. The OSF is a site where researchers can timestamp their hypotheses and planned analyses, allowing them to develop hypothesis-driven research habits. In one research group, it is necessary for all studies ranging from undergraduate projects to grant-funded projects to be registered on third-party websites such as the OSF (Munafò 2015). Some researchers such as Chris Chambers have even made it a requirement for applicants wanting to join their group to demonstrate a prior commitment to open science practices (Chambers 2016). Starting to pre-register your studies and publish RRs as a postgraduate student demonstrates this commitment, and will prove to be crucial as open science practices become an essential criterion in recruitment.

“To junior researchers I would say that pre-registration — especially as a Registered Report — is an ideal option for publishing high-quality, hypothesis-driven research that reflects an investment both in good science and your future career” – Professor Chris Chambers 

Pre-registration and RRs are both initiatives to improve the rigour and transparency of psychological science (Munafò et al. 2014). These initiatives are available to us as research students, and it is not just the responsibility of senior academics to fight against questionable research practises. We can join in too.

Acknowledgements

Thank you to Dr. Hannah Hobson who was happy to talk about her experience as a PhD student and for her expertise in recording the interview. Hannah also helped to write and revise the post. I would also like to thank Professor Chris Chambers for taking the time to provide some comments for the post.

James Bartlett

James Bartlett

I'm James Bartlett, a PhD student at Coventry University, UK. The aim of my project is to create a profile of cognitive mechanisms associated with substance use in light and heavy smokers. I keep myself occupied outside of academia by playing hockey, or watching ice hockey. You can also find me on Twitter (@JamesEBartlett).

More Posts - Website

Facebooktwitterrss