In the past, we have talked a lot about Registered Reports and their potential to increase the rigor and reproducibility of psychological science (see here, here, and here). In a previous blog post, James Bartlett interviewed Dr. Hannah Hobson, who published a Registered Report as part of her PhD project.
In this blog post, we talk with Tatiana Kvetnaya who received her Bachelor degree from the University of Tübingen, and who is currently pursuing her graduate studies at the Goethe University Frankfurt. Excitingly, Tatiana recently published her bachelor thesis as a Registered Report with the Journal of European Psychology Students. Below, she recounts how she first came in contact with Registered Reports, her experience publishing one herself, and tips for students thinking about doing the same.
Could you tell us a bit about the research you published with us?
In my thesis project, I conducted a direct replication of an experiment by Jones, Farrand, Stuart, and Morris (1995), which revolved around the irrelevant speech effect. It refers to the phenomenon that unrelated background noise or speech disrupts the memory performance in tasks in which the serial order of visual items (e.g., a sequence of letters) has to be remembered. There have been two predominant models, on the basis of which an explanation for this effect has been proposed: The working-memory model by Baddeley & Hitch (1974), which assumes that the structure of short-term memory is modular, and Jones et al.’s (1995) changing state hypothesis, which proposes that short-term memory is unitary. Basically, these two theories make contradictory predictions about the way irrelevant speech is expected to affect memory performance. In a series of experiments, Jones et al. (1995) presented evidence which was more in favor of their theory, rather than that of Baddeley and Hitch (1974).
So in my thesis project, I conducted a replication of one of the two experimental conditions from Jones et al.’s (1995) final experiment within the paper. This condition was especially crucial because it tested the influence of irrelevant speech on spatial working memory – something that I could hardly find amongst the published literature.
While in the original research, the results were in line with Jones et al.’s (1995) predictions, my replication attempt painted a different picture: The study yielded a null result, which does not support the original authors’ theory. However, this once more highlighted the importance of sufficiently large power and sample sizes in psychological research.
Nonetheless, this result would have been frustrating to me – if it would not have been pre-registered. In my experience, it seems to be a tacitly accepted preconception that for most undergraduate research projects, “nothing will come out if it anyway!”, and students are told to bury their hopes for a (statistically) significant finding in their thesis projects early on. This carries the implicit judgment that not getting a significant result is an indicator of bad research quality, for which, in turn, the undergraduate students’ (lack of) research skills are to blame. Sadly, this devalues not only the efforts of student researchers, but also the importance of null results as such.
This is one of the reasons I am glad that the Journal of European Psychology Students (JEPS) now is amongst those journals which offer this publication format – I believe that Registered Reports add to the credibility of results from undergraduate students and professional researchers alike.
What drew you to submit a Registered Report? Can you recap the experience of the whole process for us –- from first idea to final manuscript edits?
The first time I learned about Registered Reports was shortly after the results of the Reproducibility Project: Psychology (Open Science Collaboration, 2015) were published, and when the ‘reproducibility crisis’ took off in the media. The public debate around it sparked my interest, and I vigorously read up on what is wrong with psychological research methods, and on the various suggestions on how these can be improved. So I was more than happy when my supervisor, Florian Wickelmaier, suggested that I could preregister and conduct a direct replication study with JEPS. I was intrigued to contribute to the introduction of a new, promising publication format in the research landscape. I also shared Florians notion that direct replications are crucial, and that conducting one myself would teach me a lot about the underlying research methodology – which turned out to be the case.
The specific paper we chose to replicate has been published in 1995. Yet despite its interesting implications, I could not find a single direct replication of it that was published since then. Even though cognitive psychology appears to be more replicable than other fields of psychology, such as social psychology (as suggested by Open Science Collaboration, 2015), that doesn’t mean that the field of cognitive psychology may not suffer from the same problems and questionable research practices. These considerations guided the choice of research I wanted to regard in my thesis.
Once this choice was made, the preparation for the Stage 1 submission began. This required an extensive literature review, planning the specifics of the experiment and the analyses to be made, writing the R code for the data analysis, and testing the whole thing so everything was ready for Stage 1 submission. Thankfully, during this stage of the project, the original first author Prof. Dylan Jones also provided his feedback, so we could be sure that the experiment will be conducted just as originally intended. Thanks to the peer review in this stage of the publishing process, we also recognized that it was necessary to conduct a small-scale pilot study before submitting a revised version of the manuscript.
Two months after the first submission, I was happy to learn that the manuscript was granted in principle acceptance! Finally, data collection could begin, which was quick and easy thanks to the preparations made in advance. A few weeks later, once the much-anticipated results were in, I could begin writing the results and discussion section for the final Stage 2 manuscript. Once again, the reviewers’ and editor’s comments really helped to identify flaws in the manuscript that might have otherwise gone unnoticed. Overall, this additional feedback – having not only one (the supervisor’s), but even three more pairs of professionally skilled eyes to go over my work – greatly helped to improve the quality of the outcome.
What was the most difficult part of doing this type of research?
The fact that my study was a direct replication was very relevant to the challenges I encountered during this project. People might assume that conducting replications is not very demanding because you’re only repeating what others have done before you, but I disagree. In my opinion, the way of thinking and working that is required in order to conduct a direct replication is different from conducting your own, original research. In every scientific writing class, students learn how essential it is for papers to be written in a way others could potentially repeat the procedure. Yet you only really get an idea of how important this is when you try to replicate one yourself.
This also applies to the way of thinking required to plan out all aspects of a study in advance, before even viewing the data, which is unusual for most. This requires attention to detail, and thorough consideration of the analyses that are to be made, which effect sizes are realistically to be expected, and so on. This really causes you to reflect on the statistical tools you are using and why, instead of simply applying so-called ‘cookbook statistics’.
Another challenge was to consider the tradeoff between exact replication and what we thought might be most suitable to find the original effect, which is not always congruent. For example, we briefly considered to include an additional experimental condition because we believed it might make it easier to detect the original effect. However, we dropped that idea, since that would have made the replication more conceptual than direct. However, unlike in the original study, we decided to give our participants a performance-dependent reward, because we concluded it would improve data quality without deviating too much from the original study.
All of this is relevant for students who just start their scientific education, which is why I believe that direct replications should be part of any general curriculum for aspiring researchers.
Do you have any tips for students who are thinking about doing a Registered Report themselves? What are potential obstacles?
I can wholeheartedly encourage other students to go for it! As I already mentioned, the feedback I received from the editor and reviewers was very constructive and greatly helped to increase the quality of my thesis paper, and experiencing the whole publishing process first-hand was very exciting and educational.
When it comes to thesis projects, one potential obstacle might be the limited time students get to complete a thesis or research project within their degree. For German students, this is usually about 6-8 months. This might discourage a student from pre-registering their work because they might be deterred by the prospect of waiting for the outcome of their submission before they can start collecting data. However, I had the impression that JEPS is very aware of these problems and is committed to providing feedback very quickly in order to meet students’ needs. For example, about three weeks passed after my Stage 1 submission until I received the first round of reviewer feedback. Once I had submitted the revised manuscript, it was another week until I was granted in principle acceptance.
So if you start early and plan accordingly – preferably in close collaboration with your supervisor – I believe it’s quite realistic to complete a thesis project in time. The time and effort spent on elaborating your methods for pre-registration might actually save you lots of time you would otherwise require in the end, after the data is collected.
Another obstacle might be students’ concern about whether investing time and effort into such a project might negatively affect their grades. In my case, my supervisor made it clear from the beginning that my thesis paper will be graded independently from the final outcome of the review process. That’s a sensible thing to agree on to make sure that pre-registering your research project will be an enriching experience, rather than adding extra pressure.
What are your future career plans?
In my Master’s studies, which I have recently begun, I try and continue to broaden my knowledge about research methods (both quantitative and qualitative, confirmatory and exploratory). Besides that, I have lots of academic interests that I seek to explore: For example, I am interested in how psychological constructs come into being, how they can be assessed most validly, and how knowledge about this reflects back to the ‘real’, everyday world. Once I have completed my Master’s degree, I can definitely see myself pursuing these questions further in the course of a PhD, and see where it will take me from there.
Any final comments?
I would like to thank the JEPS for doing the great work it does, and for this great opportunity to advance my research skills by publishing with them. Unfortunately, I have the impression that there are still so many people – students and instructors alike – who don’t yet know about the opportunities JEPS provides for student researchers. So I will definitely continue to spread the word!
Fabian Dablander
Fabian Dablander is doing a PhD at the Department of Psychological Methods at the University of Amsterdam. You can find more information at https://fdabl.github.io/.
Follow Me:
