The past couple of years have been somewhat tumultuous for psychology. With the revelation of several high profile cases of fraud, the field has come under scrutiny, not least from psychologists themselves. Most notably, Simmons, Nelson and Simonsohn (2011) showed how the amount of possible decisions in the research process and flexibility researchers normally have can make finding significant findings increasingly likely (see their paper for recommendations to counteract this). It is likely that there are many cases of false positives (or Type 1 error) in the psychological literature (and other bodies of literature). Such errors in the literature, whether resulting from fraud or cultural norms in research practice, are difficult to remove due to the combination of the reluctance of journals to publish exact replications (see previous post on publication bias) and the reluctance of journals to publish null results (which exacerbates the file drawer effect). Thankfully, some initiatives have been established to counter this risk to the credibility of psychology in the form of systematic efforts at carrying out and publishing replications. However, negative attitudes towards replication remain. Many established researchers do not see any incentive in replication (Makel, Plucker & Hegarty, 2012). It is a waste of time that could be spent on their own projects and is perceived as more likely to cause frustration in their colleagues then be rewarded with a publication. This is where you, the students, can step in. Carrying out a replication is a great way for a student to hone his or her research skills, while providing a valuable service to psychology. Before exploring the role students could have in this issue, let’s examine exactly what replication is, the situation of replication in psychology currently, and the efforts that have been made to advocate for it.
What are replication studies?
First of all, it would be useful to define exactly what we mean by replication. It may be seen as the act of repeating an experimental procedure with the aim of establishing the truth. However, a wider view of the concept allows for the distinction between direct replication and conceptual replication (Schmidt, 2009). Direct replication is the repetition of an experimental procedure to as exact a degree as possible. This means that, as far as possible, the same equipment, material, stimuli, design and statistical analysis should be used. Conceptual replication is the use of different methods to repeat the test of a hypothesis or experimental result (Schmidt, 2009). For example, you might test the hypothesis that working memory training increases fluid intelligence using a different type of working memory training task to the one used by Jaeggi and her colleagues (Jaeggi, Buschkuehl, Jonides, & Perrig, 2008). However, conceptual replication is seen as problematic by some. Direct replications have more confirmatory power as it need not be assumed that the same phenomenon is being tested. Successful replications confirm the findings of the original studies while unsuccessful replications fail to confirm them.
Is there a place for replication studies in psychology?
A recent study by Makel, Plucker and Hegarty (2012) provides an interesting illustration of the place of replication in psychology up to now. The complete set of publications from one hundred top-ranking psychology journals (in terms of 5-year impact factor) were examined for mention of the term replication. Just 1.6% (5,051 out of 321,411) of these articles mentioned replication. A closer look revealed some curious trends. When a random sample of 500 articles mentioning replication was examined in more detail, 68.4% were found to be actual replications leading to a probable replication rate of 1.07% for psychology. Only 14% of actual replications were direct replications. 78.9% were successful replications, which may reflect publication bias as much as the general robustness of findings. Notably, there was high authorship overlap. 52.9% of replications were conducted by the same research team. This was defined as the overlap of at least one author. What makes this important is that authorship overlap was significantly related to the success of a replication. This may be an artefact of the file drawer effect. Authors may be unwilling to publish data which contradicts their previous results. This study also revealed some positives regarding replication in psychology. It showed that psychology has a similar replication rate to other fields including business, marketing and communication and in contrast to these fields the replication rate is increasing. It also showed that citation statistics suggest that replications are valued by the scientific community with a median citation count of 17 evident for replication articles.
So, we have seen that there is a need for more replications to be published, regardless of whether they are successful or not, and that more independent replications need to be published. However, there remains the difficulty of getting replications published. Publishing successful replications is seen as uninteresting as they are seen as taking up space in a journal which could be used for new knowledge. Publishing replications with null results is also difficult, though unsuccessful replications of unconventional or particularly interesting studies are more likely to be published (Schmidt, 2009). However, the situation is changing. Two groups of psychologist have established online initiatives which provide a place for replications, whether they are successful or not. Each of these has different aims and strengths but both are particularly admirable attempts to protect the field of psychology. These initiatives are Psych File Drawer, a general archive of replication attempts, and the Reproducibility Project, a systematic replication effort targeting articles published in the 2008 volumes of three of the most prestigious psychology journals.
The Psych File Drawer provides a repository where replications can be uploaded and viewed. This allows for them to be included in meta-analyses. There are currently 19 replications available there. It also provides forums for discussion of specific effects where researchers can communicate about replication attempts and gain valuable advice from peers. Most data is uploaded allowing users to run their own analyses. Check out their top 20 list of articles that users would like to see replicated.
The Reproducibility Project is a similar but far more systematic initiative. Taking articles solely from the 2008 volumes of the Journal of Personality and Social Psychology, Psychological Science, and Journal of Experimental Psychology: Learning, Memory, and Cognition, the project aims to answer key questions about the reproducibility of findings in psychology and replication itself. Researchers can access a structured protocol for designing and running a direct, high-powered replication of a key effect from the selected articles. This project will investigate predictors of successful replication, for example publishing journal, number of conceptual or direct replications in the published literature, citation impact of the original article, closeness of the replication to the original circumstances (Open Science Collaboration, 2012).
Our role as psychology students
It is heartening to see that there are genuine attempts being made to improve the field of psychology from within but how can students help in this endeavour and why are students especially suited to the task? I believe that using undergraduate research projects for direct replication attempts would be of great benefit to students, educators, and researchers, as well as the integrity of the field of psychology.
First of all, it would save time for all involved in this process. The first task for any undergraduate undertaking a research project is deciding what it is they are going to conduct their research on. This involves selecting an area of interest, learning the body of literature and then figuring out a novel addition to this literature. Bear in mind, undergraduates usually have less than nine months to do so, get ethical approval (if necessary), gather materials, collect data, analyse data, and report it. It is a big ask for someone just beginning their journey in research and it can often be very stressful. Much time and stress can be saved for students by focusing on replicating a particular study, for which an original and interesting question has already been generated, and becoming familiar with the literature supporting that study. This will in turn save time for those educators who are supervising undergraduate projects.
Second, it will even out the focus of undergraduate projects. From my experiences, methodological and statistical ability is to some extent secondary to the conceptual aspects of a project and the quest for novelty and originality. By carrying out a replication students will have the opportunity to hone their methodological and statistical skills as this is where the focus is directed. Attention to such details is crucial to replication. While this is also true of original projects, it can be lacking due to timing and focus issues referred to above. Frank and Saxe (2012) also noted that when students read with intent to replicate, they develop a keener eye for a well-written and detailed scientific report.
Thirdly, a well-run replication is a genuine contribution to the field of psychology. Not to undermine original undergraduate research, but just 10% of current undergraduate research is reported beyond the classroom (Perlman & McCann, 2005). A well-run replication, successful or not, should be submitted for publication and uploaded to Psych File Drawer (which explicitly encourages replication by students).
A system for replication in undergraduate projects has been suggested (Grahe et al., 2012). They suggest that educators provide a question-list or menu of possible studies to replicate. More recent, “cutting edge” studies are suggested to provoke the interest of students more (Frank & Saxe, 2012). Use of similar lists across multiple universities in different countries would provide even richer information about the reproducibility of effects. Following the selection of a study, the student should write a proposal for replication and then, if accepted, plan, carry out and write up the study.
Of course, there are limitations to such an approach. Studies are more or less likely to be replicated depending on ease of undertaking, time needed, availability of equipment and materials and access to special populations (Frank & Saxe, 2012). However, this approach could result in a larger number of competent student researchers who have made a real contribution to their field. I believe that the benefits of this approach outweigh the limitations and that students can ably join others in helping to protect psychology.
For more info on replication, see the special issue of The Psychologist, the recent special issue of Perspectives on Psychology, and see blogs such as Not Exactly Rocket Science, Neurochambers, and Neuroskeptic to name a few.
References
Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7(6), 600–604. doi: 10.1177/1745691612460686
Grahe, J. E., Reifman, A., Hermann, A. D., Walker, M., Oleson, K. C., Nario-Redmond, M., & Wiebe, R. P. (2012). Harnessing the undiscovered resource of student research projects. Perspectives on Psychological Science, 7(6), 605–607. doi: 10.1177/1745691612459057
Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008). Improving fluid intelligence with training on working memory. Proceedings of the National Academy of Sciences, 105(19), 6829–6833. doi: 10.1073/pnas.0801268105
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542. doi: 10.1177/1745691612460688
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.
Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90. doi: 10.1037/a0015108
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi: 10.1177/0956797611417632
Picture taken from: http://morguefile.com/archive.
______________________________________________________________________________________
Chris Noone is a PhD student at the School of Psychology at the National University of Ireland, Galway. His research focuses on the effects of mood on higher-order cognition. He is very engaged in working for EFPSA as the Member Representative Coordinator on the Board of Management.
