Journal 'Psychological Science' Seeking Preregistered Replications

Making a statement in the ongoing “replication” or “reproducibility crisis,” the journal Psychological Science will now accept a special class of research–based papers that report on attempts to re-create experiments that had influential findings and that were first published in Psychological Science.This new category of “preregistered direct replications,” or PDRs, aims “to create conditions that experts agree test the same hypotheses in essentially the same way as the original study,” explained D. Stephen Lindsay, a cognitive psychologist at the University of Victoria and editor of Psychological Science.That psychology in particular is in need of a replication tonic has been widely accepted. “To me,” Jim Grange, a senior lecturer in psychology at Keele University, wrote last year, “it is clear that there is a reproducibility crisis in psychological science, and across all sciences. Murmurings of low reproducibility began in 2011 – the ‘year of horrors’ for psychology – with a high profile fraud case. But since then, The Open Science Collaboration has published the findings of a large-scale effort to closely replicate 100 studies in psychology. Only 36 percent of them could be replicated.”Under the PDR description, the ‘direct’ part of the replication rubric refers to sticking to the original methodology as closely as possible. “It is impossible to conduct a study twice in exactly the same way,” Lindsay said in a letter announcing the PDRs, “and doing so would not always be desirable (e.g., often it would be better to test a larger number of subjects than in the original study; if the original study included some problematic items or unclear instructions, it is probably best to correct those shortcomings; if the original and replication studies were conducted in different cultural contexts then it may be appropriate to change surface-level aspects of the procedure).”It is expected, Lindsay said, that the author of the target piece will be invited to provide a review of the replication, along with at least two independent experts. And in keeping with the ‘preregistered’ aspect of the PDR, researchers are asked to submit proposals for review before data collection begins. “A virtue of pre-data-collection submissions is that decisions about scientific merit are independent of how the results come out,” he said. “Another advantage is that reviewers and the editor have an opportunity to improve a study before it is conducted. We may therefore at some point in the future require that PDRs be submitted as proposals prior to data collection.  But for now we will also consider PDRs that have already been conducted if they were preregistered.” And in keeping with an earlier announcement also focused on transparency, Lindsay said would-be authors are asked to provide the data they use in their published analysis to reviewers.Lindsay made clear that while the replication needed to leverage an article that originally appeared in Psychological Science, merely having appeared there was not sufficient for a PDR. “The primary criterion is general theoretical significance,” he emphasized.Still, he said, the journal – which takes its lumps from some observers like Andrew Gelman – has a responsibility to not turn its back on what it’s published. “One of the motivations for adding PDRs is the belief that a journal is responsible for the works it publishes (as per Sanjay Srivastava’s ‘Pottery Barn rule’ blog post). That said,” he continued, “our goal is not to publish ‘gotchas!’ Rather, inviting PDRs is one of many steps taken to increase the extent to which this journal contributes to efforts to make psychology a more cumulative science.”Regardless with whether a replication confirms or question the original findings, the outcomes from the PDRs will be “valuable and informative.” He cited a 2009 column by then Association for Psychological Science President Walter Mischel which noted “replications sometimes yield more nuanced results that spark new hypotheses and contribute to the elaboration of psychological theories.”The PDRs will launch with a replication of a 2008 an fMRI study Psychological Science published in 2008, said journal editor Steve Lindsay. That original study, conducted by William A. Cunningham, Jay J. Van Bavel and Ingrid R. Johnsen, detailed evidence that processing goals can modulate activity in the amygdala, a part of the brain that influences memory, emotions and decision-making. The new article explains how the University of Denver’s Kateri McRae and the University of California, Los Angeles’ Daniel Lumian conducted a high-powered replication of that experiment -- in consultation with Cunningham and Van Bavel – and in the process replicated the original’s central finding.“This article,” said Lindsay, “is a fine model of a preregistered direct replication,” in part because of its greater statistical power and multiple converging analyses compared to the original.Lindsay said PDRs are not the same as the registered replication reports that have appeared in the sister journal, Perspectives on Psychological Science, and which along with other multi-lab empirical papers will appear in the new journal, Advances in Methods and Practices in Psychological Science.