Many negative consequences followed the discovery of the massive fraud by social psychologist Diederik Stapel in 2011, including damage to the reputation of science in general and psychology in particular. One of the few positive outcomes of his fraud is that it has brought to light the importance of replication in science.

A recent failed replication of a high-impact result in social psychology highlights this problem.

A key finding in social psychology is the phenomenon of unconscious priming. Accumulated evidence has shown that behavior can be unconsciously influenced (primed) by a previous stimulus that activates certain stereotypes, personality traits, or other concepts. A famous example, often included in textbooks, is the study by Dijksterhuis et al. (1998) . In this experiment the researchers instructed participants to take a few minutes to write down characteristics of either a typical professor or a typical soccer hooligan. Subsequently, they asked the participants to fill out a general knowledge questionnaire. Participants who wrote about professors scored significantly higher on this test than participants who wrote about soccer hooligans.

These findings are quite spectacular: What if students could improve exam scores simply by thinking about professors for a little while? However, the results also come across as a bit surprising and counter-intuitive. This led Shanks et al. (2013) to do a direct replication of this study to further investigate if the findings were, in fact, real. In nine different experiments, using largely the same methods as described by Dijksterhuis et al., they found no evidence that priming with the concept of a professor increases test scores.

There are several explanations for this replication failure, which are given in the Discussion section of the article. These are:

• The experimental manipulations of Shanks et al. were not as good as those by Dijksterhuis et al. (unlikely because they followed very similar methods).
• The participants of Dijksterhuis et al. figured out the goal of the experiment, and were trying to ‘help’ the experimenter (participant bias) (unlikely because then this effect should also be present in the Shanks et al. study).
• The experiment of Shanks et al. lacked the power to detect small but real priming effects (unlikely because of very large effect sizes reported in Dijksterhuis et al., and Shanks et al. sometimes used even larger sample sizes).
• The effects by Dijksterhuis et al. are false positives.

According to Shanks et al., the latter option is the most likely explanation. There are several possible causes of false positives in research, of which arguably the most important are post-hoc selection of data or analyses, and publication bias (and of course, fraud as in the case of Diederik Stapel). Publication bias is the phenomenon that it is much easier for a researcher to get ‘sexy’ results published than a boring replication paper. As a result, negative replication attempts often disappear into a scientist’s file drawer.

Thankfully, and partly because of Stapel’s fraud, replication has received a lot more attention lately. For psychologists there is now a virtual file drawer where they can share unpublished replication attempts. Additionally, there is now a journal publishing ‘Registered Replication Reports’. The idea is that several research groups independently replicate an important finding, using pre-registered methods. The results are published regardless of the outcome.

In my opinion, this is an important step forwards that helps ensure that scientific results, particularly those from the field of psychology, retain their credibility.