We agree with the authors' arguments to make replication mainstream but contend that the poor replication record is symptomatic of a pre-paradigmatic science. Reliable replication in psychology requires abandoning group-level p-value testing in favor of real-time predictions of behaviors, mental and brain events. We argue for an approach based on analysis of boundary conditions where measurement is closely motivated by theory.

If we want psychological science to have a meaningful real-world impact, it has to be trusted by the public. Scientific progress is noisy; accordingly, replications sometimes fail even for true findings. We need to communicate the acceptability of uncertainty to the public and our peers, to prevent psychology from being perceived as having nothing to say about reality.

The target article makes the important case for making replicability mainstream. Yet, their proposal targets a symptom, rather than the underlying cause of low replication rates. We argue that psychological scientists need to devise stronger theories that are more clearly falsifiable. Without strong, falsifiable theories in the original research, attempts to replicate the original research are nigh uninterpretable.

The debate about whether replication studies should become mainstream is essentially driven by disagreements about their costs and benefits and the best ways to allocate limited resources. Determining when replications are worthwhile requires quantifying their expected utility. We argue that a formalized framework for such evaluations can be useful for both individual decision-making and collective discussions about replication.

A scientific claim is a generalization based on a reported statistically significant effect. The reproducibility of that claim is its scientific meaning. Anything not explicitly mentioned in a scientific claim as a limitation of the claim's scope means that it implicitly generalizes over these unmentioned aspects. Hence, so-called “conceptual” replications that differ in these unmentioned aspects from the original study are legitimate, and necessary to test the generalization implied by the original study's claim.

It was a pleasure to read Zwaan et al.'s wise and balanced target article. Here, I use it as a shining example for bolstering the argument that to make innovations such as replication mainstream, it seems advisable to move the debates from social media to respected “mainstream” psychology journals. Only then will mainstream psychologists be reached and, we hope, convinced.

A pragmatist philosophy of psychological science offers to the direct replication debate concrete recommendations and novel benefits that are not discussed in Zwaan et al. This philosophy guides our work as field experimentalists interested in behavioral measurement. Furthermore, all psychologists can relate to its ultimate aim set out by William James: to study mental processes that provide explanations for why people behave as they do in the world.

No replication is truly direct, and I recommend moving away from the classification of replications as “direct” or “conceptual” to a framework in which we accept that treatment effects vary across conditions. Relatedly, we should stop labeling replications as successes or failures and instead use continuous measures to compare different studies, again using meta-analysis of raw data where possible.

Zwaan et al. argue convincingly that replication needs to be more mainstream. Here, I suggest three practices for achieving that goal: Incremental Replications, which are built into each experiment in a series of experiments; Reciprocal Replications, which are reciprocal arrangements of co-replications across labs; and Didactic Replications, which are replications used for training.

We suggest three additional improvements to replication practices. First, original research should include concrete checks on validity, encouraged by editorial standards. Second, the reasons for replicating a particular study should be more transparent and balance systematic positive reasons with selective negative ones. Third, methodological validity should also be factored into evaluating replications, with methodologically inconclusive replications not counted as non-replications.

Making replication studies widely conducted and published requires new incentives. Academic awards can provide such incentives by highlighting the best and most important replications. The Organization for Human Brain Mapping (OHBM) has led such efforts by recently introducing the OHBM Replication Award. Other communities can adopt this approach to promote replications and reduce career cost for researchers performing them.

Replication is the cornerstone of science – but when and why? Not all studies need replication, especially when resources are limited. We propose that a decision-making framework based on Bayesian philosophy of science provides a basis for choosing which studies to replicate.

Direct replication is valuable but should not be elevated over other worthwhile research practices, including conceptual replication and checking of statistical assumptions. As noted by Rotello et al. (2015), replicating studies without checking the statistical assumptions can lead to increased confidence in incorrect conclusions. Finally, successful replications should not be elevated over failed replications, given that both are informative.

Zwaan et al. and others discuss the importance of the inevitable differences between a replication experiment and the corresponding original experiment. But these discussions are not informed by a principled, quantitative framework for taking differences into account. Bayesian confirmation theory provides such a framework. It will not entirely solve the problem, but it will lead to new insights.

Although Zwaan et al. (2018) have made a compelling case as to why direct replications should occur more frequently than they do, they do not address how such replications attempts can best be encouraged. We propose a novel method for incentivising replication attempts and discuss some issues surrounding its implementation.

Zwaan et al. integrated previous articles to promote making replications mainstream. We wholeheartedly agree. We extend their discussion by highlighting several existing initiatives – the Replication Recipe and the Collaborative Education and Research Project (CREP) - which aim to make replications mainstream. We hope this exchange further stimulates making replications mainstream.

The presumed dominance of “original discovery” over replication is an anomaly. Original discovery has more value than replication primarily when scientific investigation can immediately generate numerous discoveries most of which are true and accurate. This scenario is uncommon. A model shows how original discovery claims typically have small or even negative value. Science becomes worthy mostly because of replication.

Zwaan et al. mention that young researchers should conduct replications as a small part of their portfolio. We extend this proposal and suggest that conducting and reporting replications should become an integral part of Ph.D. projects and be taken into account in their assessment. We discuss how this would help not only scientific advancement, but also Ph.D. candidates' careers.

Randomness in the selection process of to-be-replicated target papers is critical for replication success or failure. If target papers are chosen because of the ease of doing a replication, or because replicators doubt the reported findings, replications are likely to fail. To date, the selection of replication targets is biased.

Zwaan et al. make a compelling case for the necessity of direct replication in psychological science. I build on their arguments by underscoring the necessity of direct implication for two domains of clinical psychological science: the evaluation of psychotherapy outcome and the construct validity of psychological measures.