Integrative experiment design promises to foster cumulative knowledge by changing how we design experiments, build theories, and conduct research. I support the push to increase commensurability across experimental research but raise several reservations regarding results-driven and large-team-based research. I argue that it is vital to preserve academic diversity and adversarial debate via independent efforts.

Integrative experiment design assumes that we can effectively design a space of factors that cause contextual variation. However, this is impossible to do so in a sufficiently objective way, resulting inevitably in observations laden with surrogate models. Consequently, integrative experiment design may even deepen the problem of incommensurability. In comparison, one-at-a-time approaches make much more tentative assumptions about the factors excluded from experiment design, hence still seem better suited to deal with incommensurability.

I provide a personal perspective on metastudies and emphasize lesser-known benefits. I stress the need for integrative theories to establish commensurability between experiments. I argue that mathematical social scientists should be engaged to develop integrative theories, and that likelihood functions provide a common mathematical framework across experiments. The development of quantitative theories promotes commensurability engineering on a larger scale.

This commentary analyzes the extent to which the incommensurability problem can be resolved through the proposed alternative method of integrative experiment design. We suggest that, although one aspect of incommensurability is successfully addressed (dimensional incommensurability), the proposed design space method does not yet alleviate another major source of discontinuity, which we call conceptual incommensurability.

Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.

When a measure becomes a target, it ceases to be a good measure. For example, when standardized test scores in education become targets, teachers may start “teaching to the test,” leading to breakdown of the relationship between the measure – test performance – and the underlying goal – quality education. Similar phenomena have been named and described across a broad range of contexts, such as economics, academia, machine learning, and ecology. Yet it remains unclear whether these phenomena bear only superficial similarities, or if they derive from some fundamental unifying mechanism. Here, we propose such a unifying mechanism, which we label proxy failure. We first review illustrative examples and their labels, such as the “cobra effect,” “Goodhart's law,” and “Campbell's law.” Second, we identify central prerequisites and constraints of proxy failure, noting that it is often only a partial failure or divergence. We argue that whenever incentivization or selection is based on an imperfect proxy measure of the underlying goal, a pressure arises that tends to make the proxy a worse approximation of the goal. Third, we develop this perspective for three concrete contexts, namely neuroscience, economics, and ecology, highlighting similarities and differences. Fourth, we outline consequences of proxy failure, suggesting it is key to understanding the structure and evolution of goal-oriented systems. Our account draws on a broad range of disciplines, but we can only scratch the surface within each. We thus hope the present account elicits a collaborative enterprise, entailing both critical discussion as well as extensions in contexts we have missed.

Natural selection is slow, so behavioral goals must be based on patterns of reward. Addictions are rewarded in the same way as adaptive choice, so they can be distinguished only by their time course. In addition, the reward process is more plastic than is generally recognized, so abstract goals are shaped by the “legibility” of their proxies.

The analysis of proxy failure given by John et al. provides a good starting point for interdisciplinary discussions. Here, the discussion of teleonomy is extended and updated to include more recent discourse on the topic.

In their target article, John et al. make a convincing case that there is a unified phenomenon behind the common finding that measures become worse targets over time. Here, we will apply their framework to the domain of animal welfare science and present a pragmatic solution to reduce its impact that might also be applicable in other domains.

Decision-making heuristics rely on proxies so the elements of heuristics appear to map well to the elements of proxies identified by John et al. However, unlike proxy failure, heuristics do not fail because of feedback. This may be because for successful heuristics the goals of regulators and agents are aligned, but this is not the case for proxy failure.

Proxies should not be classified as failures or successes because, in most cases, they are impossible translations of abstract, polysemous goals to supposedly univocal concrete measures. The “success” or “failure” of a proxy does not depend on its actual accuracy as a valid indicator of goal attainment, but on a social system's willingness to maintain an illusion of its accomplishment.

A societal shift has occurred toward making impactful decisions on the basis of objective metrics rather than subjective impressions. This shift is commonly justified by claims that we should not trust subjective intuitions. These are often unjust and thereby corrupt. However, the proxies used to make objective decisions are subject to a different form of corruption, characterized as proxy failure.

Identifying the conditions of proxy treadmilling is crucial for determining whether reliable signals can persist over time. I present a framework that maps evolutionary models of reliable signals according to their assumptions regarding the effects of Goodhart's law. This framework can explain the contrasting outcomes of different modelling approaches, and identify in which models proxy treadmilling is expected to occur.

We argue that proxy failure contributes to poor measurement practices in psychological science and that a tradeoff exists between the legibility and fidelity of proxies whereby increasing legibility can result in decreased fidelity.

Following John et al., we provide examples of failing proxies that might help to contextualize the role of proxy failures in applied research. We focus on examples from the sociology of science and illustrate how the notion of proxy failure can sharpen applied analysis, if used in a way that does not obscure other dysfunctional effects of proxies.

We argue that a diverse and dynamic pool of agents mitigates proxy failure. Proxy modularity plays a key role in the ongoing production of diversity. We review examples from a range of scales.

I consider the dynamics of regulator–agent interactions in situations in which there are significant mismatches in their abilities to discern and register information and calculate and act upon successful inferences.

Two of John et al.'s examples of proxy failures in ecological situations are not failures: Runaway sexual selection and marsupial neonate competition. Instead, more appropriate ecological examples may be paternal genetic kin recognition and warning coloration. These differ in proxy effectiveness and failure in ways that illustrate the importance of “costs” in the evolution of ecological proxy traits.

The analysis of John et al. is lacking a fully general account of proxy failure. It is here proposed that proxy failure can be understood as a feature of all adaptive control systems. Whether proxies “fail” or “succeed” depends on the more encompassing view one can adopt for observing such systems.