Who decides?

Pfeffer (Reference Pfeffer1993) cautioned that fields with higher levels of consensus get that way via a core group of elite scholars who wield control. Imposing a framework to foster consensus requires some mechanism for decision making. For instance, what variables are included or receive more attention? When two groups of researchers have converged on the same topic, who gets the naming rights to the theoretical space? These issues are often sorted out via peer-review processes and citation of papers, which admittedly is not “efficient” but incorporates the judgments of many other researchers in the field, based on their assessment of the evidence. And, contrary to the authors' claim that no integrating frameworks exist, we point to a few recent examples in work on team process (i.e., Marks, Mathieu, & Zaccaro, Reference Marks, Mathieu and Zaccaro2001) and team structure (i.e., Hollenbeck, Beersma, & Schouten, Reference Hollenbeck, Beersma and Schouten2012) that have been built upon by others based on the evidence supporting them. In the approach proposed by Almaatouq et al., the dimensions are “mapped” onto the design space before the experiment is run, by a “cartographer” – but how does this occur?

The solution – as proposed – is for machine learning to make such decisions for us. While an elegant solution sidestepping the potential of an individual or group of individuals making the decisions, machine-learning algorithms by their very nature have bias baked into them (e.g., Fu, Aseri, Singh, & Srinivasan, Reference Fu, Aseri, Singh and Srinivasan2022). Furthermore, while machine learning undoubtedly has a substantial role to play in many areas of science, machine-learning models can only analyze the information provided, and are typically not able to identify variables that have not yet been considered but should be. While Almaatouq et al. would argue that such machines are flexible, and adaptive to change, we see this as overly optimistic. Indeed, across multiple disciplines and experiments, we can say with some certainty that a status quo – once set – is very difficult to change, as Kuhn's (Reference Kuhn1962) observations of the difficulty of challenging “dominant paradigms” demonstrates.

Another significant challenge stems from the strong incentives for researchers to introduce novel ideas in order to advance their careers. As Almaatouq et al. acknowledge, these incentives are at odds with efforts to promote convergence since there are few rewards for researchers who contribute to “normal science.” Though the authors attempt to brush this aside by pointing to examples from physics, it is important to note that fields requiring major infrastructure investment also tend to be more hierarchical, and also have significant struggles with other issues such as sexual harassment as well as gender gaps in participation and career length (Huang, Gates, Sinatra, & Barabási, Reference Huang, Gates, Sinatra and Barabási2020; National Academies of Science, Engineering, and Medicine, 2018). Thus the efficiency that can come from centralizing decision-making authority to accelerate convergence also risks introducing some of the known problems associated with consolidating power (Pfeffer, Reference Pfeffer1993).

Limiting diversity

The imposition of a framework for fostering convergence not only risks creating problematic power dynamics but also limiting the diversity of ideas in undesirable ways. Almaatouq et al. argue that their framework enables different studies to make their measures “commensurable.” They claim this can facilitate the integration of research using different methods, however, it most naturally lends itself to the use of the “high-throughput” techniques they mention, typically online experiments, to generate the volume of data needed for sampling the design space. This is likely to result in more uniformity in the methods and measures used. While some may see this as desirable, we point out that when different studies using different methods yield convergent patterns of results, the field can have greater confidence in those effects, as advocates of “full-cycle research” (e.g., Chatman & Flynn, Reference Chatman and Flynn2005) point out. Conversely, findings using the same measures and methods might make the results of different studies “commensurable,” but could mask limits to generalizability. Indeed, just as we have made great strides to sample beyond undergraduate students, we need to continue to push scientists to replicate and extend their work beyond that of online samples, as such samples are limited in their ability to capture rich behavioral outcomes. We also need to continue to broaden connections across contexts and disciplines to enable surprising new breakthroughs to emerge (Shi & Evans, Reference Shi and Evans2023).

The social and behavioral sciences are at an exciting nexus. Diversity is finally gaining traction: Historically underrepresented groups are bringing new theory and ideas to our historically homogenous field. Taken-for-granted knowledge is being falsified, or shown to only apply to the dominant groups. Exciting perspectives are just now being brought to fruition. To borrow from the author's terminology, the “unknown unknowns” are just starting to emerge due to the burgeoning diversity in the field. Will kicking off an intervention to force convergence, facilitated by machine learning, bake today's bias into algorithms that stymie the diversity that is just starting to take hold in our fields (Daft & Lewin, Reference Daft and Lewin1990)?