We are gratified by the large number of commentaries to our focal article (Dalal, Bonaccio, et al., 2010) that advocated greater integration of industrial–organizational psychology and organizational behavior (IOOB) with the field of judgment and decision making (JDM). The commentaries were uniformly constructive and civil. Our disagreements with the commentaries are mild and are limited primarily to the roles of external validity, internal validity, and laboratory experiments in IOOB. For the majority of our response, we attempt to build on the views expressed in the commentaries and to articulate some thoughts regarding the future. We structure our response according to the following themes: barriers to cross-fertilization between IOOB and JDM, areas of existing and potential JDM-to-IOOB cross-fertilization, areas of potential IOOB-to-JDM cross-fertilization, and ways to increase (and ideally institutionalize) cross-fertilization. We hope our focal article and our response to the commentaries will help to ignite exciting basic research and important practical applications associated with decision making in the workplace.

Johnson et al. (2010) note the need for a vector defining the relationships between job components and overall job performance (OJP) in the job requirements matrix approach to synthetic validation. This need is also implicit in the job components validity approach that they discuss. In our view, relationships between job components and overall criteria such as OJP are central to synthetic validation, as the effectiveness of this approach depends on the generalizability of relationships between job and criterion components across jobs, organizations, and (of increasing importance) countries.

In reading the title of Johnson et al.'s (2010) article on synthetic validity, I immediately asked myself “Better at what?” Motor oil serves three functions in internal combustion engines: lubricate, cool, and clean engine parts. There is little doubt that synthetic motor oil does all three of these better than nonsynthetic motor oil, although some might argue that it is not as cost effective. Johnson et al. described two approaches to synthetic validity and then argued why synthetic validity is “the best approach for many situations.” I strongly agree with their contention that synthetic validity is “practically useful” and with their less directly stated contention that it also holds value in developing theory. Hence, I will limit my comments exactly to how synthetic validity inferences might best contribute to the complimentary goals of advancing theory and practice, although these might cause Johnson et al. to rethink some of their observations.

For over three-quarters of a century researchers and practitioners have analyzed rating scale data using methods that assume a dominance response process wherein an individual high on the trait assessed is assumed to answer positively with high probability. This approach derives from Likert's famous 1932 approach to the development and analysis of rating scales. In this paper, we argue that Likert scaling and related methods are misguided. Instead, we propose that methods that have evolved from Thurstone (1927, 1928, 1929) scaling provide a better representation of the choice process underlying rating scale judgments. These methods hypothesize an ideal point response process where the probability of endorsement is assumed to be directly related to the proximity of the statement to the individual's standing on the assessed trait. We review some research showing the superiority of ideal point methods for personality assessment and then describe several settings in which ideal point methods should provide tangible improvements over traditional approaches to assessment.

Our focus is on the difficulties that synthetic validity encounters in attempting to achieve discriminant validity and the implications of these difficulties. Johnson et al. (2010) acknowledge the potential problems involved in attaining discriminant validity in synthetic validity. For example, they report that Peterson et al. (2001), Johnson (2007), and other synthetic validity studies report failure to achieve discriminant validity. What this failure means is that a synthetic validity equation developed to predict validity for Job A does as well in predicting validity for Jobs B, C, D, and so forth as it does for Job A. Johnson et al. then go on to propose that this problem might be overcome by careful attention to both the criterion and predictor sides of synthetic validity. We question whether their proposals can be made to work.

The focal article (Johnson et al., 2010) provides a highly upbeat assessment regarding the potential for job component validation (JCV) and J-coefficient methods to “substantially advance the science and practice of industrial and organizational (I-O) psychology through synthetic validity” (emphasis added). It follows on the heels of earlier, similarly enthusiastic endorsements (Jeanneret, 1992; Jeanneret & Strong, 2003; LaPolice, Carter, & Johnson, 2008). For example, LaPolice et al. claimed that the JCV Rs they obtained “are all very close to the maximum correlation for each dependent variable, suggesting that our models are approaching the best possible prediction” (p. 435, emphasis added).

Synthetic validity is very much like monogamy. It is an idea that is universally admired but not widely practiced. The concept of synthetic validity has been around for almost 60 years (Lawshe, 1952) and it is described in virtually every testing and measurement book and every industrial and organizational (I-O) psychology textbook I have ever read. I still vividly remember my undergraduate Psychological Testing professor dismissing synthetic validity as “made-up validity.” I thought he was wrong then, and I still do. Nevertheless, it is worth asking why synthetic validity studies are so rare.