The trustworthiness of research findings has been questioned in many domains of science. This article calls for a review of the trustworthiness of the scientific literature in industrial–organizational (I–O) psychology and a reconsideration of common practices that may harm the credibility of our literature. We note that most hypotheses in I–O psychology journals are confirmed. Thus, we are either approaching omniscience or our journals are publishing an unrepresentative sample of completed research. We view the latter explanation as more likely. We review structural problems in the publication process and in the conduct of research that is likely to promote a distortion of scientific knowledge. We then offer recommendations to make the I–O literature more accurate and trustworthy.

In this article, we demonstrate that samples in the industrial and organizational (I-O) psychology literature do not reflect the labor market, overrepresenting core, salaried, managerial, professional, and executive employees while underrepresenting wage earners, low- and medium-skill first-line personnel, and contract workers. We describe how overrepresenting managers, professionals, and executives causes research about these other workers to be suspect. We describe several ways that this underrepresentation reduces the utility of the I-O literature and provide specific examples. We discuss why the I-O literature underrepresents these workers, how it contributes to the academic–practitioner gap, and what researchers can do to remedy the issue.

Although aging workforces result in numerous practical challenges for organizations and societies, little research has focused on successful aging at work. The limited existent research has generated rather diverse conceptualizations of successful aging at work, which are often broad and difficult to operationalize in practice. Therefore, to advance research and practice, we offer a specific and practical conceptualization of successful aging at work by developing a process model, which identifies relevant antecedents and mechanisms. In particular, we define successful aging at work as the proactive maintenance of, or adaptive recovery (after decline) to, high levels of ability and motivation to continue working among older workers. We also argue that proactive efforts to maintain, or adaptive efforts to recover and restore, high ability and motivation to continue working result from a self-regulation process that involves goal engagement and disengagement strategies to maintain, adjust, and restore person–environment fit. Further, we propose that at various levels (i.e., person, job, work group, organization, and society) more distal factors function as antecedents of this self-regulation process, with age-related bias and discrimination potentially operating at each level. Finally, we offer a roadmap for future research and practical applications.

Almost 20 years after McKinsey introduced the idea of a war for talent, technology is disrupting the talent identification industry. From smartphone profiling apps to workplace big data, the digital revolution has produced a wide range of new tools for making quick and cheap inferences about human potential and predicting future work performance. However, academic industrial–organizational (I-O) psychologists appear to be mostly spectators. Indeed, there is little scientific research on innovative assessment methods, leaving human resources (HR) practitioners with no credible evidence to evaluate the utility of such tools. To this end, this article provides an overview of new talent identification tools, using traditional workplace assessment methods as the organizing framework for classifying and evaluating new tools, which are largely technologically enhanced versions of traditional methods. We highlight some opportunities and challenges for I-O psychology practitioners interested in exploring and improving these innovations.

In this article, we highlight why and how industrial and organizational psychologists can take advantage of research on 21st century skills and their assessment. We present vital theoretical perspectives, a suitable framework for assessment, and exemplary instruments with a focus on advances in the assessment of human capital. Specifically, complex problem solving (CPS) and collaborative problem solving (ColPS) are two transversal skills (i.e., skills that span multiple domains) that are generally considered critical in the 21st century workplace. The assessment of these skills in education has linked fundamental research with practical applicability and has provided a useful template for workplace assessment. Both CPS and ColPS capture the interaction of individuals with problems that require the active acquisition and application of knowledge in individual or group settings. To ignite a discussion in industrial and organizational psychology, we discuss advances in the assessment of CPS and ColPS and propose ways to move beyond the current state of the art in assessing job-related skills.

We propose that generational differences are meaningful despite some theoretical and methodological challenges (cf. Costanza & Finkelstein, 2015). We will address five main issues: operationalizing generations, measuring generational differences, theoretical models of generations, mechanisms of generational change, and the importance of science versus stereotypes.

For over three-quarters of a century researchers and practitioners have analyzed rating scale data using methods that assume a dominance response process wherein an individual high on the trait assessed is assumed to answer positively with high probability. This approach derives from Likert's famous 1932 approach to the development and analysis of rating scales. In this paper, we argue that Likert scaling and related methods are misguided. Instead, we propose that methods that have evolved from Thurstone (1927, 1928, 1929) scaling provide a better representation of the choice process underlying rating scale judgments. These methods hypothesize an ideal point response process where the probability of endorsement is assumed to be directly related to the proximity of the statement to the individual's standing on the assessed trait. We review some research showing the superiority of ideal point methods for personality assessment and then describe several settings in which ideal point methods should provide tangible improvements over traditional approaches to assessment.

In their focal article, Landers and Behrend (2015) propose to reevaluate the legitimacy of using the so-called convenience samples (e.g., crowdsourcing, online panels, and student samples) as compared with traditional organizational samples in industrial–organizational (I-O) psychology research. They suggest that such sampling strategies should not be judged as inappropriate per se but that decisions to accept or reject such samples must be empirically or theoretically justified. I concur with Landers and Behrend's call for a more nuanced view on convenience samples. More precisely, I suggest that we should not “throw the baby out with the bathwater” but rather carefully and empirically examine the advantages and risks associated with using each sampling strategy before classifying it as suitable or not.

Situational judgment tests (SJTs) are typically conceptualized as contextualized selection procedures that capture candidate responses to a set of relevant job situations as a basis for prediction. SJTs share their sample-based and contextualized approach with work samples and assessment center exercises, although they differ from these other simulations by presenting the situations in a low-fidelity (e.g., written) format. In addition, SJTs do not require candidates to respond through actual behavior because they capture candidates’ situational judgment via a multiple-choice response format. Accordingly, SJTs have also been labeled low-fidelity simulations. This SJT paradigm has been very successful: In the last 2 decades, scientific interest in SJTs has grown, and they have made rapid inroads in practice as attractive, versatile, and valid selection procedures. Contrary to their popularity and the voluminous research on their criterion-related validity, however, there has been little attention to developing a theory of why SJTs work. Similarly, in SJT development, often little emphasis is placed on measuring clear and explicit constructs. Therefore, Landy (2007) referred to SJTs as “psychometric alchemy” (p. 418).

Assessment centers (ACs) are often designed with the intent of measuring a number of dimensions as they are assessed in various exercises, but after 25 years of research, it is now clear that AC ratings that are completed at the end of each exercise (commonly known as postexercise dimension ratings) substantially reflect the effects of the exercises in which they were completed and not the dimensions they were designed to reflect. This is the crux of the long-standing “construct validity problem” for AC ratings. I review the existing research on AC construct validity and conclude that (a) contrary to previous notions, AC candidate behavior is inherently cross-situationally (i.e., cross-exercise) specific, not cross-situationally consistent as was once thought, (b) assessors rather accurately assess candidate behavior, and (c) these facts should be recognized in the redesign of ACs toward task- or role-based ACs and away from traditional dimension-based ACs.

Landers and Behrend (2015) are the most recent in a long line of researchers who have suggested that online samples generated from sources such as Amazon's Mechanical Turk (MTurk) are as good as or potentially even better than the typical samples found in psychology studies. It is important that the authors caution that researchers and reviewers need to carefully reflect on the goals of research when evaluating the appropriateness of samples. However, although they argue that certain types of samples should not be dismissed out of hand, they note that there is only scant evidence demonstrating that online sources can provide usable data for organizational research and that there is a need for further research evaluating the validity of these new sources of data. Because the target article does not directly address the potential problems with such samples, we will review what is known about collecting online data (with a particular focus on MTurk) and illustrate some potential problems using data derived from such sources.

In this paper, I suggest that our methods for how to train are based historically on what we understand learning to be. I briefly trace the history of instructional design models, which embody both first-generation objectivist and second-generation cognitive constructivist views of learning. I then suggest we are on the cusp of a third-generation instructional model, one that places greater emphasis on the learner forming understanding largely through a process of social negotiation, either in training or on the job. I argue that Web-based instruction offers a technology ideally suited to the social constructivist approach at the core of this third-generation model. I discuss the importance of interaction in learning and suggest why Web-based networked learning may result in more, not less, interaction during training. Finally, I discuss other applications of third-generation learning such as peer mentoring and professional forums.

Most organizational justice studies focus primarily on theoretical issues and identify implications for practice only in passing. I advocate moving to the next step by testing such implications in theory-based studies that implement and assess the impact of interventions designed to promote organizational justice. Research that promotes organizational justice promises to benefit both organizations and their employees, and may be considered morally appropriate. Although usually not considered in this connection, theory-based application studies also promise to shed light on the theories from which they are derived. Despite these benefits, there are several reasons why such investigations are conducted only rarely. First, because managers tend to be unaware of justice-related problems, they are unlikely to accede to researchers’ requests to address them in research. Second, many researchers erroneously believe that studies assessing ways of improving conditions in organizations lack scientific objectivity. Third, scholarly values favor research that addresses theoretical issues and that eschew practical applications. Finally, the challenges of conducting intervention studies are inclined to be formidable, and these may deter researchers from undertaking such efforts.

Early in 2006, my colleagues and I published an article on unproctored Internet testing (UIT) in employment settings (Tippins et al., 2006). The purpose of that article was to identify the issues surrounding UIT and the ways in which those issues might be resolved. The panel of experts addressed a number of important questions about (a) the uses and applications of UIT and (b) the major issues and known problems associated with UIT, including test security, examinee identification, cheating, ethical use of tests, subgroup and cultural issues, standardization, and context effects. In addition, the panelists attempted to predict the future of UIT, highlight the research needed to facilitate UIT and provide advice to practitioners contemplating UIT. In the present article, “UIT” is used to refer to Internet-based testing completed by a candidate without a traditional human proctor. Thus, nontraditional forms of or alternatives to proctoring may be in place, such as quantitative analyses of response patterns, the use of video cameras, or follow-up testing with traditional proctoring.

In this article, we provide a brief review of the current and past research on leadership perceptions and performance. We then describe four principles that have emerged from many decades of research, which can provide new directions for future leadership theory and research.

Stereotypes about different groups persist in organizations. Employees from such groups may experience stereotype threat, or the concern that they are being judged on the basis of demeaning stereotypes about groups to which they belong. The goal of this focal article is to discuss whether stereotype threat is a useful construct for organizational psychology research and practice. To this end, we focus on consequences other than acute performance deficits in laboratory settings. In particular, we examine studies that highlight the effects of stereotype threat on intrapersonal outcomes (e.g., job attitudes), interpersonal outcomes (e.g., negotiation), and on the relationship between employees and their organization. The research reviewed suggests that stereotype threat is a potentially important phenomenon in organizations, but it also highlights the paucity of research in an organizational context. We provide suggestions for future research directions as well as for the prevention and amelioration of stereotype threat in the workplace.

We emphasize that science and practice issues are equally salient when pursuing thinking and research on employee engagement. We agree with much of what the commentaries have to say, especially that organizational competitive advantage is the relevant focus of engagement research and practice and that engagement is not a new construct but one that required clarification vis-a-vis existing constructs. We also agree that state engagement can be highly variable, that disengagement needs study, that negative situations can induce engagement behaviors, that engagement surveys should yield actionable data, and that people can be hired who are more likely to be engaged. We disagree with the idea that all employee attitudes are essentially equal and that existing conceptualizations of performance make engagement behavior a nonuseful construct.