As industrial–organizational (I-O) psychologists and longtime employees, we have developed and implemented appraisal systems and have been subjected to and have subjected others to appraisals. We have thus viewed performance appraisals from all angles, seeing the good, the bad, and the downright ugly. We believe that all of the points discussed by Adler et al. (2016) about retaining or eliminating performance ratings have merit and address the realities of the current state of affairs in performance appraisal practice and research. However, as Wiese and Buckley (1998) point out, organizations survived quite well for centuries without formal appraisal systems, which raises the question, “Why do formal performance appraisal systems exist?” One inescapable yet surprisingly undiscussed reason is that it is a legal and/or regulatory mandate for 4,185,000 U.S. federal government employees (Office of Personnel Management, 2015a). Eliminating performance ratings for these workers would literally require an act of Congress.

To enhance interactions between police officers and citizens, industrial–organizational (I-O) psychologists will have to utilize a systems thinking approach to understand the multifaceted challenges facing Baltimore and other cities across the nation and develop holistic solutions that include the whole community. Ruggs et al. (2016) overlooked the systems view of Baltimore's challenges by focusing solely on isolated incidents of racial bias and proposing solutions predominantly inside the police station. To develop a comprehensive solution that has the potential to truly extinguish Baltimore's flames, we must also account for the interplay of police bias where it exists in the community; otherwise, we risk simply fanning the flames and escalating the challenges further with linear solutions.

In their focal article, Pratt and Bonaccio (2016) describe the potential value that qualitative research can bring to the field of industrial–organizational psychology (I-O) and also highlight several challenges (and myths) that must be overcome for this value to be fully realized. We agree with these authors, particularly with regard to the barriers that appear to stand in the way of fully integrating qualitative approaches with the science and practice of our field. Our purpose in this commentary is to build on the ideas of Pratt and Bonaccio by expanding the discussion of barriers against qualitative research. It is our view that further highlighting such barriers will illuminate several paths forward toward the increased adoption of qualitative methods, ideas, and approaches. More specifically, we focus on the following three barriers: (a) categorical thinking, (b) the uncertainty of that which is unknown, and (c) an overemphasis on generalizability in psychological research. We discuss each of these in turn below.

What is the role of the situation in situational judgment tests (SJTs)? Lievens and Motowidlo (2016) assert that SJTs are somewhat of a misnomer because they do not actually measure how individuals would behave in a given situation per se. According to these researchers, SJTs assess general domain knowledge—whether potential employees recognize the “utility of expressing certain traits” (p. 4). As a result, SJTs map onto personality measures, which are a summary of behavior across time and situations. SJTs provide predictive validity in part because they tap into personality. However, rather than renaming SJTs, it is possible to reintroduce the concept of a situation to provide even greater predictive power. Thus, the goals of this commentary are to (a) clarify what constitutes a situation, (b) describe what SJTs might actually measure, and (c) set forth a path for a taxonomy of workplace situations.

Dissatisfaction with performance appraisal is at an all-time high (Adler et al., 2016). In this commentary we argue that one of the reasons why performance appraisal is unable to get the most out of employees is the way in which employees are evaluated against a uniform set of criteria, leading to a focus on deficits and little attention for unique individual qualities and strengths. By comparing the performance of an employee with a set of predetermined criteria, and by expecting the employee to perform well across all these criteria, the performance appraisal tends to focus on those areas where employees perform below the norm, irrespective of how excellently they may perform in other areas. For many employees, this leads to the frustrating experience that there is more attention for their weaknesses than for the areas in which they excel. By focusing on employee strengths and on how to make optimal use of those strengths, and by allowing for diversity in the way that employees execute their jobs, the performance review can be replaced by a dialogue about development that may truly stimulate the motivation for performance improvement.

Policing in United States has been under media and public scrutiny for a while now. The focal article “Baltimore Is Burning: Can I-O Psychologists Help Extinguish the Flames?” (Ruggs et al., 2016) is a critical piece of work that not only sheds light on how racial bias may impact police–citizen interactions but also outlines possible ways in which industrial–organizational (I-O) psychologists can intervene to remedy the situation. Although the call to use evidence-based I-O interventions as levers to facilitate a more effective and unbiased police force is timely, it is critical to acknowledge that the charged environment and the controversial nature of this issue imply that some members of the police force might not accept the suggested interventions at face value. Many police officers are upstanding and hard-working individuals and may see the change interventions as an attack. Thus, before attempting to execute any strategies offered in the focal article, I-O psychologists will have to do significant groundwork from a change management perspective: The current crisis needs to be approached from a whole systems change perspective so that we can (a) understand the factors causing the “flames” and define the problem space comprehensively; (b) understand the type of change we need to effect in response to the crisis, stay closely attuned to the system's readiness for change, and also anticipate resistance to change from different levels of the system; and (c) chart a clear outline of what our role should entail as change agents in helping resolve this crisis. Thus, this commentary complements the focal article by juxtaposing authors’ recommendations against a whole systems change paradigm, questions their implicit assumptions, and recommends that I-O psychologists act as change agents to help police forces define the problem, find solutions, anticipate resistance, and execute interventions at different levels of the policing system.

As someone trained exclusively as a quantitative researcher, who recently became a semi-autodidactic qualitative researcher (see Rothausen, Henderson, Arnold, & Malshe, in press; “semi” in part because I am still learning and in part because my coauthors have taught me), I would like to extend the argument made by Pratt and Bonaccio (2016) for increasing qualitative research in the domains of industrial–organizational psychology (IOP), organizational behavior (OB), and human resources (HR), and I would also add industrial relations (IR), which was my doctoral field of study and “where workers went” within business and management studies as HR became more aligned with organizational interests (see Lefkowitz, 2016, from this journal). I extend their argument by deepening one of their reasons, understanding the “why” of work, and adding another potential use, understanding the “what could be” of work.

Although Lievens and Motowidlo (2016) made a strong case for reconceptualizing situational judgment tests (SJTs) as measures of general domain knowledge, we disagree with their view that the judgment or assessment of the situation itself is not important. We contend that situation assessment is an integral yet ignored factor in SJTs and that both general domain knowledge and situation assessment are needed to better understand how SJTs work.

At the heart of the debate between Colquitt's and Adler's (Adler et al., 2016) camps is a disagreement about the degree to which employees can be expected to respond favorably to challenging, negative, or critical feedback. Colquitt and colleagues argue that we often try and avoid blame, select jobs that don't rate us against others, and respond unhappily to accurate appraisals. Adler and his collaborators, by contrast, are more optimistic. They point to how feedback drives us to seek new strategies, change our behavior, and improve our skills.

Pratt and Bonaccio's (2016) focal article properly reviews and identifies the need for qualitative research methods in our field. However, they overlooked one important benefit—team science—that is crucial to current organizations. Despite the fact that qualitative research in team science is lacking, we suggest that with qualitative research we can gain more insight into what teams need in order to be effective. According to Kozlowski and Bell (2003), team dynamics are historically looked at in a static way in teams research, solely focusing on individuals’ perceptions of the team at a given time as opposed to multilevels over time. In an attempt to further expand on how qualitative research can examine constructs that purely quantitative methods may not, the purpose of this commentary is to highlight importance of qualitative research regarding its ability to capture team dynamics as they occur in the real world. The need for qualitative methods exists across various components (i.e., inputs, team emergent states, processes, outputs) when it comes to teams. We argue that how these components appear, happen, and, more importantly, evolve over time should be taken into consideration. The current commentary highlights how qualitative research can start to fill the gap of understanding team dynamics and how to improve team practices by taking time into consideration.

The construct validity of situational judgment tests (SJTs) is a “hot mess.” The suggestions of Lievens and Motowidlo (2016) concerning a strategy to make the constructs assessed by an SJT more “clear and explicit” (p. 5) are worthy of serious consideration. In this commentary, we highlight two challenges that will likely need to be addressed before one can develop SJTs with clear and explicit constructs. We also offer critiques of four positions presented by Lievens and Motowidlo that are not well supported by evidence.

The authors of the focal article ask a very important question: “Baltimore Is Burning: Can I-O Psychologists Help Extinguish the Flames?” (Ruggs et al., 2016). The answer is yes, emphatically, yes. The applied science of industrial–organizational (I-O) psychology knows a great deal that can be brought to bear to help solve this real-world problem. As the title of this commentary indicates, personnel selection is one area that is highly relevant to this issue. Personnel selection is one of I-O psychology's specialty areas and is thought of by many around the world, including those in the U.S. legal system, as “owned” by I-O psychology. Other I-O specialty areas such as recruitment, training and development, onboarding/socialization, performance management, leadership, culture/climate, and culture/climate change also clearly have much to offer to help solve the problem. Others can address relevant research and practice in those areas; this commentary, however, is devoted to personnel selection. Revising a police department's hiring strategy is not a quick fix, nor is it, by itself, sufficient. Nonetheless, it is an important part of the package that I-O psychology has to offer.

Lievens and Motowidlo (2016) argue compellingly that situational judgment tests (SJTs) measure job-relevant general domain knowledge, conceptualized as implicit trait policies (ITPs). ITPs are defined as a person's knowledge about the utility of expressing certain traits. They develop through the feedback a person receives when acting in accordance with their trait profiles in different environments (work, life, leisure). Positive feedback reinforces the knowledge that behavior in accordance with one's own traits is appropriate, and negative feedback reinforces the knowledge that an approach that differs from one's trait tendencies may be more effective. As such, ITPs represent a person's knowledge about the effectiveness of behaviors across a variety of contexts.

Research and practice in performance appraisal and performance management seem to suffer from the same “delusion of absolute performance” that Rosenzweig (2007, p. 112) described with respect to commentators’ evaluations of company performance in a competitive market economy. Commentators on business success factors have tended to speciously neglect or downplay the relative nature of performance (Rosenzweig, 2007). Downplaying the relative nature of performance is apparently the strategy endorsed by most performance appraisal scholars, too. Goffin, Jelley, Powell, and Johnston (2009) estimated that less than 4% of the published performance rating research has involved relative or social-comparative approaches, despite demonstrable advantages for relative over absolute rating formats (discussed below). Similarly, social comparison research and organizational scholarship have not traditionally been closely integrated (Buunk & Gibbons, 2007; Greenberg, Ashton-James, & Ashkanasy, 2007).

Pratt and Bonaccio's (2016) article is oriented to the position of qualitative research in U.S. industrial–organizational (I-O) psychology, although brief reference is made to innovations in the UK psychology field. As European work and organizational (W/O) psychologists who have championed the use of qualitative research in our field for the last 25 years, we share Pratt and Bonaccio's concerns about the lack of qualitative research in what are described as the “top” I-O psychology journals, and we agree that this situation is detrimental to the development of the discipline in many ways (see Cassell & Symon, 2006, for further discussion). Here we want to present a European perspective on this issue, which sheds some light on why qualitative research may be more accepted in European W/O psychology but also highlights the power relations that tend, even in Europe, to maintain it in a rather second rate position. Our intention is to engage in a process of mutual learning across the American and European situations.

We applaud the authors for tackling the important issue of policing and race from the unique perspective of industrial–organizational (I-O) psychology. Here, we propose a framework by which to examine the authors’ recommended interventions along the same implicit–explicit dimension employed in the focal article's conceptualization of racial bias. Mirroring current thinking within the diversity literature, the focal article notes that racial bias is “often very subtle” (Ruggs et al., 2016, p. 531) and can include “unconscious and implicit” aspects (p. 531). Extending this notion of implicit versus explicit bias to interventions themselves, we advocate for increased attention toward more implicitly focused interventions, as opposed to some of the more explicitly focused interventions suggested in the focal article. We conceptualize explicitly focused interventions as those that deal directly and openly with race, diversity, or demographic differences. Below, we discuss three potential advantages of implicitly focused interventions.

We were honored and flattered to be recognized in the feature article for our commitment to qualitative methods training. As an interdisciplinary program focused on organizational science, we strive not to privilege one form of training, thought, or inquiry over another. We recognize that a number of the problems and questions faced by organizational scholars and practitioners cannot adequately be addressed solely by a single discipline or method. Instead, we emphasize the synergy between different methods and modes of thought. Ultimately, our philosophy is that organizational phenomena are inherently interdisciplinary, thus training should reflect that. Our students are trained in industrial–organizational (I-O) psychology, management, organizational sociology, and organizational communication. We use this commentary as an opportunity to explain why our program values qualitative methods equally with quantitative methods, describe how we integrate that training, highlight a few success stories resulting from qualitative projects in our program, and then share some advice to other programs considering additional qualitative training.

The situational judgment test (SJT) development procedures outlined by the authors of the focal piece (Lievens & Motowidlo, 2016) provide an excellent framework to design SJTs that help answer fundamental questions about what SJTs measure and why they work. This article expands on this framework to explore further some of the issues faced in the development of SJTs. These issues include the implied assumption of linearity between general domain knowledge and effectiveness, whether the SJT measures a single construct or multiple constructs, and when a more criterion-centered approach to SJT development might be preferred.

The company I work for is one of the leading providers of performance management technology (Jones & Wang-Audia, 2013). This technology is used by more than 3,000 organizations worldwide, including several of the companies mentioned in Adler et al. (2016). The technology is highly configurable. It is currently being used to support performance management processes with no annual manager ratings, processes with traditional annual rating evaluations, processes that only evaluate competencies, processes that only evaluate goal accomplishment, processes that mix goals and competencies, processes that require forced-ranked comparisons between employees, processes that make no direct comparisons between employees, and much more. The capabilities of this and other human resources (HR) technology systems are allowing companies to radically rethink performance management because they enable companies to do things far differently from what was possible when they were constrained to more fixed electronic or paper forms (Hunt, 2011, 2015a). The result is an explosion in the diversity of approaches being taken toward performance management design.

Ruggs et al. (2016) describe paths through which industrial–organizational (I-O) psychology can make a dent in the ongoing policing problems in the United States. These paths include traditional I-O areas such as improved selection models, increased training, and changed organizational climates. However, there might be one fairly straightforward way in which police organizations can quickly reduce use-of-force problems: women. Because Title VII of the Civil Rights Act prevents selection based on sex, police departments obviously cannot hire women just because they are women. But police departments can and, we argue, should recruit more women to apply for police officer positions, create work practices and experiences that are attractive to and supportive of women (Hassell & Brandl, 2009), and make efforts to retain female officers because of the evidence that female officers use less force when policing (Bolger, 2015). Additionally, police organizations and I-O psychologists should also work together to discover why women are less likely to use force and, subsequently, determine whether these characteristics can be selected or trained for in either sex.