In this essay, we review the Technology and Measurement around the Globe chapters with an eye toward integration and synthesis. We primarily focus on implications for testing, and then make connections to the broader world of nontest assessment. We identify themes of privacy, fairness, workplace applications, and emerging technologies, and offer a research agenda for future investigations that seek to understand culture, technology, and measurement.

An overview of testing and measurement in North America is provided, covering topics related to privacy laws and regulations, online proctoring, artificial intelligence, accommodations, accessibility, and the “opt out of testing” movement that are currently defining measurement in North America. This is not to say that these challenges are unique to North America; in fact, the challenges related to these topics are being faced all over the world in varying degrees and the same opportunities exist, but these topics are of particular importance when it comes to measurement and assessment in North America. Building on these observations, a discussion of how advances in technology and computing power provide an opportunity to challenge the status quo related to assessment; these advancements will allow assessment of skills in more authentic ways that will provide better insight into someone’s knowledge, skills, and abilities. The question we should be asking and attempting to answer is “How can assessment developers leverage the power of the cloud and technology to measure skills more accurately and create higher fidelity in the assessment process?”

Chapter 10 opens by asking readers to choose two kinds of people they might encounter in informal learning settings and to identify questions those people might have about their general topic and about their specific activity. Returning to the fact that a successful conversation is cooperative, this chapter emphasizes asking questions and listening. Asking questions of an audience gives the expert substantive information to listen to. Sets of questions give people choice, and the sets can include questions about explanations of the phenomena being shown. A "juicy question," for example, is one that nonexperts can address by using the materials/examples at hand – in effect, encouraging scientific reasoning. Giving people time to answer questions and then listening carefully as they do so shows respect, as does asking new questions that reflect people’s earlier responses. Readers are cautioned to avoid testing their audience or to feel that they themselves are being tested. The Worked Example finds juicy questions in a map-based demonstration of variation in regional dialects.

We begin by demonstrating the importance of the ideas of Karl Popper to corpus linguistics. On that basis we begin an exploration of his ideas in order to understand both his account of the scientific method and how this method may illuminate aspects of the corpus-based approach to the study of language.

The bulk of medical care system management in the United States does not ground decisions in use of available, high quality evidence. The previous chapters in this book demonstrate both that there sometimes is a rich scientific foundation in health care organization, delivery, and financing that could, if applied, lead to better outcomes, and that sometimes there is little or no evidence on effectiveness of interventions. However, we also observed that the bulk of health care management in the United States does not ground decisions in evidence, using it if available and taking uncertainty into account if not – instead “magical thinking” is often used to make choices.1 Ironically, management holds evidence in high esteem for decision-making by clinicians. In this chapter, we explore why management holds itself to a lower standard regarding its organizational, staffing, and planning choices, seeing experience, intuition, and opinions as good-enough evidence for decisions. We explore what needs to happen for this to change.

Testing vocabulary is similar to testing in other areas of language knowledge and use. The same criteria of reliability, validity, practicality, and washback need to be considered when designing and evaluating vocabulary tests. In some ways testing vocabulary is easier than testing grammatical knowledge or control of discourse because the units to test are more obviously separate. It is not too difficult to identify what a word type is. However, there are problems and issues and we look at these in this chapter. This chapter has two major divisions. The first major division looks at the purposes of vocabulary tests, covering diagnostic, placement, achievement, and proficiency tests. The second major division looks at different test formats, answering questions like: Should choices be given? Should words be tested in context? How can I measure words that learners don’t know well? This section covers a wide range of vocabulary test formats, along with comments on their design and use.

Managerial economics provides a toolbox for solving problems that managers frequently face. It addresses issues relating to any aspect of decision making that ultimately affects the profit of a firm. Although the general methodology of managerial economics has not changed over the decades, there have been rapid and significant changes in the business environment in the last ten years or so, and three new themes have become increasingly important: digitization; behavioural aspects; and globalization. The first of these developments involves aspects of big data and advanced data analytics, the human-machine interface and the interconnectedness of electronic devices. The second relates to psychological aspects of decision making that cause both consumers and managers to engage in behaviour normally referred to as ‘irrational’. The third development is that improvements in technology relating to digitization have made the business world more interconnected. The text makes heavy use of recent case studies involving these three themes, for example on tech firms, Covid-19 and climate change, so students can see how the tools of managerial economics can be applied in real-life situations.

Chapter 8 looks at an Article 36 review of autonomous weapons systems. Having first addressed the progress of debates about potential future regulation of such systems at the international level, it delineates the notions of autonomy, artificial intelligence and learning systems. The chapter shows that military capabilities underpinned by machine learning and deep learning technologies require a new understanding of how provisions of targeting law can be meaningfully translated to the context of autonomous systems and examined in the framework of Article 36. It also demonstrates that determining the novel character of learning autonomous capabilities to ensure timely provision of legal advice, gaining assurance of complex adaptive systems’ performance accuracy and reliability, or identifying a set of requisite control measures, are all issues that ‘traditional’ weapons reviews do not have to grapple with at all, or at least not to the same extent. [143 words]

Chapter 9 focuses on an Article 36 review of (autonomous) cyber capabilities. As in the case of autonomous weapons systems, the most burning questions for a weapons review in the cyber context remain: How to meaningfully incorporate advice on the law of targeting as part of the weapons review? How to determine when a given capability is ‘new’ for review purposes and when to initiate a review? How and when can testing and evaluation processes meaningfully inform the review outcome? Furthermore, some challenges distinct to cyber capabilities also exist. Most importantly, cyber capabilities test the underlying assumptions of the law of armed conflict. The effects of their use may be more deleterious than the consequences produced by traditional weaponry, and yet they may fall outside the legal review requirement because the effects produced do not constitute an ‘attack’ in its conventional interpretation. [142 words]

This chapter gives an overview of the main findings regarding the multimedia principle in testing, which states that testing material that combines text and adequate pictures is more comprehensible and more efficient to process for test takers than text alone. Even though learning and testing with multimedia have much in common, substantial differences exist that are discussed in this chapter. Overall, the findings indicate that adding pictures to verbal test material is mainly associated with facilitative effects. However, the findings differ with respect to different outcome measures and depending on the investigated picture type (i.e., decorative, representational, organizational, and informational). Furthermore, effects also appear to be moderated by item and test-taker characteristics that need to be subjected to future research.

This chapter provides an overview of pedagogical heritage language (HL) assessment issues and an agenda for enhancing positive washback effects of assessment on HL learners in the future. After an introduction, the chapter first discusses the definition of HL learners used here and elsewhere. Second, it explores two distinct types of tests that are important and relevant to the subsequent discussions of HL assessment issues. The discussions reveal an apparent lack of validation studies for proficiency assessment tools in general and HL assessment in particular. For example, postsecondary placement investigations dealing with HL learners largely focus on Spanish but little is known for placement practices in other HLs. Also, the availability of assessment tools for pre-collegiate HL programs is very limited, especially for less commonly taught languages. Validation studies of existing assessment procedures for pre-collegiate HL programs are scarce. This chapter also points out that bilinguals’ dynamic and holistic use of linguistic repertoires adds complexity to assessing bilinguals’ HL proficiency. This chapter mainly discusses HL assessment in pedagogical contexts, but where relevant, it examines the use of HL tests for research as well.

Since July 1945, more than 2,000 nuclear explosive devices have been detonated in tests in more than 60 different locations. The total yield of nuclear tests conducted from 1945 to 1995 is equivalent to 34,000 Hiroshima-size bombs. Only North Korea has test-detonated nuclear weapons this century. While the 1963 Partial Test-Ban Treaty remains in force the 1996 Comprehensive Nuclear-Test-Ban Treaty has not entered into force. The chapter explains why and what customary rules restrain nuclear testing today.

This chapter provides a critical examination of Arabic study-abroad numbers and outcomes, of recurrent problems, and of research-based interventions. Trentman’s study highlights both qualitative and quantitative assessments of students’ skills before and after the study-abroad experience, especially in oral proficiency. Arabic sociolinguistic variation has long played a destabilizing role in proficiency gain, measurement, and accuracy, and the challenge of developing networks of informal, relaxed relationships with native speakers is raised and addressed. Also addressed are the less-noted ‘ideologies’ of study abroad that focus either on a ‘touristy’ intercultural experience or on an instrumental ‘orientalist perspective’ that discourages student engagement with local communities. Research interventions such as reflective practice and ethnography projects are discussed and evaluated as ways to raise the awareness and progress of study-abroad students.