Chapter 5 gives an extended empirical example of the Benford agreement procedure for assessing the validity of social science data. The example uses country-level data collected and estimated by the Sea Around Us organization on the dollar values of reported and unreported fish landings from 2010 to 2016. We report Benford agreement analyses for the Sea Around Us data (1) by reporting status, (2) by decade, (3) for a large fishing region of 22 West African countries, and (4) foreach of the 22 individual countries in West Africa.

Chapter 4 begins with a discussion of the types and kinds of data most suitable for an analysis that uses the Benford probability distribution. Next we describe an R computer program – program Benford – designed to evaluate observed data for agreement with the Benford probability distribution; and we give an example of output from the program using a typical dataset. We then move to an overview of our workflow of Benford agreement analyses where we outline our process for assessing the validity of data using Benford agreement analyses. We end the chapter with a discussion of the concept of Benford validity, which we will employ in subsequent chapters.

Chapter 7 takes a closer look at some of the Sea Around Us fish-landings data that we assessed for Benford agreement in Chapter 5. We chose these data because of the mixed agreement findings among them: while the full dataset and several sets of subgroups indicated that the data exhibited Benford validity, when we analyzed West African countries individually, a number of them were found to have unacceptable Benford agreement and therefore problematic Benford validity. We present ways in which researchers can assess the impact of unacceptable Benford agreement on their analyses.

Chapter 3 describes and illustrates the Benford probability distribution. A brief summary of the origin and evolution of the Benford distribution is drawn and the development and assessment of various measures of goodness of fit between an empirical distribution and the Benford distribution are described and illustrated. These masures are Pearson’s chi-squared, Wilks’ likelihood-ratio, Hardy and Ramanujan’s partition theory, Fisher’s exact test, Kuiper’s measure, Tam Cho and Gaines’ d measure, Cohen’s w measure, and Nigrini’s MAD measure.

Chapter 6 provides a second empirical example of the Benford agreement procedure: here we analyze new daily COVID-19 cases at the US state level and at the global level across nations. Both the state-level and the global analyses consider time as a variable. Specifically we examine, (1) for the United States, new reports of COVID-19 between January 22, 2020 and November 16, 2021 at the state level, and (2) for the cross-national data, new reports of COVID-19 between February 24, 2020 and January 13, 2022. At the state level, we report Benford agreement analyses for (1) the full dataset, (2) cases grouped alphabetically, (3) cases grouped regionally, (4) cases grouped by days of the week, and (5) cases grouped by their governor’s party (Republican or Democratic). We then turn our Benford agreement analysis to global cross-national COVID-19 data to assess whether Benford agreement of COVID-19 varies across countries.

This chapter gives an overview of the remainder of the book. We first provide commonsense and social science examples of reliability and validity, two necessary conditions that data must posses to have trustworthy conclusions based upon it. We next introduce Benford’s law and offer a brief overview of other social science studies that have employed it to check the accuracy of their data. We then turn to an overview of our Benford agreement analysis procedure and introduce the concept of Benford validity. The chapter concludes with a plan for the remainder of the book.

Here we develop a discussion of the concept of validity in the social sciences. We first highlight the history of validity and how it has been conceptualized and measured over time. Next we discuss a type of social science data that is often overlooked in the validity measurement and assessment literature: data that are based on self-reporting. Despite the widespread use of self-reported data in various social science disciplines such as economics, political science, and sociology, there are still few reported attempts to check data accuracy. By way of giving examples, we overview self-reported data in four areas: (1) US prison population data, (2) COVID-19 case data, (3) toxic releases, and (4) fish landings. We then discuss the need for a tool and for an established workflow for assessing the accuracy and validity of quantitative self-reported data in the social sciences. We suggest that applying Benford’s law to these types of data can provide a measure of validity assessment for data that would otherwise not be assessed for accuracy; then we briefly introduce the concept of Benford validity. We conclude the chapter with a short review of existing studies that have applied Benford’s law to social science data in some manner.

Chapter 8 concludes that Benford agreement analyses are a proper process for assessing the validity of self-reported data when these data meet certain characteristics. The Benford agreement analysis workflow developed in previous chapters is summarized. Recommendations as to when researchers should use Benford agreement analyses to assess their data for Benford validity are discussed. The chapter concludes with some thoughts on the utility of Benford validity analyses in the social sciences.

Introduction

At the start of a seminar he convened at UCLA in 1992, Harold Garfinkel warned a visitor, “you must do what I call perspicuous settings, or tutorial problems, or when I’m in a bad mood I call them exercises. I never call them experiments.” He did call them experiments decades earlier, though he qualified this by saying that they were not “properly speaking experimental,” and that they were “demonstrations” that (in a phrase he attributed to phenomenologist Herbert Spiegelberg) provided “aids to a sluggish imagination” (Garfinkel 1967, 38). Regardless of what names he chose to give them, the various “experiments” he devised over many years demonstrated anything but a sluggish imagination, and they offered varied tasks, difficulties, and lessons for his students. Most, but not all, of the exercises used “trouble” as leverage for gaining insight into the routine organization of social activities. Some troubles were deliberately induced disruptions, such as in the simple case of playing the game of tic-tac-toe (noughts and crosses) when the “experimenter” would make a move by inscribing a mark straddling a line separating adjacent squares in the 3 × 3 matrix. This maneuver regularly drew the objection from the opponent that it was “against the rules,” but the rule in question was invoked consequent to the move, rather than being an explicitly stated precondition of play. Other exercises required students to perform activities while remaining alert to troubles that regularly occur—such as getting lost in the course of following directions.

Drawing from Garfinkel's published writings, as well as transcripts of lectures and seminars in the Garfinkel Archive, I begin with a discussion of some of his early experiments, comparing them with contemporaneous social psychology experiments and discussing how they were commonly presented in sociology textbooks as methods for exposing background knowledge and tacit presuppositions that operate beneath the awareness of social actors. After suggesting that some of the early experiments and Garfinkel's commentaries on them exhibited more problematic implications for sociology, this chapter goes on to examine a raft of exercises and demonstrations that Garfinkel deployed in a later phase of his teaching and research.