Scholars and policy makers need systematic assessments of the validity of the measures produced by V-Dem. In Chapter 6, we present our approach to comparative data validation – the set of steps we take to evaluate the precision, accuracy, and reliability of our measures, both in isolation and compared to extant measures of the same concepts. Our approach assesses the degree to which measures align with shared concepts (content validation), shared rules of translation (data generation assessment), and shared realities (convergent validation). Within convergent validity, we execute two convergent validity tests. First, we examine convergent validity as it is typically conceived – examining convergence between V-Dem measures and extant measures. Second, we evaluate the level of convergence across coders, considering the individual coder and country traits that predict coder convergence. Throughout the chapter, we focus on three indices included in the V-Dem data set: polyarchy, corruption, and core civil society. These three concepts collectively provide a “hard test” for the validity of our data, representing a range of existing measurement approaches, challenges, and solutions.

This chapter sets forth the conceptual scheme for the V–Dem project. We begin by discussing the concept of democracy. Next, we lay out seven principles by which this key concept may be understood – electoral, liberal, majoritarian, consensual, participatory, deliberative, and egalitarian. Each defines a “variety“ of democracy, and together they offer a fairly comprehensive accounting of the concept as used in the world today. Next, we show how this seven-part framework fits into our overall thinking about democracy, including multiple levels of disaggregation – to components, subcomponents, and indicators. The final section of the chapter discusses several important caveats and clarifications pertaining to this ambitious taxonomic exercise.

This chapter recounts how a project of this scale came together and why it has succeeded. Five main factors were responsible for V–Dem’s success: timing, inclusion, deliberation, administrative centralization, and fund–raising. First, planning for V-Dem began at a time when both social scientists and practitioners were realizing that they needed better democracy measures. This made it possible to recruit collaborators and find funding. Second, the leaders of the project were always eager to expand the team to acquire whatever expertise they lacked and share credit with everyone who contributed. Third, the project leaders practiced an intensely deliberative decision–making style to ensure that all points of view were consulted and only decisions that won wide acceptance were adopted. Fourth, centralizing the execution of the agreed–upon tasks helped tremendously by streamlining processes and promoting standardization, documentation, professionalization, and coordination of a large number of intricate steps. Finally, successful fund–raising from a mix of both research foundations and bilateral and multilateral organizations has been critical.

In this chapter we focus on the measurement of five key principles of democracy – electoral, liberal, participatory, deliberative, and egalitarian. For each principle, we discuss (1) the theoretical rationale for the selected indicators, (2) whether these indicators are correlated strongly enough to warrant being collapsed into an index, and (3) the justification of aggregation rules for moving from indicators to components and from components to higher–level indices. In each section we also (4) highlight the top– and bottom–five countries on each principle of democracy in early (1812 or 1912) and late (2012) years of our sample period, as well as the aggregate trend over the whole time period 1789–2017 (where applicable). Finally, we (5) look at how the different principles are intercorrelated in order to assess the trade–offs involved between the conceptual parsimony achieved by aggregating to a few general concepts and the retention of useful variation permitted by aggregating less.

Four characteristics of V-Dem data present distinct opportunities and challenges for explanatory analysis: (1) the large number of democracy indicators (i.e., variables), (2) the measurement of concepts by multiple coders filtered through the V-Dem measurement model, (3) the large number of years in the data set, and 4) the ex ante potential for dependence across countries (generically referred to as spatial dependence). This chapter discusses 3 challenges and 10 opportunities that are implied by these characteristics. At the end of this chapter, we also discuss three assumptions that are implicit in most analyses of observational indicators of macro-features at the national level, which aim to draw conclusions about causal relationships.

Users of V–Dem data should take care to understand how the data are generated because the data collection strategies have consequences for the validity, reliability, and proper interpretation of the values. Chapters 4 and 5 explain how we process the data after collecting the raw scores and how we aggregate the most specific indicators into more general indices. In this chapter we explain where the raw scores come from. We distinguish among the different types of data that V–Dem reports and describe the processes that produce each type and the infrastructure required to execute these processes.

V-Dem relies on country experts who code a host of ordinal variables, providing subjective ratings of latent – that is, not directly observable – regime characteristics. Sets of around five experts rate each case, and each rater works independently. Our statistical tools model patterns of disagreement between experts, who may offer divergent ratings because of differences of opinion, variation in scale conceptualization, or mistakes. These tools allow us to aggregate ratings into point estimates of latent concepts and quantify our uncertainty around these estimates. This chapter describes item response theory models that can account and adjust for differential item functioning (i.e., differences in how experts apply ordinal scales to cases) and variation in rater reliability (i.e., random error). We also discuss key challenges specific to applying item response theory to expert–coded cross-national panel data, explain how we address them, highlight potential problems with our current framework, and describe long-term plans for improving our models and estimates. Finally, we provide an overview of the end–user–accessible products of the V-Dem measurement model.