1. Introduction
Climate change is a major threat, and its mitigation can be considered a pivotal challenge for humanity. There is also growing consensus in the empirical literature that whether and how individuals and the general public think about and view climate change influences both support for climate mitigation policies and individuals’ willingness to make personal choices that may help to mitigate climate change (Ding et al., Reference Ding, Maibach, Zhao, Roser-Renouf and Leiserowitz2011; Drews & van den Bergh, Reference Drews and van den Bergh2015; Gifford, Reference Gifford2011; Leiserowitz, Reference Leiserowitz2005; Vainio & Paloniemi, Reference Vainio and Paloniemi2013). Correspondingly, an ever-increasing number of studies and, more recently, even meta-analyses have sought to identify the antecedents of people’s beliefs about climate change and whether these can be altered through interventions (Furnham & Robinson, Reference Furnham and Robinson2022; Hornsey et al., Reference Hornsey, Harris, Bain and Fielding2016; Rode et al., Reference Rode, Dent, Benedict, Brosnahan, Martinez and Ditto2021; Vlasceanu et al., Reference Vlasceanu, Doell, Bak-Coleman, Todorova, Berkebile-Weinberg, Grayson, Patel, Goldwert, Pei and Chakroff2024).
Upon closer inspection, however, the empirical literature is characterized by the use of a notable variety of items and scales to measure climate change beliefs. In fact, even setting aside the inconsistency in terminologyFootnote 1 , there is huge variation across measures in terms of breadth or comprehensiveness: On the most frugal end, it is not atypical to simply ask for a single truth judgment on an item such as ‘climes change is real’ or ‘climate change is happening’. On the more comprehensive end, a few studies have also targeted perceptions of and attitudes toward (climate-related) risks and occasionally even predictions of future events. More specifically, there are at least four major themes or ‘types of attitudes’(Rode et al., Reference Rode, Dent, Benedict, Brosnahan, Martinez and Ditto2021) which can be summarized as (a) the belief that the climate is changing, (b) the belief that climate change is caused by humans, (c) the belief that climate change is a threat, and (d) beliefs about ‘whether human efforts to mitigate climate change can be effective in limiting increasing global temperature’ (de Graaf et al., Reference de Graaf, Stok, de Wit and Bal2023, p. 2) and at how large a cost, e.g. to the economy. Whereas (a) and (b) can be considered judgments of truth, (c) is primarily a matter of which risks are currently perceived and one’s attitudes towards these, and (d) more strongly taps into opinions and future predictions.
Problematically, there remain large idiosyncrasies about which of these themes are even included in research, and, correspondingly, there is limited knowledge, let alone consensus, on whether to treat them as structurally related and, if so, how. Whereas (a) and (b) are almost always included in studies (but for a recent exception, see Vlasceanu et al., Reference Vlasceanu, Doell, Bak-Coleman, Todorova, Berkebile-Weinberg, Grayson, Patel, Goldwert, Pei and Chakroff2024) and treated as largely interchangeable, fewer studies concurrently consider (c) and/or (d), and these are treated as more independent from (a), (b), and each other—and therefore more often studied in isolation (Rode et al., Reference Rode, Dent, Benedict, Brosnahan, Martinez and Ditto2021). In fact, (d) is typically not considered at all—not even in relatively recent research (Chryst et al., Reference Chryst, Marlon, van der Linden, Leiserowitz, Maibach and Roser-Renouf2018; Latkin et al., Reference Latkin, Dayton, Coyle, Yi, Winiker and German2022; van Valkengoed et al., Reference van Valkengoed, Steg and Perlaviciute2021). In summary, the measurement of judgments and beliefs about climate change is highly inconsistent in terms of content, with little to no consideration whether and how certain themes are related.
Moreover, items or scales used to measure climate change beliefs do not only differ in the themes included and thus content, but also on basic psychometric dimensions. In particular, they differ notably in the number of items and thus internal consistency and reliability. Indeed, whereas climate change belief scales often include only a handful of items at best (Anderson & Becker, Reference Anderson and Becker2018; Chryst et al., Reference Chryst, Marlon, van der Linden, Leiserowitz, Maibach and Roser-Renouf2018; Dias et al., Reference Dias, Vidal, Sousa, Dinis and Leite2020; Lewandowsky et al., Reference Lewandowsky, Gignac and Oberauer2013; Poortinga et al., Reference Poortinga, Spence, Whitmarsh, Capstick and Pidgeon2011; Rothermich et al., Reference Rothermich, Johnson, Griffith and Beingolea2021; Taylor et al., Reference Taylor, Bruine de Bruin and Dessai2014), others are more extensive (Christensen & Knezek, Reference Christensen and Knezek2015; Douenne & Fabre, Reference Douenne and Fabre2020). Another major (and increasing) problem is skew. Given ever more widespread awareness of climate change in the general public, items such as “Do you think the world’s climate is changing?” are often answered in the affirmative by more than 90% of respondents across countries (Poortinga et al., Reference Poortinga, Whitmarsh, Steg, Böhm and Fisher2019), implying heavy skew. Scales further differ in the number of reverse-keyed items, if any; many include no reverse-keyed items whatsoever and are thus prone to response biases (especially acquiescence). Some also vary response formats (only some use Likert-type response scales and thus many do not allow for straightforward aggregation across items let alone scales). Indeed, it is still common to use ad hoc single items or scales with entirely unknown psychometric properties and even if scales (or at least subsets of items) are re-used or evaluated in psychometric terms, crucial psychometric properties remain unknown—most strikingly, long-term retest reliability. Arguably, if a construct of interest is not at least somewhat rank-order stable, it makes little sense to investigate its stable antecedents (such as demographics or personality) and it is also difficult to judge how meaningful change (e.g. due to an intervention) actually is.
Even the most recent measure available (at the onset of the present research), the climate change perceptions scale (van Valkengoed et al., Reference van Valkengoed, Steg and Perlaviciute2021), is indicative of some of these psychometric limitations. Whereas its development laudably started out with a relatively large initial item set and ended with a relatively short (8 item) scale yielding excellent internal consistency and covering themes a-c (missing out only on the mitigation theme (d)), it is strongly imbalanced (with only a single reverse-coded item) and has unknown retest reliability. More worryingly still, a pilot study of this scale conducted prior to the present research revealed that the majority of its items and subscales and indeed the full scale suffer from moderate to severe skew (.91 ≤ |s| ≤ 1.88) and that unidimensionality of the full scale does not hold (in light of notably poor fit of a single factor model: χ2(20) = 230, p < .001; RMSEA = 0.26; SRMR = 0.09) which suggests the scale does not actually indicate a single, unitary constructFootnote 2 . A more detailed description of the pilot study along with complete raw data and analysis code can be found on the OSF (https://osf.io/tqy6j).
In summary, even ignoring the worrisome psychometric limitations (very few items, none reverse-coded) or unknowns (unidimensionality, retest-reliability), climate change scales differ in content. Thus, the measurement of climate change beliefs will not be nomologically consistent (Thielmann & Hilbig, Reference Thielmann and Hilbig2019), implying that substantive conclusions will often depend heavily on the particular scale used. This commensurability (Steel et al., Reference Steel, Schmidt and Shultz2008) or ‘apples and oranges’ problem (Sharpe, Reference Sharpe1997) is a major caveat for the study of individual differences (Leising et al., Reference Leising, Thielmann, Glöckner, Gärtner and Schönbrodt2022), will necessarily increase heterogeneity in findings, can incur the false impression of ‘failed replication’ (using a different climate change belief scale is not a conceptual replication if the scale actually measures a different construct), and may well bias qualitative and quantitative research summaries (i.e., reviews and meta-analyses) which, by their very nature, need to aggregate across studies and thus measure and therefore assume comparability.
Given the problems and caveats outlined so far, the goal of the present research is straightforward: To (i) start with a sufficiently broad item set that is (ii) comprehensive in terms of the four themes sketched above (a-d) and to distill from this a questionnaire measure of climate change beliefs that (iii) covers all themes, (iv) includes a sufficient number of items to ensure that unidimensionality can be tested (i.e., at least 4 items per (sub)scale), (v) balances positively and negatively keyed items (for each (sub)scale) so that the influence of acquiescence can be modeled and controlled for, and (vi) that has satisfactory psychometric properties (including long-term re-test reliability). Note that none of these desiderata, especially in combination, are fulfilled by any of the measures used in current research—including the most recently developed climate change skepticism questionnaire (de Graaf et al., Reference de Graaf, Stok, de Wit and Bal2023).
In Study 1, a baseline set of items will be generated, followed by item selection based on fully specified criteria. The following studies (Studies 2a, 2b and 3) will all serve to confirm the latent structure and basic psychometric properties; Study 2a will additionally generate an ultrashort (4 item) scale, the psychometric viability of which is evaluated in Study 2b. Study 3 will provide evidence of 1-year retest reliability and predictive validity for policy preferences as expressed in voting behavior and individual pro-environmental behavior. For each study, full materials (in verbatim), raw data, and code for all analyses are provided on the OSF (https://osf.io/tqy6j). Information on how sample sizes were determined, data exclusions (if any), and all measures per study are reported below. Sample sizes were determined heuristically (to allow for reliable model identification); nonetheless, even the smallest sample size (Study 1, N = 197) reported on yields high power (>.90) to detect even miniscule model misfit (RMSEA = .03) in all models run (Moshagen & Bader, Reference Moshagen and Bader2024; Moshagen & Erdfelder, Reference Moshagen and Erdfelder2016). All studies were compatible with ethical requirements, based on informed consent, and did not involve deception.
2. Study 1
2.1. Materials, procedure, and participants
To generate a sufficiently large item set for subsequent psychometric item selection, published papers (Arıkan & Günay, Reference Arıkan and Günay2020; Christensen & Knezek, Reference Christensen and Knezek2015; Devine-Wright et al., Reference Devine-Wright, Price and Leviston2015; Dias et al., Reference Dias, Vidal, Sousa, Dinis and Leite2020; Douenne & Fabre, Reference Douenne and Fabre2020; Eisenstadt & West, Reference Eisenstadt and West2017; Häkkinen & Akrami, Reference Häkkinen and Akrami2014; Heath & Gifford, Reference Heath and Gifford2016; Howe et al., Reference Howe, Mildenberger, Marlon and Leiserowitz2015; Kellstedt et al., Reference Kellstedt, Zahran and Vedlitz2008; Leiserowitz, Reference Leiserowitz2005; Lewandowsky et al., Reference Lewandowsky, Gignac and Oberauer2013; Maibach et al., Reference Maibach, Kreslake, Roser-Renouf, Rosenthal, Feinberg and Leiserowitz2015; Markowitz, Reference Markowitz2012; Milfont et al., Reference Milfont, Milojev, Greaves and Sibley2015; Myers et al., Reference Myers, Maibach, Roser-Renouf, Akerlof and Leiserowitz2012; Poortinga et al., Reference Poortinga, Spence, Whitmarsh, Capstick and Pidgeon2011; Rothermich et al., Reference Rothermich, Johnson, Griffith and Beingolea2021; Seroussi et al., Reference Seroussi, Rothschild, Kurzbaum, Yaffe and Hemo2019; Taylor et al., Reference Taylor, Bruine de Bruin and Dessai2014; von Borgstede et al., Reference von Borgstede, Andersson and Johnsson2013; Wachholz et al., Reference Wachholz, Artz and Chene2014; Whitmarsh, Reference Whitmarsh2011) measuring climate change beliefs were reviewed, and items (if available in verbatim) potentially relevant to one of the four themes (a-d) were collected. If items were limited to certain countries, the corresponding specifier was omitted (e.g. “Recent floods and heat-waves [in this country] are due to climate change”). If items did not conform to an agree-disagree response format (e.g. “I believe the cause of global warming is… i) natural fluctuations, ii) human greenhouse gas emissions, iii) increased solar activity”), they were either adapted or, if impossible, dropped. Next, item duplicates—that is, items extremely similar in content (e.g., “there is scientific consensus that climate change is happening” and “practically all scientist agree that climate change is happening”)—were identified, and only one variant was retained. Then, item opposites—i.e., direct negations or logical contraries (e.g., “climate change is a major threat to humanity” and “the so-called ‘climate threat’ is greatly exaggerated”)—were identified and one variant retained with the goal in mind to ultimately achieve approximately 50% of reverse-keyed items. Finally, new items (especially reverse-keyed ones which were grossly underrepresented) were generated, in particular to ensure a sufficient number of distinct items for each theme. The resulting item set consisted of 40 items in total (20 reverse-keyed), with 7 (3 reverse-keyed) reflecting the ‘real’-theme (a), 9 (5 reverse-keyed) reflecting the ‘human-caused’-theme (b), 11 (6 reverse-keyed) reflecting the ‘threat’-theme (c), and 13 (6 reverse-keyed) reflecting the ‘mitigation’-theme (d).
These 40 items (all of which are listed in verbatim along with the instructions in the materials on the OSF) comprising the initial set for psychometric item selection were presented in a web-based study to 197 participants (99 female, aged 18 to 78 years, M = 33.9, SD = 11.7 years) recruited via prolific academic and compensated monetarily for participation. All provided informed consent and were debriefed about the purpose of the study. The study language was English, and all participants were residents of the UK and native speakers. Item order was randomized per participant. The answer scale was a 5-point Likert-type scale ranging from ‘completely disagree’ to ‘completely agree’.
2.2. Results and item selection
Psychometric evaluation of items was based on raw item descriptives and loadings obtained from structural equation modeling of raw scores using the lavaan package in R (Rosseel, Reference Rosseel2012). To account for non-normality in the data, maximum likelihood estimation with robust standard errors and scaled test statistics was employed in all models (Satorra & Bentler, Reference Satorra and Bentler2001). To determine which loadings to consider in the item selection process, models representing different latent structures were specified.
First, a correlated factors model was specified with four latent factors, one for each of the four themes (a-d), and the respective items of each theme serving as indicators. This model fit the data well (χ2(734) = 1152, p < .001; RMSEA = .06; SRMR = .06; CFI = .84; TLI = .83), suggesting that the assumption of four separate (correlated) themes is generally viable. All items except oneFootnote 3 yielded significant positive loadings on their respective factors. As reported in Table 1, loadings were substantial throughout, and latent factor reliabilities were high.
Table 1 Item loadings, latent factor reliabilities and correlations in Study 1 (correlated factors model, 40 items)
As is also reported in Table 1, the four latent factors were very highly correlated, indeed to an extent that could be considered indicative of a single, unidimensional construct. Thus, a single factor model was specified with all items loading on one latent factor—thus essentially assuming unidimensionality across all items. This model, too, fit the data well (χ2(702) = 1126, p < .001; RMSEA = .06; SRMR = .06; CFI = .83; TLI = .82), all item loadings were positive and significant (M λ = .56, SD = .14), highly comparable to those obtained from the four-factors model (r = .99) and the reliability of the latent factor was excellent (ω = .94). Thus, the structural assumption of one single, unidimensional construct across all items was also viable.
Finally, given the viability of both a four-factor and one-factor solution, a higher-order model was specified with the four themes (a-d) as first-order factors (facets) and a single 2nd-order general factor unifying these. This model also fit the data well (χ2(698) = 1086, p < .001; RMSEA = .06; SRMR = .06; CFI = .85; TLI = .84), item loadings (on the four facets) were highly comparable to those obtained in both previous models (both r > .98) and the four facets yielded very high second-order loadings on the general factor (all λ > .88). Thus, the structure of climate change beliefs can also be understood as a higher-order latent construct—general climate change beliefs—indicated by four facets (the themes a-d). In summary, all three models and thus structural assumptions appeared viable and, more importantly, produced highly similar results—especially in terms of item loadings, which constituted the essential information for the subsequent step of item selection.
Item selection proceeded by trading off the following goals and criteria: First, raw item responses needed to yield limited skew (< |1|) at most, and their mean +/− 1SD needed to be within the range of the response scale (1–5). Second, items were preferred that yielded high loadings (λ > .50) in all models considered above. Third, approximately 50% of the selected items per theme (a-d) needed to be reverse-coded. Finally, to arrive at a balanced and economic measure of the four themes and general climate change beliefs, the goal was set to select a similar number of items per theme (a-d) and no more than 16 items total, provided that such a solution yielded good fit (RMSEA < .08 SRMR < .05; CFI, TLI > .90) of the most parsimonious model (the single-factor model) and high reliability (ω ≥ .90).
A corresponding solution jointly maximizing all these criteria to a satisfactory level was found. Specifically, 16 items were selected (see Table 2), exactly four per theme (two of which were reverse-coded per theme) that yielded moderate skew (M = −.75, SD = .33) and high loadings throughout (M λ = .62, SD = .09). A single-factor model across only these 16 items fit the data very well (χ2(104) = 143, p = .01; RMSEA = .05; SRMR = .05; CFI = .96; TLI = .95) and the latent factor reliability was high (ω = .90). To independently verify these findings and the viability of the set of selected items, a psychometric replication/confirmation study was next run.
Table 2 Items selected for the final climate change beliefs scale
Note: # = original item number (out of 40; see OSF) in parenthesis; “R” = reverse-coded; * = selected for ultrashort 4-item scale in Study 2a.
3. Study 2 a)
3.1. Materials, procedure, and participants
The 16 items selected in Study 1 (see Table 2) were completed by 1226 participants (621 female, aged 18 to 77 years, M = 38.5, SD = 12.3 years), again on a 5-point Likert-type scale ranging from ‘completely disagree’ to ‘completely agree’ and with item order randomized per participant. Participants were recruited via darkfactor.org, a website offering self-assessments and feedback on aversive personality. Participants were asked whether they would be willing to complete a few additional, unrelated questions (prior to receiving feedback on their personality) voluntarily and without compensation (for details on consent and a priori fixed in/exclusion criteria, see https://osf.io/93tw6). Those who agreed were presented with the 16 items as specified above. All were native speakers of English, and the vast majority were residents of the USA (n = 1,122; all others were residents of Canada).
3.2. Results and discussion
Modeling proceeded as in Study 1. First, a correlated factors model was specified with four latent factors, one for each of the four themes (a-d), and the four respective items serving as indicators. This model fit the data well (χ2(98) = 513, p < .001; RMSEA = .07; SRMR = .03; CFI = .95; TLI = .94). As reported in Table 3, item loadings were very substantial throughout (all λ ≥ .52, all p < .001) and indeed notably larger than in Study 1, latent factor reliabilities were satisfactory (mitigation) to excellent (all other themes) given only four items per factor, and the four factors were again very strongly correlated. Correspondingly, a single factor model also fit the data well (χ2(104) = 724, p < .001; RMSEA = .08; SRMR = .04; CFI = .93; TLI = .92), yielding high item loadings (M λ = .71, SD = .11) and excellent reliability (ω = .94). Finally, the higher-order model (four themes (a-d) as first-order factors and one 2nd-order general factor) also fit the data well (χ2(100) = 517, p < .001; RMSEA = .07; SRMR = .04; CFI = .95; TLI = .94) and the four facets yielded very high and similar second-order loadings on the general factor (all λ > .90). Thus, overall, results confirmed the psychometric properties of the selected items and highly comparable item properties under each of the structural assumptions (four correlated themes, one single factor, or four facets with a higher-order factor, respectively), all of which were viable (that is accounted well for the data).
Table 3 Item loadings, latent factor reliabilities and correlations in Study 2a (correlated factors model, 16 items)
Nonetheless, it seemed prudent to determine a preferred structural representation, not only to select the most appropriate measurement model but also to gain further theoretical understanding of the construct(s) involved. Thus, to compare models while trading-off model fit and complexity (as a proxy for theoretical parsimony), the normalized evidence ratio (ER; Wagenmakers & Farrell, Reference Wagenmakers and Farrell2004) was computed from Bayesian Information Criterion (BIC) model weights, expressing the likelihood that one model is superior over another given the data and, thus, the degree of (posterior) belief that it reflects the true modelFootnote 4 . Results revealed that the higher-order model was superior to both the four-factor model (ER = .98) and, still more strongly, the single-factor model (ER > .999). Thus, although all models fit the data well in absolute terms, the preferred structural representation—when trading-off fit and parsimony—yields four facets (the themes a-d) and a 2nd-order factor (general climate change beliefs) essentially uniting these.
As one of the main goals in item selection was to arrive at a balanced set of positively vs negatively keyed items for each (sub)scale so as to avail control of acquiescence, it was next tested whether acquiescence can be estimated, how strong it is, and whether it biases the (relationships between) content. To this end, the correlated factor model (one latent factor for each theme, indicated by four respective items) was extended to include a latent acquiescence factor. In this model (closely following Billiet & McClendon, Reference Billiet and McClendon2000) which, crucially, is fit to the raw responses (prior to recoding negatively keyed items), all (unstandardized) loadings on the acquiescence factor are fixed to 1 (reflecting the common assumption that all items are equally affected by acquiescence) and covariances between the content factors (representing the four themes) and the acquiescence factor are set to 0, thus ensuring independence of content and response style. The model fit the data well (χ2(97) = 416, p < .001; RMSEA = 0.06; SRMR = 0.03; CFI = 0.96; TLI = 0.95) and indeed slightly improved upon the fit of the original correlated factor model (Δχ2(1) = 103, p < .001; ΔCFI = .01), confirming the presence of a corresponding response style. However, the overall extent of acquiescence was limited (as indicated by very small loadings on the acquiescence factor, M λ = .12, SD λ = .01). Most importantly, mean (absolute) loadings of items on the four content factors and latent correlations between these were exactly the same (all |Δ| ≤ .01) as in the correlated factor model (see Table 3), thus confirming that acquiescence did not bias the results of interest.
As several of the reverse-coded items rely on negation, it was additionally tested whether these items differ systematically from those that do not. To this end, a model with two latent factors was specified (one indicated by all items involving negation and the other by all the remaining items). The model fit the data well (χ2(103) = 724, p < .001; RMSEA = 0.08; SRMR = 0.04; CFI = 0.93; TLI = 0.91), but it did not improve upon the fit of the single factor model (Δχ2(1) < 0.16, p = .69; ΔCFI = .001) and the two latent factors were almost perfectly correlated (r = .98). As such, items involving negation did not differ systematically from those without negation.
3.3. Item selection for an ultrashort scale
The data of Study 2a was further used to implement another round of item selection with the intent of creating an ultrashort 4-item scale—matching, as closely as possible, the overall construct, maintaining coverage of all themes, retaining balance between positively and negatively keyed items, supporting unidimensionality (i.e., good fit of the single-factor model), and maintaining acceptable reliability despite its brevity. To this end, four items (one per theme, marked with an asterisk in Table 2) were selected based on high loadings (λ > .75) in all three models. A single-factor model involving only these four items fit the data well (χ2(2) = 8.4, p = .02; RMSEA = .07; SRMR = .01; CFI = .99; TLI = .99), yielding high item loadings (all λ > .71, p < .001) and highly satisfactory reliability given only four items (ω = .85). Finally, factor scores of this single latent factor were extracted and subsequently included in the original 16-item higher-order model (four facets and one 2nd-order factor) as an observed variable. The factor scores on the ultrashort 4-item scale correlated almost perfectly with the higher-order general factor (r = .97) and very strongly with all four facets (all r ≥ .90), thus confirming that the ultrashort scale almost perfectly represents the full construct and provides adequate coverage of all four themes. Nonetheless, the psychometric properties of the ultrashort scale were additionally replicated/confirmed in an independent sample (Study 2b).
4. Study 2 b)
4.1. Materials, procedure, and participants
The ultrashort scale with 4 items as selected in Study 2a (see Table 2) was completed by 769 participants (427 female, aged 18 to 77 years, M = 37.1, SD = 11.7 years), again on a 5-point Likert-type scale ranging from ‘completely disagree’ to ‘completely agree’ and with item order randomized per participant. Participants were again recruited via darkfactor.org, exactly as described in Study 2a, with one difference: All were native speakers of German, and the vast majority were residents of Germany (n = 673; practically all others were residents of Austria or Switzerland). Thus, all instructions and items (of the full 16-item set) were first translated by independent bilinguals using the backtranslation technique (Brislin, Reference Brislin1970).
4.2. Results and discussion
The single-factor model across all four items fit the data very well (χ2(2) = 5.6, p = .06; RMSEA = .06; SRMR = .02; CFI > .99; TLI = .98), thus supporting unidimensionality. Item loadings were all positive and significant and indeed substantial in size, though somewhat smaller than in the corresponding model in Study 2a (M λ = .63, SD λ = .16). Correspondingly, reliability of the latent factor was also lower (ω = .70), but nonetheless satisfactory for a four-item scale covering the content of a construct with four facets. Thus, the ultrashort scale can serve as a psychometrically acceptable proxy for climate change beliefs when time is scarce.
5. Study 3
The third and final study served two main goals: First, to provide replication of the psychometric properties of the full 16-item and the ultrashort 4-item climate change beliefs scales as developed in the previous studies and, crucially, to extend these in terms of 1-year retest reliabilities. Second, the goal was to establish basic predictive validity by showing that climate change beliefs are associated with corresponding political and policy preferences and a consequential measure of pro-environmental behavior.
5.1. Materials, procedure, and participants
Data for this study stem from three follow-up waves of the Prosocial Personality Project, a longitudinal project assessing a large number of constructs related to pro-social versus aversive behavior in a professionally managed German online sample (see https://osf.io/m2abp for detailed documentation including a priori specified exclusion criteria and documentation of other publications relying on other subsets of data from the project). Data for T1 of this study stem from two follow-up waves run shortly after the German National Parliament election in September of 2021. Therein, participants completed the full 16-item climate change beliefs measure, reported which party (if any) they had just voted for in said election, and completed a consequential measure of pro-environmental behaviorFootnote 5 .
Reported voting behavior served as a proxy for policy preferences. Specifically, two of the major German parties’ platforms are almost perfectly (in)compatible with climate change beliefs, respectively: On the one hand, the right-wing populist party ‘Alternative für Deutschland’ (AfD) is the only party with seats in the national parliament that explicitly denies climate change or its associated threats and is strictly opposed to climate change mitigation measures. Clearly, a vote for this party is largely incompatible with belief in climate change. On the other hand, the Green Party (“Bündnis ‘90/Die Grünen”) is widely considered ‘Germany’s major environmentalist party’ (Garside & Zhai, Reference Garside and Zhai2022) and the only party with seats in the national parliament that consistently advocates for stricter climate change mitigation policyFootnote 6 . As such, a vote for this party is most compatible with belief in climate change.
As a proxy for individual pro-environmental behavior, choices in the greater-good-game (GGG; Klein et al., Reference Klein, Hilbig and Heck2017) were considered. In the GGG, a nested public goods game (Thielmann et al., Reference Thielmann, Böhm, Ott and Hilbig2021), participants chose between three mutually exclusive options for allocating a monetary endowment: (i) a selfish option (keeping the endowment), (ii) a pro-social option benefiting a small in-group (contributing the endowment to a group account), or (iii) a pro-environmental option (contributing the endowment to a pro-environmental donation account). Thereby, the GGG provides a measure of how much individuals directly prioritize the environment over themselves and a social in-group. Of note, the GGG was implemented in a fully consequential manner, that is, with actual monetary stakes (Lange, Reference Lange2023). Participants completed 10 trials of the GGG (with varying endowment sizes), one of which was randomly selected to determine the actual monetary consequences (payoffs and/or donations, see OSF for details). The sum of participants’ donations to the pro-environmental option was donated to the German Federation for the Environment and Nature Conservation (a federal NGO dedicated to environmental protection, cf. https://www.bund.net); participants were aware of this prior to making their choices. Each participants’ proportion of pro-environmental choices (out of 10) served as the variable of interest—the more pronounced one’s belief in climate change, the larger this proportion should be.
A total of 486 participants (182 female, aged 22 to 67 years, M = 49.7, SD = 11.1 years) completed T1 of the present study. So as to determine 1-year retest reliability, all participants who had completed T1 were re-invited one year later (November 2022) to complete T2 of this study, which included only the full 16-item climate change beliefs measure. A total of 277 participants also completed T2. All measurement occasions involved full informed consent and debriefing, and all participants were compensated monetarily for participating.
5.2. Results and discussion
First, all analyses reported in the previous studies were repeated separately for T1 and T2, respectively. Results (the full details of which are provided on the OSF) were well in line with previous findings, showing adequate fit of all models considered. The themes (a-d) were once again very strongly associated (all r ≥ .80) and 2nd-order loadings in the higher-order model were large (all λ > .80). Item loadings were high in all models, and, correspondingly, reliabilities were satisfactory (ω = .75 for the mitigation theme as a single factor) to excellent (ω = .94 for general climate change beliefs as a single factor across all 16 items). The single factor involving only the 4 items of the ultrashort scale had good reliability (ω = .84) and its factor scores correlated close to perfectly (r = .97) with the overall construct (the 2nd-oder factor in the higher-order model) and very strongly with all facets (all r > .88).
To extend the psychometric analyses to 1-year retest reliability, latent correlations between T1 and T2 were estimated. First, the correlated factors model was extended to both measurement occasions, thus specifying one factor per theme and per measurement occasion (i.e., 8 latent factors in total), indicated by four items each. The model fit the data adequately (χ2(436) = 884, p < .001; RMSEA = .07; SRMR = .06; CFI = .91; TLI = .90) and revealed very high retest-reliabilities of all latent factors (all r tt > .94). Next, the single factor model was extended to both measurement occasions, that is, one latent factor (indicated by all 16 items) per measurement occasion. The model fit the data adequately (χ2(463) = 1208, p < .001; RMSEA = .09; SRMR = .06; CFI = .85; TLI = .84) and yielded a very high retest-reliability (r tt = .93). A single factor model per measurement occasion involving the four items of the ultrashort scale also yielded very high retest-reliability (r tt > .99). Finally, the higher-order model was extended to both measurement occasions, that is, one facet per theme (indicated by four items) and a 2nd-order general factor per measurement occasion (i.e., 10 latent factors in total). This model involved acceptable fit (χ2(455) = 1110, p < .001; RMSEA = .08; SRMR = .06; CFI = .87; TLI = .86) and revealed that the 2nd-order factor also yielded very high retest-reliability (r tt = .94). As a robustness check, all analyses were re-run employing full information maximum likelihood estimation (and thus only assuming “missing at random” rather than “missing completely at random”), yielding highly similar findings. In summary, 1-year retest-reliabilities were excellent throughout, showing that each of the four themes (as a single 4-item scale) and general climate change beliefs—either as a 2nd-order factor across four facets or a single factor (across all 16 items or the four items of the ultrashort form)—are psychometrically sound and that the underlying construct is rank-order stable over a substantial period of time. As a point of comparison, for each of their climate change skepticism (sub)scales, de Graaf et al. (Reference de Graaf, Stok, de Wit and Bal2023) reported consistently smaller re-test correlations across a timespan of only one week.
Next, serving the second major goal of the present study, predictive validity of climate change beliefs was tested. Importantly, as research typically uses observed variables (i.e., means or sums across items of a scale) and corresponding tests, all analyses were performed without latent modeling and with three different indicators of climate change beliefs. First, factor scores of the 2nd-order factor (in the higher order model) were extracted, arguably representing the most comprehensive and appropriate indicator of climate change beliefs. Second, factor scores of the single factor across the four items of the ultrashort scale were extracted, representing the most condensed indicator of climate change beliefs. Third and finally, the simple mean across all 16 items (that is, the typical observed variable) served as an indicator. Since these three indicators were very highly associated (all r > .94) it is unsurprising that all findings were fully replicated with each of the indicators. Thus, for the sake of simplicity, only the specific results based on the factor scores of the 2nd-order factor will be reported here (all others can be reproduced with the data and code provided on OSF).
To test whether climate change beliefs predict policy preferences expressed through voting behavior, two logistic regressions were run, one regressing a vote for the AfD (coded 1, otherwise 0) on climate change beliefs, and the other regressing a vote for the Green Party (coded 1, otherwise 0) on climate change beliefs. As can be seen in the left panel of Figure 1, the probability of voting AfD sharply decreased and the probability of voting Green sharply increased with increasing climate change beliefs. Odds ratios were OR = .24 and OR = 6.2 (all p < .001), respectively, thus corresponding to large effect sizes (Heck et al., Reference Heck, Thielmann, Moshagen and Hilbig2018; Rosenthal, Reference Rosenthal1996). To test whether climate change beliefs predict individual pro-environmental behavior, the correlation between the proportion of pro-environmental choices in the GGG and climate change beliefs was computed. As can be seen in the right panel of Figure 1, both were positively associated (r = .31, p < .001), corresponding to a medium-sized effect (Cohen, Reference Cohen1988). Taken together, the findings clearly support predictive validity of climate change beliefs.
Figure 1 Climate change beliefs (factor scores of the 2nd-order factor) predicting voting AfD or Greens in logistic regressions (left) and individual pro-environmental choices in an OLS regression (right). The shaded area represents the 95%-CI of each prediction.
6. General discussion
Questionnaire measures of attitudes towards, beliefs about, or opinions on climate change have become omnipresent in the literature—often to study their variation (e.g. across nations), antecedents (e.g. demographics), correlates (e.g. conspiracy mentality), or malleability (e.g. through interventions). However, this rapidly expanding area of research is also beset by a striking variety in scales used to measure climate change beliefs, many of which do not even cover the four major themes—that climate change a) is real, b) is human-caused, c) is a threat, and d) can be mitigated—or rely on untested structural assumptions about these (e.g. that they are sufficiently independent and thus actually represent distinct beliefs). Scales entail anything from a single item to 20 or more, often few (if any) reverse-coded items, multiple response formats, questionable or unknown psychometric properties (e.g. skew, unidimensionality, and long-term retest-reliability) and are practically never validated with behavioral measures that are consequential rather than hypothetical (Klein & Hilbig, Reference Klein and Hilbig2019). Therefore, the field is at risk of suffering increasingly obvious heterogeneity in findings (e.g. whether/which interventions might be effective, Rode et al., Reference Rode, Dent, Benedict, Brosnahan, Martinez and Ditto2021), failed replications (both false-alarm and actual), and potential biases in qualitative and quantitative research summaries (i.e., reviews and meta-analyses).
To alleviate such problems, the present research set out to develop scales to measure climate change beliefs that are comprehensive and balanced in covering the four major themes (a-d)), come with specified and tested structural assumptions, include a sufficient number of items (with 50% reverse-coded items per theme), yield known and good psychometric properties (including longer-term retest-reliability), and have predictive validity. To this end, items from previously used scales were collected, adapted, and novel items created so as to generate a large enough item set for initial item selection. In Study 1, a total of 40 items were subjected to item selection based on descriptives and latent modeling, resulting in a final set of 16 items (4 per theme; see Table 2).
This 16-item full climate change beliefs scale subsuming four subscales was thoroughly tested in Study 2a, demonstrating highly satisfactory psychometric properties (both per subscale and overall) and no bias due to acquiescence. Moreover, all structural assumptions considered—four independent (highly correlated) factors, one single unidimensional construct, or four facets of one overarching construct—fit the data appropriately. The higher-order model proved superior when trading-off fit and complexity, and thus the most appropriate representation of climate change beliefs maintains that there are four facets (the themes a-d), united by one 2nd-order general factor, that is, general climate change beliefs.
Data from Study 2a was also used to extract a 4-item ultrashort scale for climate change beliefs. The highly satisfactory psychometric properties of this ultrashort scale were demonstrated in Study 2b and replicated in both measurement occasions of Study 3. It was further shown (and replicated) that the ultrashort scale appropriately covers the content of the full construct (correlating almost perfectly with each facet and the 2nd-order factor in the higher-order model). In summary, general climate change beliefs can be measured with these four items and thus in a psychometrically sound and nonetheless highly economic fashion.
Completing the psychometric picture, Study 3 established very high 1-year retest-reliability for each subscale and thus facet and general climate change beliefs—in all representations (correlated factors, single factor (16-item and 4-item ultrashort), and higher order). Indeed, 1-year rank-order stability was so high (consistently > .90) that it matched or even exceeded what is commonly found for broad, stable personality traits (Bleidorn et al., Reference Bleidorn, Schwaba, Zheng, Hopwood, Sosa, Roberts and Briley2022; Roberts & DelVecchio, Reference Roberts and DelVecchio2000; Specht et al., Reference Specht, Egloff and Schmukle2011). This notable stability suggests that i) it is indeed reasonable to study the stable antecedents of climate change beliefs (Hornsey et al., Reference Hornsey, Harris, Bain and Fielding2016), but also that ii) external events or interventions cannot always be expected to produce large or sustained effects.
Finally, Study 3 provided evidence for predictive validity, as climate change beliefs were strongly negatively associated with voting for the only major German party that openly denies climate change (the AfD) and, concurrently, positively associated with voting for the prime environmentalist party in Germany (the Greens). Moreover, climate change beliefs were also positively associated with (monetarily consequential) individual pro-environmental behavior, yielding a medium-sized effect. In summary, the proposed scale is suitable for the measurement of beliefs that are indeed consequential both for policy preferences on a societal level and behavior on the individual level.
6.1. Limitations
Despite these advantages, certain limitations ought to be acknowledged. For one, as early as the item collection and adaptation phase at the onset of Study 1, it became apparent that the very dynamics of climate change pose a notable challenge for item construction. As ever more people around the world accept climate change as a fact (Poortinga et al., Reference Poortinga, Whitmarsh, Steg, Böhm and Fisher2019) and ever more recognize its dangers (Vlasceanu et al., Reference Vlasceanu, Doell, Bak-Coleman, Todorova, Berkebile-Weinberg, Grayson, Patel, Goldwert, Pei and Chakroff2024), flat-out denialism items that were perfectly viable in the early 2010s are now increasingly problematic in psychometric terms. Arguably, some items of the scale developed herein may meet a similar fate. For example, an item such as ‘The economic and societal consequences of climate change will be severe’ may soon need to be worded in present rather than future form. Similarly, the item ‘the increase in global temperatures can be limited to about 1.5 °C or 2 °C’ may become obsolete once warming has reliably surpassed these thresholds. Then again, one might argue that there will be very little need for climate change belief questionnaires at that point anyway. In any case, scales measuring peoples’ views on an issue as dynamic as climate change will need to be re-evaluated and adapted at regular intervals.
Another limitation lies in the samples studied herein. Even though participants were recruited via very different sources and came from different countries, practically all reside(d) in western Europe or North America. Even though the countries studied actually cover the recently observed cross-country variance in climate change beliefs quite well—ranging from Canada with relatively pronounced beliefs across Germany and the UK on the intermediate level to the USA at the lowest end (Vlasceanu et al., Reference Vlasceanu, Doell, Bak-Coleman, Todorova, Berkebile-Weinberg, Grayson, Patel, Goldwert, Pei and Chakroff2024)—they are all western, industrialized, and highly developed economies. Certain politically motivated perceptions—e.g., that climate change mitigation is economically more costly than climate change itself—may be rather unique to these countries and far less likely elsewhere. This implies that future research will need to critically test whether scales such as the one presented herein are equally viable in other societal contexts.
6.2. A note on latent factors versus observed variables
Much questionnaire research still relies on observed variables such as means (or sums) across items of a scale. Given the findings presented herein, unidimensionality in conjunction with simple structure appears to hold approximately for all themes jointly; also, approximate unidimensionality holds for the full 16-item scale and the ultrashort 4-item scale alike. As such, it is certainly a viable option to compute observed means (or sums) for any of these. Indeed, observed scores across all 16 items were highly similar and produced highly consistent results. As such, apart from the well-known general advantages of latent modeling, there is no reason to advise against observed variables computed across the four items of a theme, all 16 items of the full scale, or the four items of the ultrashort scale.
However, whereas general climate change beliefs can be approximated by the four items of the ultrashort scale, it is not recommended to attempt to measure the themes (a-d) with only these four items (i.e., one item per theme); once the goal is to represent the themes (rather than just what unites them, that is, general climate change beliefs), the 16-item scale ought to be used (i.e., four items per theme). Of course, whether or not to distinguish between the themes at all is primarily a matter of the research question at hand—e.g., whether an intervention impacts these differently—and therefore cannot be subject to a general recommendation. However, it should be noted that the very high correlation between the themes implies that it is a priori relatively unlikely to find effects that are unique to subsets of these. As such, it cannot be recommended that a theme (or subset of themes) be measured in isolation while ignoring the remaining theme(s), as this bears the risk of falsely attributing an effect to a particular theme even though it holds for all of them.
6.3. Conclusion
As research on climate change beliefs moves forward, it will arguably benefit from shifting towards scales that vary less in content and psychometric properties so that measurement can be considered psychometrically adequate and nomologically consistent—ensuring that, across studies, research is based on largely interchangeable sets of indicators of the same underlying (latent) construct. This is not to claim that the scales put forward herein are the most viable, let alone the only solution; however, it would seem beneficial if other scales either entailed a comparable (or greater) level of comprehensiveness in terms of content and/or came with known and comparable psychometric properties. To this end, the item sets identified herein may serve as a useful baseline.
Acknowledgements
I am greatly indebted to Isabel Thielmann for assistance in running Studies 1 and 3 and to Rebekka Heunisch and Hendrik Henze for assistance in collecting and reviewing items from prior studies. This work was supported by the German Research Foundation (DFG, grant numbers KL 3365/1-1 and HI 1600/7-1).
Open access
