A study recently published in this journal showed that agreement with conspiracy theories about the Covid-19 pandemic is associated with risky, non-compliant behaviours (Freeman et al., Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a). It also indicated that this agreement is very common: 45% of British participants seemed to agree that Covid-19 is a bioweapon designed by China to destroy the West, while 20% seemed to agree that the pandemic is a conspiracy by Jews or Muslims. Accurate or not, these statistics paint a worrying picture. If accurate, millions of British people need to be disabused of wild conspiracy theories. If inaccurate, especially if they exaggerate the popularity of conspiracy theories, they could normalise Antisemitic, Islamaphobic, and conspiracist viewpoints (McManus, D'Ardenne, & Wessely, Reference McManus, D'Ardenne and Wessely2020), and misdirect policy, interventions, and further research.

McManus et al. (Reference McManus, D'Ardenne and Wessely2020) pointed out that Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) study indeed runs these risks, because of a response scale that gave participants four options to agree (from ‘agree a little’ to ‘agree completely’), and only one other option (‘do not agree’). This imbalance of options is likely to cause participants who tend to acquiesce to perceived demands of survey questions to report inflated levels of agreement (Hibbing, Cawvey, Deol, Bloeser, & Mondak, Reference Hibbing, Cawvey, Deol, Bloeser and Mondak2019).

We agree with this critique. As researchers who have published many papers on conspiracy theories, including their conceptualisation and measurement (Douglas, Sutton, & Cichocka, Reference Douglas, Sutton and Cichocka2017; Douglas et al., Reference Douglas, Uscinski, Sutton, Cichocka, Nefes, Ang and Deravi2019; Douglas & Sutton, Reference Douglas and Sutton2018; Lantian, Muller, Nurra, & Douglas, Reference Lantian, Muller, Nurra and Douglas2016; Sutton & Douglas, Reference Sutton and Douglas2020), we do not recall seeing a scale like Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a). Scales typically provide an even balance of responses to reject or accept conspiracy theories. This allows participants to express any view on the assumed continuum between strong disagreement and strong agreement. Since responses are typically below or near the midpoint on such scales (e.g. Imhoff and Lamberty, Reference Imhoff and Lamberty2017; Jolley and Douglas, Reference Jolley and Douglas2014), omitting degrees of disagreement seems an important mistake. Participants who disagree with a conspiracy theory, but are willing to admit that it might have some merit, may feel that they have no option but to select one of the ‘agree’ responses. This hypothetical dilemma lends new meaning to the saying ‘agreeing to disagree’.

To test the hypothesis that Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) scale exaggerates agreement with conspiracy theories, we ran a brief, pre-registered study. Materials, anonymised data, and results are available on the Open Science Framework website: https://osf.io/xpvrz. We chose three conspiracy theories from Freeman et al., targeting Jews, Muslims, and China, that featured prominently in a press release (University of Oxford, 2020) and attracted media attention. We presented each to 748 British participants recruited from Prolific, a widely used survey platform (Peer, Brandimarte, Samat, & Acquisiti, Reference Peer, Brandimarte, Samat and Acquisiti2017), who were British nationals resident in the UK aged 18 or over, and not currently students since this group is over-represented on Prolific. Their age ranged from 18 to 80 (M = 38.75, s.d. = 12.70); 506 were female, 238 male, and 4 were gender queer; 681 were White, 18 Black, 29 Asian, and 20 were mixed race.

Participants were then randomly assigned to three groups. The first group (n = 251) were given Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) response scale. The second (n = 251) responded on a conventional five-point scale featuring two options to disagree, two to agree, and a ‘neither agree nor disagree’ option (see Douglas et al., Reference Douglas, Uscinski, Sutton, Cichocka, Nefes, Ang and Deravi2019 for a summary of different conspiracy belief measures). The third group (n = 246) responded on a nine-point scale constructed by mirroring each of the four agreement responses in Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) study with a corresponding disagreement response, and included a ‘neither agree nor disagree’ option.

Following Freeman et al. (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a, Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambeb) and our pre-registration, we coded as agreement (1) the four response options expressing agreement on Freeman et al.'s scale and the nine-point extension, and either of the options expressing agreement on the conventional five-point scale. Responses were otherwise coded as not expressing agreement (0). Thus, across the three conspiracy theories, participants could score between 0 (agreed with none) and 3 (agreed with all).

All conspiracy theories, response options and response proportions are presented in Table 1. It reveals strikingly lower rates of agreement than in Freeman et al. (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a). Even on the same response scale, 2% or 3% of our participants agreed with the conspiracy theories about Jews and Muslims (compared to 20% in Freeman et al.), and 32% (compared to 45%) agreed with the China theory. These differences between studies were expected (see pre-registration) and significant (ps < 0.001). Their magnitude is surprising and noteworthy, but also difficult to interpret since the studies differ in many ways. For example, our study was run in late June 2020 and Freeman et al.'s study was run in early May; ours used a relatively educated sample, among whom slightly lower agreement with conspiracy theories can be expected (Douglas, Sutton, Callan, Dawtry, & Harvey, Reference Douglas, Sutton, Callan, Dawtry and Harvey2016).

Table 1. Proportion of respondents selecting each response to the three conspiracy theories

Note: Responses coded as agreement are shaded. ‘Sum agreement’ represents the sum of these responses. For comparison, Freeman et al. (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) report 19.2% sum agreement for the Jewish conspiracy theory, 19.9% with the Muslim conspiracy theory, and 45.4% with the China conspiracy theory. For the nine-point response scale, ‘No agreement’ subsumes the first five responses (Disagree completely, Disagree a lot, Disagree moderately, Disagree a little, Neither agree nor disagree). For the Jewish, Muslim, and China conspiracy theories, the ‘Neither agree nor disagree’ option on the nine-point scale was selected by 1.6%, 2.4%, and 8.9% of participants, respectively. The remaining responses were disagree responses.

More pertinent, we found levels of agreement half as low again, or lower, when we used conventional agree−disagree scales. Agreement with the China conspiracy theory reduced to roughly 10%; agreement with the conspiracy theories about Jews and Muslims fell to around 1–1.5%. The levels of agreement on the five-point and nine-point agree−disagree scales were not significantly different from each other, p = 0.712, but were significantly lower than on Freeman et al.'s scale (both ps < 0.001).

Our results suggest that Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) estimates of the popularity of Covid-19 conspiracy theories were overestimated. In their reply to McManus et al. (Reference McManus, D'Ardenne and Wessely2020), Freeman et al. (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020b) wrote that, ‘the item content, not the scale, seems to us to merit the real focus’, but in our study the scale doubled the apparent popularity of the item content. As happens often (Lee, Sutton, & Hartley, Reference Lee, Sutton and Hartley2016), striking results of Freeman et al.'s (Reference Freeman, Waite, Rosebrock, Petit, Causier, East and Lambe2020a) study were highlighted in a press release that stripped them of nuance and caveats, and led to some sensational and misleading media reporting that may have complicated the very problems that we all, as researchers, are trying to help solve.