2.1. Hierarchical Beta Fusion Model

In many cases, forecasters provide their forecasts to only a subset of the available queries. In order to also be able to accurately model the forecasting behavior of forecasters who only provided a small number of forecasts, we propose the Hierarchical Beta Fusion Model. This allows taking advantage of the statistical properties of hierarchical models and their psychological interpretations. Their statistical properties are increased statistical power since parameter inference is based on more data that share information across groups, and reduced variance of parameter estimates, known as shrinkage (Britten et al., Reference Britten, Mohajerani, Primeau, Aydin, Garcia, Wang, Pasquier, Cael and Primeau2021). In addition, they allow modeling of individual differences between forecasters, make it possible to model forecasters who only provided few forecasts, and provide information on the distribution of the forecasters’ individual parameters (Lee, Reference Lee, Farrell and Lewandowsky2018b). In particular, the model’s hyperparameters indicate how the forecasters behave on average, how variable their behavior is, and the associated hyperpriors make assumptions about the forecasters’ average behavior and variability explicit.

In the Hierarchical Beta Fusion Model, we use a beta prior on the model parameter $\mu _j^k$ , parameterized with mean $u_j^\mu $ and maximum variance proportion $p_j^\mu $ . The proportion of the maximum variance $\rho _j^k$ is also modeled with a beta distribution with mean $u_j^{\rho }$ and maximum variance proportion $p_j^{\rho }$ . To avoid values of $\rho _j^k$ too close to 0 or 1 that may get the Gibbs sampler in trouble, we constrained this beta distribution and all other beta priors on maximum variance proportions $\rho , p$ between 0.001 and 0.999. As prior distribution for the proportion $\pi $ of true queries, we chose a uniform beta distribution Beta(1,1). The graphical model of the Hierarchical Beta Fusion Model is shown in Figure 2(a). A complete overview over the corresponding modeling distributions and priors, also including the priors of the hyperparameters $u_j^\mu , p_j^\mu , u_j^{\rho }, p_j^{\rho }$ , is given as

(2.3)

Based on labeled training data, the model parameters $\mu _j^k$ , $\rho _j^k$ , and $\pi $ can be inferred using Gibbs sampling (e.g., by specifying this model in JAGS). Since we assume that human forecasters are on average consistent between different queries, we can use the learned parameters to infer the posterior probability of the true label $t_n$ of new unseen forecasts $x_n^k$ as the fusion result. Inference of $t_n$ can either also be realized using Gibbs sampling, or the posterior probability of $t_n$ can be computed analytically using the closed-form probability density function of the beta distribution:

(2.4) $$ \begin{align} p(t_n=j|{\boldsymbol{x}_{\boldsymbol{n}}},{\boldsymbol{\mu}_{\boldsymbol{j}}}, {\boldsymbol{\rho}_{\boldsymbol{j}}}, \pi) &\propto \pi^j(1-\pi)^{1-j}\prod_{k=1}^{K}{\text{Beta}'(x_n^k;\mu_j^k,\rho_j^k)}. \end{align} $$

Figure 2 Graphical models of the hierarchical (a) and non-hierarchical (b) beta fusion models.

2.2. Non-hierarchical Beta Fusion Model

The normative Hierarchical Beta Fusion model represents the forecasting behavior of each forecaster individually, i.e., for each forecaster an individual set of beta parameters $\mu _j^k$ , $\rho _j^k$ is learned. In this way, we can model interindividual differences between forecasters, e.g., different levels of expertise, and can exploit these learned properties for fusion by giving more weight to a better-performing forecaster. However, since the related approach by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) also compared hierarchical and non-hierarchical versions of their fusion models, we also compare our normative Hierarchical Beta Fusion Model to a non-hierarchical version of the model, which assumes exchangeable forecasters that behave similarly. In this non-hierarchical Beta Fusion Model, we model the forecasts $x_n^k$ with the same beta distributions for all forecasters with shared parameters $\mu _j$ and $\rho _j$ for $k=1,\dots ,K$ and $j\in \{0,1\}$ . Thus, we learn only two beta distributions for all forecasters, the beta distribution with mean $\mu _0$ and maximum variance proportion $\rho _0$ models the forecasting behavior of all forecasters for false queries, the beta distribution with mean $\mu _1$ and maximum variance proportion $\rho _1$ models their forecasting behavior for true queries. The priors on $\mu _j$ and $\rho _j$ are uniform distributions Beta(1,1). As for the Hierarchical Beta Fusion Model in Section 2.1, the prior for proportion $\pi $ is an uninformative beta prior Beta(1,1). We illustrate the graphical model of the non-hierarchical Beta Fusion Model in Figure 2(b). All corresponding modeling distributions and priors can be summarized as

(2.5)

Again, given labeled training data, we can infer the model parameters $\mu _j$ , $\rho _j$ , and $\pi $ using Gibbs sampling, and the fused result for some unseen forecasts $x_n^k$ of multiple forecasters is the posterior probability over $t_n$ given the learned model parameters and forecasts $x_n^k$ . Using equation (2.4) as for the Hierarchical Beta Fusion Model and dropping index k for $\mu $ and $\rho $ allows the analytical computation of the posterior over $t_n$ .

2.3. Beta calibration

A forecaster is well calibrated if their probability estimate matches the respective relative frequency of occurrence, i.e., if $100x\%$ of the statements to which the forecaster assigns a probability of x are true or $100x\%$ of the events to which the forecaster assigns a probability of x occur (Brenner et al., Reference Brenner, Koehler, Liberman and Tversky1996). The calibration of a human forecaster can be measured empirically by binning the provided probability estimates and computing the proportions of true events for each bin. It is customary to illustrate this relationship with the so-called calibration curve, which plots the proportions of true events as a function of the human forecast probabilities. If the resulting calibration curve is the identity function, the forecaster is perfectly calibrated. If not, a function can be fitted to the empirical calibration curve. This calibration function can then also be used to recalibrate probability estimates, i.e., to correct for overconfident or underconfident judgments (Turner et al., Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014).

While various different functions can serve as calibration functions, the LLO function is frequently used. For example, Lee and Danileiko (Reference Lee and Danileiko2014) and Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) explicitly include the LLO function in their Bayesian fusion models for calibrating the provided forecasts. However, by modeling the provided forecasts with a probability distribution, one can also calibrate them implicitly. This means that given a probabilistic generative fusion model, as we provide in this work, the calibration function is not chosen empirically, but instead the normative calibration function for the respective model can be derived.

With the Hierarchical Beta Fusion Model, we model forecasts $x_n^k$ of forecaster k conditioned on the true label $t_n$ with a beta distribution

(2.6) $$ \begin{align} P(x_n^k = x|t_n=j) &= \frac{x^{\alpha_j^k-1}(1-x)^{\beta_j^k-1}}{\text{B}(\alpha_j^k, \beta_j^k)}. \end{align} $$

The corresponding calibration function, which is called the beta calibration function, can be derived using Bayes’ rule

(2.7) $$ \begin{align} \text{BC}(x) = P(t_n=1|x_n^k=x) &= \frac{\frac{x^{\alpha_1^k-1}(1-x)^{\beta_1^k-1}}{\text{B}(\alpha_1^k, \beta_1^k)}\pi}{\frac{x^{\alpha_1^k-1}(1-x)^{\beta_1^k-1}}{\text{B}(\alpha_1^k, \beta_1^k)}\pi+\frac{x^{\alpha_0^k-1}(1-x)^{\beta_0^k-1}}{\text{B}(\alpha_0^k, \beta_0^k)}(1-\pi)} \nonumber\\ &= \frac{1}{1+\frac{\text{B}(\alpha_1^k,\beta_1^k)}{\text{B}(\alpha_0^k,\beta_0^k)}\frac{(1-x)^{\beta_0^k-\beta_1^k}}{x^{\alpha_1^k-\alpha_0^k}}\frac{1-\pi}{\pi}} \end{align} $$

with $\pi =P(t_n = 1)$ as introduced above. The function in (2.7) has been first introduced by Kull et al. (Reference Kull, Silva Filho and Flach2017) in the context of machine learning for calibrating the probabilistic outputs of classification algorithms.

Interestingly, the LLO calibration function used by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) and Lee and Danileiko (Reference Lee and Danileiko2014) can be derived as a special case of the beta calibration function (Kull et al., Reference Kull, Silva Filho and Flach2017). If we constrain the beta distributions to be symmetric around $\frac {1}{2}$ , i.e., $\alpha = \alpha _0^k = \beta _1^k$ and $\beta = \beta _0^k = \alpha _1^k$ , the resulting calibration function is

(2.8) $$ \begin{align} \text{LLO}(x) = P(t_n=1|x_n^k=x) &= \frac{\frac{\pi}{1-\pi}x^{\beta-\alpha}}{\frac{\pi}{1-\pi}x^{\beta-\alpha}+(1-x)^{\beta-\alpha}} \nonumber\\ &= \frac{\delta x^\gamma}{\delta x^\gamma+(1-x)^\gamma} \end{align} $$

with $\delta =\frac {\pi }{1-\pi }$ and $\gamma = \beta - \alpha $ . Beta calibration in (2.7) is more flexible than LLO calibration in (2.8) because it does not assume symmetric beta distributions for $t_n=0$ and $t_n=1$ . Thus, beta calibration can consider that the forecasters’ behavior might be different for different truth values $t_n$ . In contrast, LLO calibration assumes symmetric beta distributions for $t_n=0$ and $t_n=1$ and therefore symmetric forecasting behavior for true and false queries, which might not hold for real forecasts. A more detailed comparison of beta calibration and LLO calibration including example calibration functions can be found in Section 3.5.

2.4. Hierarchical Symmetric Beta Fusion Model

Since modeling the forecasts with symmetric beta distributions results in calibrating them with the LLO calibration function (Section 2.3), we are interested in comparing the original beta fusion model using asymmetric beta distributions to a symmetric beta fusion model using symmetric beta distributions, which assumes humans to show symmetric forecasting behavior given true or false queries. In the Hierarchical Symmetric Beta Fusion Model, we thus model forecasts $x_n^k$ with two symmetric beta distributions with parameters $\mu _0^k = \mu ^k$ , $\rho _0^k = \rho ^k$ and $\mu _1^k = 1-\mu ^k$ , $\rho _1^k = \rho ^k$ . We set a beta prior on $\mu ^k$ with mean $u^{\mu }$ and maximum variance proportion $p^{\mu }$ and model $\rho ^k$ with a beta distribution with mean $u^{\rho }$ and maximum variance proportion $p^{\rho }$ . Similar to the asymmetric beta fusion models, the proportion $\pi $ of true queries is modeled with an uninformed prior Beta(1,1). Figure 3(a) shows the graphical model of the Hierarchical Symmetric Beta Fusion Model. All modeling distributions and priors are given as

(2.9)

Figure 3 Graphical models of the hierarchical (a) and non-hierarchical (b) symmetric beta fusion models.

The model parameters $\mu ^k$ , $\rho ^k$ , and $\pi $ are estimated from labeled training data using Gibbs sampling. For fusing unseen forecasts $x_n^k$ , the posterior probability of their true label $t_n$ can be computed analytically given the learned model parameters:

(2.10) $$ \begin{align}\quad \!p(t_n= 0|{\boldsymbol{x}_{\boldsymbol{n}}},\boldsymbol{\mu}, \boldsymbol{\rho}, \pi) &\propto (1-\pi)\prod_{k=1}^{K}{\text{Beta}'(x_n^k;\mu^k,\rho^k)}, \end{align} $$

(2.11) $$ \begin{align} p(t_n= 1|{\boldsymbol{x}_{\boldsymbol{n}}},\boldsymbol{\mu}, \boldsymbol{\rho}, \pi) &\propto \pi\prod_{k=1}^{K}{\text{Beta}'(x_n^k;1-\mu^k,\rho^k)}. \end{align} $$

2.5. Non-hierarchical Symmetric Beta Fusion Model

As for the asymmetric beta fusion model, we also examine a non-hierarchical version of the symmetric beta fusion model. In the non-hierarchical Symmetric Beta Fusion Model, all forecasters’ forecasts $x_n^k$ are modeled with the same two symmetric beta distributions for $t_n=0$ and $t_n=1$ with parameters $\mu _0 = \mu $ , $\rho _0 = \rho $ and $\mu _1 = 1-\mu $ , $\rho _1 = \rho $ . The priors for $\mu $ and $\rho $ are uniform distributions Beta(1,1). As in all models, we set an uninformative prior Beta(1,1) on the proportion of true queries $\pi $ . The graphical model of the non-hierarchical Symmetric Beta Fusion Model is presented in Figure 3(b). An overview of all modeling distributions and priors is given as

(2.12)

The model parameters $\mu $ , $\rho $ , and $\pi $ are estimated from labeled training data using Gibbs sampling. For fusing unseen forecasts $x_n^k$ , the posterior probability of their true label $t_n$ can be computed analytically using equations (2.10) and (2.11) with dropping index k for parameters $\mu , \rho $ .

3.1. Data sets

We evaluated the performances of the proposed Bayesian fusion models and the reference models by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) on two data sets, namely the Turner data set (Section 3.1.1), which is the data set Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) used for evaluating their fusion models, and our new data set (Section 3.1.2).

3.2. Cross-validation

In order to evaluate the different models on the data sets described above (Section 3.1), we split the data into training and test sets using leave-one-out (LOO) cross-validation. While Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) evaluated with 10-fold cross-validation, we preferred leave-one-out cross-validation over k-fold cross-validation since it allows training the model on almost the entire data set, which reduces the evaluation bias. Also, it comes with zero randomness in the partitioning of the data and is therefore straightforwardly reproducible.

If a data set consists of M queries, we obtain M LOO training sets, each including $M-1$ data points. For each of the resulting M training sets, we inferred the posterior distribution of each model’s parameters using Gibbs sampling. We implement Gibbs sampling for inference using JAGS (Plummer, Reference Plummer2003). For fitting the model parameters given labeled training data, we ran 2 parallel chains, each consisting of 1,000 samples with a burn-in of 1,000 samples.

For fusing the one example in the test set, one could now use the means of the obtained posterior distributions as point estimates for the parameters and compute the posterior over the true label $t_n$ analytically given these point estimates. However, in order to consider the uncertainty of our parameter estimates, we computed the posterior over $t_n$ analytically for each sample of the model parameters’ posterior distribution, as for example in (2.4), and averaged all obtained posteriors to the final fused forecast.

3.3. Performance measures

As performance measures, we consider Brier score, 0–1 loss, and mean absolute error. The Brier score (Brier, Reference Brier1950) is a popular metric for quantifying human forecast performance, used by e.g., Karvetski et al. (Reference Karvetski, Olson, Mandel and Twardy2013), Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), Hanea et al. (Reference Hanea, Wilkinson, McBride, Lyon, van Ravenzwaaij, Singleton Thorn, Gray, Mandel, Willcox, Gould, Smith, Mody, Bush, Fidler, Fraser and Wintle2021), and Satopää (Reference Satopää2022). It is a strictly proper scoring rule (Murphy, Reference Murphy1973), meaning that it is optimized when people report their true beliefs of the probability instead of intentionally providing more or less extreme probabilities. The Brier score is defined as the mean squared error between the predicted probabilities $x_n$ and the true labels $t_n$ ,

(3.1) $$ \begin{align} \text{BS} &= \frac{1}{N} \sum_{n=1}^{N}{(x_n-t_n)^2}. \end{align} $$

Thus, the best attainable Brier score is 0 and the worst is 1. Interestingly, if a forecaster always provides 0.5 as her estimate, the resulting Brier score will be 0.25. Thus, a model should at least achieve a Brier score below 0.25.

It is controversial whether the Brier score is a suitable metric for comparing the performances of different forecasting systems, since as a strictly proper scoring rule it was originally developed in order to measure if forecasters report their true beliefs, not to compare different forecasters (Steyvers et al., Reference Steyvers, Wallsten, Merkle and Turner2014). Also, it can be dominated by outliers (Canbek et al., Reference Canbek, Temizel and Sagiroglu2022), though not as much as e.g., the log-loss. Still, it is commonly used for comparing forecasters’ performances (Baron et al., Reference Baron, Mellers, Tetlock, Stone and Ungar2014; Hanea et al., Reference Hanea, Wilkinson, McBride, Lyon, van Ravenzwaaij, Singleton Thorn, Gray, Mandel, Willcox, Gould, Smith, Mody, Bush, Fidler, Fraser and Wintle2021; Karvetski et al., Reference Karvetski, Olson, Mandel and Twardy2013; Ranjan & Gneiting, Reference Ranjan and Gneiting2010; Satopää, Reference Satopää2022; Turner et al., Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), so we report it here, too.

The 0–1 loss describes the proportion of incorrect forecasts to the total number of all forecasts,

(3.2) $$ \begin{align} \text{L}_{01} &= \frac{1}{N} \sum_{n=1}^{N}{\left\{ \begin{array}{ll} 1 & \, \textrm{if} \quad |t_n-x_n|\geq 0.5 \\ 0 & \, \textrm{else} \\ \end{array} \right.}, \end{align} $$

and thus ranges between 0 and 1, with lower values indicating better performances. In comparison to the Brier score, the 0–1 loss is more easily interpretable and directly compares the forecasters’ performances. However, it disregards their uncertainty by considering a forecast as correct, if its corresponding probability is closer to the true label $t_n$ of the respective query.

To overcome the limitations of the Brier score and 0–1 loss, we additionally evaluate the different fusion methods in terms of mean absolute error (Canbek et al., Reference Canbek, Temizel and Sagiroglu2022; Ferri et al., Reference Ferri, Hernández-Orallo and Modroiu2009). Mean absolute error (MAE) measures the absolute difference between the forecasted probability and the true label:

(3.3) $$ \begin{align} \text{MAE} &= \frac{1}{N} \sum_{n=1}^{N}{|x_n-t_n|}. \end{align} $$

Similar to the Brier score (mean squared error) and 0–1 loss, it ranges between 0 and 1, with lower values indicating higher performance. However, in contrast to the Brier score, which can be dominated by outliers, MAE is more robust to outliers (Canbek et al., Reference Canbek, Temizel and Sagiroglu2022). Also, it is straightforwardly interpretable and a more natural and intuitive metric for comparing different fusion models without disregarding their uncertainty. However, note that MAE is an improper scoring rule (Buja et al., Reference Buja, Stuetzle and Shen2005), so it incentivizes overconfident forecasts.

3.4. Model performances

On both data sets described in Section 3.1, we evaluate the four Bayesian fusion models introduced in Section 2 in terms of Brier score, 0–1 loss, and mean absolute error (MAE). In addition, we compare our fusion methods’ performances to the models by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014). Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014) present several Bayesian fusion models, which explicitly consider the calibration of forecasts with the LLO calibration function. Their key question is whether it is better first to average the forecasts or first to calibrate them. Hence, the proposed models are three non-hierarchical models, Average then Calibrate (ATC), Calibrate then Average (CTA), Calibrate then Average using log-odds (CTALO), and two hierarchical models, Hierarchical Calibrate then Average (HCTA), and Hierarchical Calibrate then Average on log-odds (HCTALO).Footnote ⁶ For reference, they also evaluate the performance of Unweighted Linear Opinion Pool (ULINOP) as a baseline. Since ULINOP is known to be biased toward 0.5 (Baron et al., Reference Baron, Mellers, Tetlock, Stone and Ungar2014), we additionally evaluate the performance of Probit Average (PAVG) (Satopää et al., Reference Satopää, Salikhov, Tetlock and Mellers2023) as another benchmark. For PAVG, we first transform all forecasts with probit, then average the transformed forecasts, and finally transform this average back to probability score. Here, we investigate how a normative model that implicitly calibrates the forecasts by modeling them with beta distributions will perform relative to all these models. In particular, we investigate whether the normative approach increases the performance of the fused forecast.

Figure 4 shows the means and standard errors of the mean of Brier score, 0–1 loss, and MAE on the Turner data set (Section 3.1.1). The beta fusion models introduced in this work are abbreviated as HB (Hierarchical Beta Fusion Model), B (non-hierarchical Beta Fusion Model), HSB (Hierarchical Symmetric Beta Fusion Model), and SB (non-hierarchical Symmetric Beta Fusion Model). According to the Brier score, the three non-hierarchical Turner models ATC, CTA, and CTALO perform best with $\text {BS} \approx 0.125$ . These results are different to the results reported by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), which were obtained using 10-fold cross-validation and favored the HCTALO model. The best beta model is HSB with $\text {BS}=0.151$ , which performs comparably to HB and the hierarchical Turner models HCTA and HCTALO. The non-hierarchical beta models B and SB perform worst with the Brier scores of about 0.25, which is close to the performance of a forecaster that always forecasts 0.5. As per 0–1 loss, the ranking of the models is different. HSB performs similarly to ATC, CTA, CTALO, and HCTA with $\text{L}_{01} = 0.176$ , HB is approaching ( $\text{L}_{01} = 0.21$ ). The hierarchical Turner model HCTALO ( $\text{L}_{01} = 0.267$ ) performs clearly worse than both hierarchical beta models. According to MAE, both hierarchical beta models HB ( $\text {MAE} = 0.219$ ) and HSB ( $\text {MAE} = 0.185$ ) perform best. All non-hierarchical models, Turner and beta models, perform similarly ( $\text {MAE} \approx 0.25$ ), while the hierarchical Turner models perform worst, similarly to ULINOP and PAVG with $\text {MAE} \approx 0.4$ . Consistently over all performance measures, HSB outperforms HB. Also, the hierarchical beta models outperform the non-hierarchical beta models, while the non-hierarchical Turner models outperform the hierarchical Turner models.

Figure 4 Model performances on the Turner data set according to Brier score, 0–1 loss, and mean absolute error. We compare the scores’ means and standard errors of the mean of our beta fusion models, the Hierarchical Beta Fusion Model (HB), the non-hierarchical Beta Fusion Model (B), the Hierarchical Symmetric Beta Fusion Model (HSB), and the non-hierarchical Symmetric Beta Fusion Model (SB), the models by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), Average then Calibrate (ATC), Calibrate then Average (CTA), Calibrate then Average using log-odds (CTALO), Hierarchical Calibrate then Average (HCTA), and Hierarchical Calibrate then Average on log-odds (HCTALO), and the two baseline methods Unweighted Linear Opinion Pool (ULINOP) and Probit Average (PAVG).

In Figure 5, we compare the performances of beta and Turner fusion models on the newly introduced KTeC data set. Compared to the results on the Turner data set, the differences between the models’ performances are generally smaller over all three performance measures Brier score, 0–1 loss, and MAE. Based on the Brier score, HCTALO performs best with $\text {BS} = 0.125$ , however, CTALO, ATC, HSB, and PAVG perform quite similarly. HB and HCTA perform worst with Brier scores of $\text {BS} = 0.179$ and $\text {BS} = 0.185$ . However, as per 0–1 loss, CTA performs worst, and HSB and HCTALO are performing best with $\text{L}_{01} = 0.156$ and $\text{L}_{01} = 0.15$ . According to MAE, all beta fusion models clearly outperform the Turner models and perform quite comparably. Still, HSB is again the best performing beta fusion model with an MAE of 0.153.

Figure 5 Model performances on the Knowledge Test Confidence data set according to Brier score, 0–1 loss, and mean absolute error. We compare the scores’ means and standard errors of the mean of our beta fusion models, the Hierarchical Beta Fusion Model (HB), the non-hierarchical Beta Fusion Model (B), the Hierarchical Symmetric Beta Fusion Model (HSB), and the non-hierarchical Symmetric Beta Fusion Model (SB), the models by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), Average then Calibrate (ATC), Calibrate then Average (CTA), Calibrate then Average using log-odds (CTALO), Hierarchical Calibrate then Average (HCTA), and Hierarchical Calibrate then Average on log-odds (HCTALO), and the two baseline methods Unweighted Linear Opinion Pool (ULINOP) and Probit Average (PAVG).

In the evaluations shown so far, we combine the forecasts of 1,290 forecasters for the Turner data set and 85 forecasters for the KTeC data set. For both data sets, the Hierarchical Symmetric Beta Fusion model (HSB) is best or among the best models according to 0–1 loss or MAE. However, according to Brier score, some Turner models outperform HSB on both data sets, which might indicate that HSB or the beta models in general are overconfident. To investigate this, we also consider two subsets of the two data sets that contain fewer forecasters since fusing a lower number of forecasters should attenuate overconfidence. For the Turner data set, the subset of forecasters must be chosen with care, since all forecasters only provided forecasts for only a subset of queries. To be able to evaluate fusion methods, we must guarantee that all queries are answered by at least two forecasters, which we can then fuse. In order to do so, we selected the 20 forecasters providing the most forecasts. The results on the reduced Turner data set are shown in Figure 6. As for the full Turner data set, the hierarchical beta fusion models outperform the non-hierarchical ones, achieve a 0–1 loss ( $\text{L}_{01} = 0.171$ for HB and $\text{L}_{01} = 0.165$ for HSB) comparable to the best Turner model ATC ( $\text{L}_{01} = 0.176$ ), and outperform all other models clearly according to MAE with $\text {MAE} \approx 0.2$ . In addition, and in contrast to the full Turner data set, here HB ( $\text {BS} = 0.129$ ) and HSB ( $\text {BS} = 0.128$ ) also perform comparably to the best Turner model ATC ( $\text {BS} = 0.132$ ) regarding Brier score. Thus, for the reduced Turner data set, the hierarchical beta fusion models HB and HSB are among the best fusion models for all performance measures.

Figure 6 Model performances on the reduced Turner data set consisting of a subset of the 20 forecasters of the Turner data set that provided the most forecasts. We compare the means and standard errors of the mean of Brier scores, 0–1 losses, and mean absolute errors of our beta fusion models, the Hierarchical Beta Fusion Model (HB), the non-hierarchical Beta Fusion Model (B), the Hierarchical Symmetric Beta Fusion Model (HSB), and the non-hierarchical Symmetric Beta Fusion Model (SB), the models by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), Average then Calibrate (ATC), Calibrate then Average (CTA), Calibrate then Average using log-odds (CTALO), Hierarchical Calibrate then Average (HCTA), and Hierarchical Calibrate then Average on log-odds (HCTALO), and the two baseline methods Unweighted Linear Opinion Pool (ULINOP) and Probit Average (PAVG).

For the KTeC data set, it is more straightforward to create a reduced data set since all forecasters replied to all queries. Therefore, we simply selected the first 10 forecasters as a subset. In Figure 7, we see that similar to the reduced Turner data set, on the reduced KTeC data set, HSB is the best model according to all three performance measures ( $\text {BS} = 0.124, \text{L}_{01} = 0.15, \text {MAE} = 0.192$ ). As per Brier score and 0–1 loss, HCTALO ( $\text {BS} = 0.132, \text{L}_{01} = 0.156$ ) performs comparably, but regarding MAE, again all beta fusion models clearly outperform all Turner models. Similar to the full KTeC data set, the symmetric beta model (HSB) generally outperforms the asymmetric one (HB) and the hierarchical beta models achieve better scores than the non-hierarchical ones.

Figure 7 Model performances on the reduced Knowledge Test Confidence (KTeC) data set consisting of a subset of the first 10 forecasters of KTeC data set. We compare the means and standard errors of the mean of Brier scores, 0–1 losses, and mean absolute errors of our beta fusion models, the Hierarchical Beta Fusion Model (HB), the non-hierarchical Beta Fusion Model (B), the Hierarchical Symmetric Beta Fusion Model (HSB), and the non-hierarchical Symmetric Beta Fusion Model (SB), the models by Turner et al. (Reference Turner, Steyvers, Merkle, Budescu and Wallsten2014), Average then Calibrate (ATC), Calibrate then Average (CTA), Calibrate then Average using log-odds (CTALO), Hierarchical Calibrate then Average (HCTA), and Hierarchical Calibrate then Average on log-odds (HCTALO), and the two baseline methods Unweighted Linear Opinion Pool (ULINOP) and Probit Average (PAVG).

3.5. Beta calibration vs LLO

The results presented in Section 3.4 show that the Hierarchical Symmetric Beta Fusion Model (HSB) outperforms the Hierarchical Beta Fusion Model (HB). While this outcome is rather unexpected, since HSB constrains the modeling beta distributions to be symmetric and is therefore less expressive than HB, we can explain it with the calibration functions that are implied by modeling the forecasts with symmetric or asymmetric beta distributions. As shown in Section 2.3, by modeling the forecasts with beta distributions conditioned on the true label, we implicitly calibrate them using the beta calibration function (2.7). If the beta distributions are symmetric, the beta calibration function reduces to the LLO calibration function (2.8).

In most cases, the LLO and beta calibration function do not differ significantly. However, there are special cases, in which the beta calibration deviates drastically from the LLO calibration function. From (2.8), we can directly see that $\text {LLO}(0) = 0$ and $\text {LLO}(1) = 1$ if $\gamma = \beta - \alpha> 0$ with $\alpha = \alpha _0^k = \beta _1^k$ and $\beta = \beta _0^k = \alpha _1^k$ . The latter condition should hold for the most forecasters, since otherwise they would be biased toward always predicting the wrong answer.

In contrast, also for such unbiased forecasters, for which $\alpha _0^k<\beta _0^k$ and $\alpha _1^k>\beta _1^k$ , the beta calibration function BC(x) is not always defined at $x=0$ and $x=1$ , depending on the beta parameters. In particular, looking at (2.7), we see that

(3.4) $$ \begin{align} \lim_{x \rightarrow 0} \text{BC}(x) &= 1 \quad \text{if} \quad \alpha_1^k - \alpha_0^k < 0 \quad \text{and} \nonumber\\ \lim_{x \rightarrow 1} \text{BC}(x) &= 0 \quad \text{if} \quad \beta_0^k - \beta_1^k < 0. \end{align} $$

In Figure 8, we show the calibration curves of two exemplary forecasters from the KTeC data set (top row) together with the densities of the respective beta distributions for $t=0$ and $t=1$ when assuming asymmetric or symmetric beta distributions (bottom row). The respective parameters of the beta distributions that model their forecasts and define their calibration curves are taken from training split 1 of the cross-validation done for HB and HSB described in Section 3.2. Figure 8(a) shows the calibration curves and corresponding beta distributions of forecaster 57 with parameters $\alpha _0^{57} = 0.41, \beta _0^{57} = 0.65, \alpha _1^{57} = 0.67, \beta _1^{57} = 0.47$ for the asymmetric Hierarchical Beta Fusion Model (HB) or the beta calibration function, respectively, and parameters $\alpha _0^{57} = \beta _1^{57} = 0.44, \beta _0^{57} = \alpha _1^{57} = 0.65$ for the Hierarchical Symmetric Beta Fusion Model (HSB) and the LLO calibration function. We can see that the learned beta distributions for HB (asymmetric) and HSB (symmetric) look very similar. Also, LLO and beta calibration curves look very similar since $\alpha _1^{57} - \alpha _0^{57}> 0$ and $\beta _0^{57} - \beta _1^{57}> 0$ .

Figure 8 The empirical, beta calibration (BC), and LLO curves of two exemplary forecasters from the Knowledge Test Confidence data set (top row) together with the respective asymmetric and symmetric beta distributions (bottom row). (a) shows the calibration curves and respective beta distributions of forecaster 57, where LLO and beta calibration curves are similar. (b) shows the calibration curves and beta distributions of forecaster 46, for which the beta calibration function tends to 0 for $x\rightarrow 1$ causing miscalibrations.

In contrast, in Figure 8(b), we see that for forecaster 46, the beta calibration curve looks very different from the LLO curve. Since the forecaster is modeled with parameters $\alpha _0^{46} = 0.73,\ \beta _0^{46} = 1.16,\ \alpha _1^{46} = 2.81,\ \beta _1^{46} = 1.73$ for HB or beta calibration and $\alpha _0^{46} = \beta _1^{46} = 1.07, \beta _0^{46} = \alpha _1^{46} = 1.72$ for HSB or LLO, $\beta _0^{46} - \beta _1^{46} < 0$ and BC(x) tends to 0 for $x \rightarrow 1$ . This can also be seen in the corresponding beta distributions in the bottom of Figure 8(b). If we assume asymmetric beta distributions, at $x=1$ the probability density for $t=0$ is higher than the probability density for $t=1$ , leading to a beta calibration function that tends to 0 for $x \rightarrow 1$ . If we assume symmetric beta distributions, this does not happen. In this case, fusing with HB and thereby calibrating with beta calibration can induce miscalibration of forecasts close to 1 that lead to forecasting the opposite of the forecast that was originally provided. This can result in worse performance of the HB fusion model in comparison to the HSB fusion model, which we see in the results presented in Section 3.4.