Skip to main content Accessibility help
  • Get access
    Check if you have access via personal or institutional login
  • Cited by 6
  • Print publication year: 2005
  • Online publication date: February 2010

14 - Confidence in Aggregation of Opinions from Multiple Sources


To the memory of my friend and colleague, Janet A. Sniezek


This chapter summarizes research related to several interrelated questions regarding the process by which single decision makers (DMs) aggregate probabilistic information regarding a certain event from several, possibly asymmetric, advisors who rely on multiple and, possibly overlapping and correlated, sources of information. In particular I seek to understand and characterize (a) the nature of the aggregation rules used by DMs and (b) the factors that affect the DMs' confidence in the final aggregate.

The chapter starts with a short literature overview whose main goal is to set the stage by placing this work within the broad area of information aggregation. Next I present a descriptive model of confidence in information integration that is based on two principles: (a) People combine multiple sources of information by applying simple averaging rules; and (b) the DM's level of confidence in the aggregate is a monotonically decreasing function of its perceived variance. Some of the model's predictions regarding the structural and natural factors that affect the DM's confidence are discussed and tested with empirical data from four experiments (Budescu & Rantilla, 2000; Budescu et al., 2003). The results document the relation between the DM's confidence and the amount of information underlying the forecasts (number of advisors and cues), the advisors' accuracy, and the distribution of cues over judges with special attention to the level of interjudge overlap in information.

Related content

Powered by UNSILO
Anderson, N. H. (1981). Information integration theory. Hillsdale, NJ: Lawrence Erlbaum
Ariely, D., Au, W. T., Bender, R. H., Budescu, D. V., Dietz, C. B., Gu, H.Wallsten, T. S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147
Armstrong, J. S. (2001). Combining forecasts. In Armstrong, J. S. (Ed.), Principles of forecasting: A handbook for researchers and practitioners. Norwell, MA: Kluwer Academic Publishers
Arrow, K. (1963). Social choice and individual values (2nd edition). New Haven, CT: Yale University Press
Ashton, R. H. (1986). Combining the judgments of advisors: How many and which ones?Organizational Behavior and Human Decision Processes, 38, 405–414
Black, D. (1958). The theory of committees and elections. Cambridge, UK: Cambridge University Press
Bordley, R. F. (1982). A multiplicative formula for aggregating probability assessments. Management Science, 28, 1137–1148
Budescu, D. V. (2001). Aggregation of probabilistic forecasts and opinions. Presidential address at the annual meeting of the Society for Judgment and Decision Making (SJDM), Orlando, FL
Budescu, D. V., Erev, I., & Wallsten, T. S. (1997). On the importance of random error in the study of probability judgment, Part I: New theoretical developments. Journal of Behavioral Decision Making, 10, 157–172
Budescu, D. V., & Rantilla, A. K. (2000) Confidence in aggregation of expert opinions. Acta Psychologica, 104, 371–398
Budescu, D. V., Rantilla, A. K., Yu, H., & Karelitz, T. M. (2003). The effects of asymmetry among advisors on the aggregation of their opinions. Organizational Behavior and Human Decision Processes, 90, 178–194
Chapman, L. J., & Chapman, J. P. (1967). Genesis of popular but erroneous psychodiagnostic observations. Journal of Abnormal Psychology, 72, 193–204
Chapman, L. J., & Chapman, J. P. (1969). Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. Journal of Abnormal Psychology, 74, 271–280
Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583
Clemen, R. T., & Winkler, R. L. (1993). Aggregating point estimates: A flexible modeling approach. Management Science, 39, 501–515
Clemen, R. T., & Winkler, R. L. (1985). Limits for the precision and value of information from dependent sources. Operation Research, 33, 427–442
Clemen, R. T., & Winkler, R. L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46
Davis, J. H. (1973). Group decision and social interaction: A theory of social decision schemes. Psychological Review, 80, 97–125
Davis, J. H. (1996). Group decision making and quantitative judgments: A consensus model. In Witte, E. H. & Davis, J. H. (Eds.), Understanding group behavior (Vol 1): Small group processes and interpersonal relations. Hillsdale, NJ: Lawrence Erlbaum
Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85, 395–416
Erev, I., Wallsten, T. S., & Budescu, D. V. (1994). Simultaneous over- and underconfidence: The role of error in judgment processes. Psychological Review, 101, 519–527
Ferrel, W. R. (1985). Combining individual judgments. In Wright, G. (Ed.), Behavioral decision making (pp. 111–145). New York: Plenum
Fiedler, K. (1996). Explaining and simulating judgment biases as an aggregation phenomenon in probabilistic multiple-cue environments. Psychological Review, 103, 193–214
Fischer, I., & Harvey, N. (1999). Combining forecasts: What information do judges need to outperform the simple average?International Journal of Forecasting, 15, 227–246
Flores, B. E., & White, E. M. (1988). A framework for the combination of forecasts. Academy of Marketing Science, 16, 95–103
Gigerenzer, G., & Goldstein, D. G. (1999). Betting on one good reason: The take the best heuristic. In Gigerenzer, G., Todd, P. M., & the ABC Research Group (Eds.), Simple heuristics that make us smart (pp. 75–96). New York: Oxford University Press
Gigerenzer, G., Hoffrage, U., & Kleinbölting, H. (1991). Probabilistic mental models: A Brunswikian theory of confidence. Psychological Review, 98, 506–528
Harvey, N., & Harries, C. (2003). Effects of judges' forecasting on their later combination of forecasts for the same outcomes. International Journal of Forecasting, in press
Hastie, R. (1993). Inside the juror: The psychology of jury decision-making. Cambridge, UK: Cambridge University Press
Hogarth, R. M. (1977). Methods for aggregating opinions. In Jungermann, H. & Zeeuw, G. (Eds.), Decision making and change in human affairs (pp. 231–255). Boston: Riedel
Hogarth, R. M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46
Johnson, T. R., Budescu, D. V., & Wallsten, T. S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140
Juslin, P., Olsson, H., & Bjorkman, M. (1997). Brunswikian and Thurstonian origins of bias in probability assessment: On the interpretation of stochastic components of judgment. Journal of Behavioral Decision Making, 10, 189–210
Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 237–251
Kuhn, K. M., & Sniezek, J. A. (1996). Confidence and uncertainty in judgmental forecasting: Differential effects of scenario presentation. Journal of Behavioral Decision Making, 9, 231–247
Larrick, R. B., & Soll, J. B. (2003). Lay intuitions about combining quantitative judgments. Paper presented at the workshop on Information Aggregation in Decision Making, Silver Spring, MD
Laughlin, P. R., & Ellis, A. L. (1986). Demonstrability and social combination processes on mathematical intellective tasks, Journal of Experimental Social Psychology, 22, 177–189
Maines, L. (1996). An experimental examination of subjective forecast combination. International Journal of Forecasting, 12, 223–233
Mojzisch, A., & Schulz-Hardt, S. (2004). Information sampling in group decision making: Sampling biases and their consequences. This volume
Morris, P. A. (1983). An axiomatic approach to expert resolution. Management Science, 29, 24–32
Morris, P. A. (1986). Comment on Genest and Zideck's “Combining probability distributions: A critique and annotated bibliography.”Statistical Science, 1, 141–144
Payne, J. W., Bettman, J. R., & Johnson, E. R. (1993). The adaptive decision maker. New York: Cambridge University Press
Price, P. C., & Stone, E. R. (2004). Intuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17, 39–57
Rantilla, A. K., & Budescu, D. V. (1999). Aggregation of advisor opinions. In Proceedings of the 32nd Annual Hawai'i International Conference on Systems Sciences
Salvadori, L., Swol, L. M., & Sniezek, J. A. (2001). Information sampling and confidence within groups and judge advisor systems. Communication Research, 28, 737–771
Sniezek, J. A. (1986). The role of variable labels in cue probability learning tasks. Organizational Behavior and Human Decision Processes, 38, 141–161
Sniezek, J. A. (1992). Groups under uncertainty: An examination of confidence in group decision making. Organizational Behavior and Human Decision Processes, 52, 124–155
Sniezek, J. A., & Buckley, T. (1995). Cueing and cognitive conflict in judge-advisor decision making. Organizational Behavior and Human Decision Processes, 62, 159–174
Soll, J. B. (1999). Intuitive theories of information: Beliefs about the value of redundancy. Cognitive Psychology, 38, 317–346
Soll, J. B., Larrick, R. (2000). The 80/20 rule and the revision of judgment in light another's opinion: Why fo we believe ourseleves so much? Paper presented at the annual meeting of the Society for Judgment and Decision Making (SJDM), Los Angeles
Trafimow, D., & Sniezek, J. A. (1994). Perceived expertise and its effect on confidence. Organizational Behavior and Human Decision Processes, 57, 290–302
van Swol, L. M., & Sniezek, J. A. (2002). Trust me, I'm an expert: Trust and confidence and acceptance of expert advice. Paper presented at the 8th Conference on Behavioral Decision Research in Management (BDRM), Chicago
Wallsten, T. S., Budescu, D. V., Erev, I., & Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–268
Wallsten, T. S., & Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18
Walz, D. T., & Walz, D. D. (1989). Combining forecast: Multiple regression versus a Bayesian approach. Decision Sciences, 20, 77–89
Winkler, R. L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685
Winkler, R. L., & Makridakis, S. (1983). The combination of forecasts. Journal of the Royal Statistical Society: Series A, 146, 150–157
Winkler, R. L., & Poses, R. M. (1993). Evaluating and combining physician's probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543
Wittenbaum, G., & Stasser, G. (1996). Management of information in small groups. In Nye, J. L. & Brower, A. M. (Eds.), What's social about social cognition? Research on socially shared cognition in small groups. (pp. 3–28). Thousand Oaks, CA: Sage
Yaniv, I. (1997). Weighting and trimming: Heuristics for aggregating judgments under uncertainty. Organizational Behavior and Human Decision Processes, 69, 237–249
Yaniv, I. (2004). Receiving other people's advice: Influence and benefit. Organizational Behavior and Human Decision Processes, 93, 1–13
Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision-making: Egocentric discounting and reputation formation. Organizational Behavior and Human Decision Processes, 83, 260–281