References
Published online by Cambridge University Press: 05 November 2015
Summary
A summary is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
- Type
- Chapter
- Information
- Trusting JudgementsHow to Get the Best out of Experts, pp. 176 - 200Publisher: Cambridge University PressPrint publication year: 2015
References
2009. A Kullback-Leibler view of linear and log-linear pools. Decision Analysis 6, 25–37.CrossRefGoogle Scholar
1995. Indigenous and scientific knowledge: some critical comments. Indigenous Knowledge and Development Monitor 3, 3–6.Google Scholar
, , , , and 2012. Combining expert opinions in prior elicitation. Bayesian Analysis 7, 503–532.CrossRefGoogle Scholar
and 2007. Age differences in the value of statistical life: revealed preference evidence. Review of Environmental Economics and Policy 1, 241–260.CrossRefGoogle Scholar
ALRC. 1985. Evidence. Australian Law Reform Commission Report 26. Volume 1. Australian Government Publishing Service, Canberra.
ALRC. 2000. Managing justice: a review of the Federal Civil Justice system. Australian Law Reform Commission Report 89. Australian Government Publishing Service, Canberra.
1998. Embracing uncertainty: the interface of Bayesian statistics and cognitive psychology. Conservation Ecology 2, 2.CrossRefGoogle Scholar
2012. The half-life of facts. Why everything we know has an expiration date. Penguin, New York.Google Scholar
1980. The seer-sucker theory: the value of experts in forecasting. Technology Review 82, 16–24.Google Scholar
2001. Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Dordrecht.CrossRefGoogle Scholar
2006. Findings from evidence-based forecasting: methods for reducing forecast error. International Journal of Forecasting 22, 583–598.CrossRefGoogle Scholar
2000. A review and analysis of research on the test–retest reliability of professional judgment. Journal of Behavioral Decision Making 13, 277–294.3.0.CO;2-B>CrossRefGoogle Scholar
and 2011. The value of feedback in forecasting competitions. International Journal of Forecasting 27, 845–849.Google Scholar
2001. Elicitation of expert opinions for uncertainty and risks. CRC Press, Boca Raton, Florida.CrossRefGoogle Scholar
, and 2009. Judges, courts, the legal profession and public risk. Risk and Regulation Advisory Council, UK.
1980. The base-rate fallacy in probability judgments. Acta Psychologica 44, 211–233.CrossRefGoogle Scholar
and 2001. Boys will be boys: Gender, overconfidence, and common stock investment. Quarterly Journal of Economics 116, 261–292.CrossRefGoogle Scholar
and 2006. Contracting: a new form of professional practice. Academy of Management Perspectives 20, 45–66.CrossRefGoogle Scholar
and 2001. Probabilistic risk analysis: foundations and methods. Cambridge University Press.CrossRefGoogle Scholar
, and 2006. Expert elicitation for reliable system design. Statistical Science 21, 428–450.Google Scholar
and 1998. On the use of demographic models of population viability in endangered species management. Journal of Wildlife Management 62, 821–841.CrossRefGoogle Scholar
, , , , , , and 2002. How to avoid train wrecks when using science in environmental problem solving. Bioscience 52, 1127–1136.CrossRefGoogle Scholar
2006. Information-gap decision theory: decisions under severe uncertainty. Academic Press, San Diego.Google Scholar
and 1982. An investigation of motivational bias in subjective predictive probability distributions. Decision Sciences 13, 225–239.CrossRefGoogle Scholar
and 2008. Overconfidence as a cause of diagnostic error in medicine. American Journal of Medicine 121, S2–S23.CrossRefGoogle ScholarPubMed
, , and 1985. An elementary approach to thinking under uncertainty. Lawrence Erlbaum Associates, Hillside, New Jersey.Google Scholar
and 1999a. Chinese and Americans agree on what is fair, but disagree on what is best in societal decisions affecting health and safety risks. Risk Analysis 19, 439–452.CrossRefGoogle ScholarPubMed
and 1999b. Patterns of fairness judgments in North America and the People's Republic of China. Journal of Consumer Psychology 8, 301–320.CrossRefGoogle Scholar
2004. Implications of the research on expert overconfidence and dependence. Reliability Engineering and System Safety 85, 321–329.CrossRefGoogle Scholar
, , , and 1999. A survey of approaches for assessing and managing the risk of extremes. Risk Analysis 19, 83–94.CrossRefGoogle Scholar
, , and 2011. Does the Delphi process lead to increased accuracy in group-based judgmental forecasts or does it simply induce consensus amongst judgmental forecasters?Technological Forecasting and Social Change 78, 1671–1680.CrossRefGoogle Scholar
, and 2003. An integrated approach to TMDL development for the Neuse River Estuary using a Bayesian probability network model (Neu-BERN).Journal of Water Resources Planning and Management 129, 271–282.CrossRefGoogle Scholar
, and 2002. Propagation of individual bias through group judgment: error in the treatment of asymmetrically informative signals. Journal of Risk and Uncertainty 25, 147–163.CrossRefGoogle Scholar
1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1–3.2.0.CO;2>CrossRefGoogle Scholar
2005. Risks and decisions for conservation and environmental management. Cambridge University Press.CrossRefGoogle Scholar
, , , , and 2000. Inferring threat from scientific collections: power tests and application to Western Australian Acacia species. In and , editors. Quantitative methods for conservation biology. Springer-Verlag, New York.Google Scholar
, , , , , , , and 2011a. Expert status and performance. PLoS ONE 6, e22998.CrossRefGoogle ScholarPubMed
, , , , , and 2011b. Redefining expertise and improving ecological judgement. Conservation Letters 4, 81–87.CrossRefGoogle Scholar
and 1991. The process-performance paradox in expert judgment: how can experts know so much and predict so badly? In and , editors. Towards a general theory of expertise: prospects and limits. Cambridge University Press, New York.Google Scholar
2002. Science and sustainable use: views of marine turtle conservation experts. Ecological Applications 12, 1229–1246.CrossRefGoogle Scholar
1976. The difficulty of assessing uncertainty. Journal of Petroleum Technology 28, 843–850.Google Scholar
2009. Strategies to improve the communication of probability information in risk analyses. International Journal of Risk Assessment and Management 12, 380–395.CrossRefGoogle Scholar
, , and 2001. Risk based characterization of contaminated industrial site using multivariate and geostatistical tools. Environmental Pollution 111, 417–427.CrossRefGoogle Scholar
, and 1999. Ecological and social dynamics in simple models of ecosystem management. Social Systems Research Institute, University of Wisconsin.
, , and 1993. Graphical elicitation of a prior distribution for a clinical trial. The Statistician 42, 341–353.Google Scholar
2003. Statistical thought: a perspective and history. Oxford University Press.CrossRefGoogle Scholar
and 2002. Information aggregation mechanisms: concepts, design and implementation for a sales forecasting problem. Social Science Working Paper 1131, California Institute of Technology.
1978. Predator–prey theory and variability. Annual Review of Ecology and Systematics 9, 323–347.CrossRefGoogle Scholar
2006. Two approaches to the study of experts’ characteristics. In , , and , editors. The Cambridge handbook of expertise and expert performance. Cambridge University Press.Google Scholar
and 1981. Physicians use of probabilistic information in a real clinical setting. Journal of Experimental Psychology: Human Perception and Performance 7, 928–935.Google Scholar
and 1999. Combining probability distributions from experts in risk analysis. Risk Analysis 19, 187–203.CrossRefGoogle Scholar
and 1978. Prediction in psychiatry: an example of misplaced confidence in experts. Social Problems 25, 265–276.CrossRefGoogle Scholar
2001. Is probability the only coherent approach to uncertainty?Risk Analysis 28, 645–652.Google Scholar
1991. Experts in uncertainty: opinion and subjective probability in science. Oxford University Press.Google Scholar
, and 2008. On the performance of social network and likelihood based expert weighting schemes. Reliability Engineering and System Safety, Special Issue 93, 745–756.CrossRefGoogle Scholar
and 2000. Procedures guide for structured expert judgement in accident consequence modelling. Radiation Protection and Dosimetry 90, 303–309.CrossRefGoogle Scholar
and 2000. Processing expert judgements in accident consequence modelling. Radiation Protection Dosimetry 90, 311–315.CrossRefGoogle Scholar
2004. On the use of imprecise probabilities in reliability. Quality and Reliability Engineering International 20, 193–202.CrossRefGoogle Scholar
and 2011. The Polymath Projects: lessons from an experiment in large-scale online collaboration in mathematics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1865–1874. ACM.
1999. Risk-based environmental decisions: method and culture. Kluwer Academic Publishers, Boston.CrossRefGoogle Scholar
, , , , , 2006. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Journal of the American Medical Association 296, 1094–1102.Google ScholarPubMed
1979. The robust beauty of improper linear models in decision making. American Psychologist 34, 571–582.CrossRefGoogle Scholar
1994. House of cards: psychology and psychotherapy built on myth. Simon and Schuster, New York.Google Scholar
, , , , , , , , and 1995. Coherent combination of experts’ opinions. Test 4, 263–313.CrossRefGoogle Scholar
, , and 2003. Why people fail to recognize their own incompetence. Current Directions in Psychological Science 12, 83–87.Google Scholar
, and 2010. The effects of presenting imprecise probabilities in intelligence forecasts. Risk Analysis 30, 987–1001.CrossRefGoogle ScholarPubMed
1999. Hazard identification of agricultural biotechnology: finding relevant questions. International Books, Utrecht.Google Scholar
, , and 2010. Using expert opinion surveys to rank threats to endangered species: a case study with sea turtles. Conservation Biology 24, 1586–1595.Google Scholar
, , and 2005. When emotions get the better of us: the effect of contextual top-down processing on matching fingerprints. Applied Cognitive Psychology 19, 799–809.CrossRefGoogle Scholar
and 2008. Meta-analytically quantifying the reliability and biasability of fingerprint experts’ decision making. Journal of Forensic Sciences 53, 900–903.CrossRefGoogle Scholar
and 1976. An actual application of collective choice theory to the selection of trajectories for the Mariner Jupiter/Saturn 1977 project. Operations Research 24, 220–244.CrossRefGoogle Scholar
and 1978. Confidence in judgment: persistence of the illusion of validity. Psychological Review 85, 395–416.CrossRefGoogle Scholar
, , and 2010. A Delphi method expert survey to derive standards for flood damage data collection. Risk Analysis 30, 107–124.CrossRefGoogle ScholarPubMed
1995. Clinical reasoning in medicine. In: , editor. Clinical Reasoning in the Health Professions. Butterworth-Heinemann Limited, Oxford, England.Google Scholar
and 1986. Patterns of controversy and closure: the interplay of knowledge, values, and political forces. In and , editors. Scientific controversies: case studies in the resolution and closure of disputes in science and technology. Cambridge University Press, New York.Google Scholar
, , and 2009. Expert financial advice neurobiologically “offloads” financial decision-making under risk. PLoS ONE 4, e4957.CrossRefGoogle ScholarPubMed
and 2009. Moody experts – how mood and expertise influence judgmental anchoring. Judgment and Decision Making 4, 41–50.Google Scholar
, , and , editors. 2006. The Cambridge Handbook of Expertise and Expert Performance. Cambridge University Press, New York.CrossRefGoogle Scholar
European Food Safety Authority 2011. Guidance on expert knowledge elicitation in food and feed safety risk assessment. European Food Safety Authority (EFSA), Parma, Italy.
2008. Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology 59, 255–278.CrossRefGoogle ScholarPubMed
, and 2006. Professionalization, scientific expertise, and elitism: a sociological perspective. In , , and , editors. The Cambridge handbook of expertise and expert performance. Cambridge University Press.Google Scholar
1990. The descent of Icarus: science and the transformation of contemporary democracy. Harvard University Press, Cambridge, Massachusetts.Google Scholar
, and 2007. Integrating science and local knowledge in environmental risk management: a decision-focused approach. Ecological Economics 64, 47–70.CrossRefGoogle Scholar
FAO / WHO. 2001. Call for experts for the Joint FAO / WHO risk assessment activities in the areas of Campylobacter in broilers and Vibrio in seafood. Joint Expert Consultations on Risk Assessment of Microbiological Hazards in Food. Food and Nutrition Division, Food and Agriculture Organization of the United Nations, Rome, Italy.
, , , and 2003. Constructing probability boxes and Dempster–Shafer structures. SAND Report, SAND2002-4015. Sandia National Laboratories, Albuquerque, New Mexico.
1986. Appendix F – Personal observations on the reliability of the Shuttle. In Hearing on the Space Shuttle Accident and the Rogers Commission Report. Report of the Presidential Commission on the Space Shuttle Challenger Accident. U.S. Senate Committee on Commerce, Science and Transportation, Subcommittee on Science, Technology and Space, Washington, DC.
1979. Subjective information and market efficiency in a betting market. Journal of Political Economy 87, 75–88.CrossRefGoogle Scholar
1994. Acceptable risk: a conceptual proposal. Risk: Health, Safety and Environment 1, 1–28.Google Scholar
1995. Risk perception and communication unplugged: twenty years of progress. Risk Analysis 15, 137–145.CrossRefGoogle Scholar
, and 1982. Lay foibles and expert fables in judgements about risk. American Statistician 36, 240–255.Google Scholar
1998. Environmental science and environmental law. Journal of Environmental Law 10, 3–8.CrossRefGoogle Scholar
., , and 2012. Facilitated expert judgment of environmental risks: acquiring and analysing imprecise data. International Journal of Risk Assessment and Management 16, 199–212.CrossRefGoogle Scholar
2005. The quality of expert judgment: an interdisciplinary investigation. PhD Thesis. Department of Mechanical Engineering, University of Maryland. URL: http://hdl.handle.net/1903/3267.
, , and 2008. Evaluating extreme risks in invasion ecology: learning from banking compliance. Diversity and Distributions 14, 581–591.CrossRefGoogle Scholar
1995. The challenge of junk psychiatry, psychology and science: the evolving role of the forensic expert. In , editor. Tomorrow's law. Federation Press, Sydney.Google Scholar
2011. Aggregating expert judgement. Revista De La Real Academia De Ciencias Exactas Fisicas Y Naturales Serie a-Matematicas 105, 181–206.
2012. Expert judgment, meta-analysis and participatory risk analysis. Decision Analysis 9, 119–127.CrossRefGoogle Scholar
1992. Heuristics, biases, and the not-so-general publics: expertise and error in the assessment of risks. In and , editors. Social theories of risk. Praeger Publishing, Westport, Connecticut.Google Scholar
1996. Risky thinking: irrational fears about risk and society. Annals of the American Academy of Political and Social Science 545, 44–53.CrossRefGoogle Scholar
1999. Tools for understanding the socioeconomic and political settings for environmental decision making. In and , editors. Tools to aid environmental decision making. Springer, New York.Google Scholar
, , and 1996. Media coverage of hazard events: analyzing the assumptions. Risk Analysis 16, 31–42.CrossRefGoogle Scholar
2009. System and estimator variables in eyewitness identification: a review (Chapter 3). In and , editors. Psychological expertise in court. Ashgate Publishing, Farnham.Google Scholar
2007. Subjective probability distribution elicitation in cost risk analysis. United States Airforce Report, RAND Corporation, Santa Monica.
, and 2005. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association 100, 680–700.CrossRefGoogle Scholar
2002. Calculated risks: how to know when numbers deceive you. Simon and Schuster, New York.Google Scholar
, , and 2005. A 30% chance of rain tomorrow: How does the public understand probabilistic weather forecasts?Risk Analysis 25, 623–629.CrossRefGoogle ScholarPubMed
, and the ABC Research Group. 1999. Simple heuristics that make us smart. Oxford University Press, New York.Google Scholar
and 1997. Proper analysis of the accuracy of group judgments. Psychological Bulletin 121, 149–167.CrossRefGoogle Scholar
, , and 2002. Heuristics and biases: the psychology of intuitive judgement. Cambridge University Press.CrossRefGoogle Scholar
and . 2004. Neuroeconomics: the consilience of brain and decision. Science 306, 447–452.CrossRefGoogle ScholarPubMed
1965. Diagnosticians vs. diagnostic signs: the diagnosis of psychosis vs. neurosis from the MMPI. Psychological Monographs 79 (Whole No. 602).CrossRefGoogle Scholar
and 2010. The limits of forecasting methods in anticipating rare events. Technological Forecasting and Social Change 77, 355–368.CrossRefGoogle Scholar
and 2001. Expert judgement elicitation in risk assessment. In and , editors. Assessment and management of environmental risks. Kluwer Academic Publishers, Dordrecht.Google Scholar
1924. Group judgments in the field of lifted weights. Journal of Experimental Psychology 7, 398–400.CrossRefGoogle Scholar
and 2009. Massively collaborative mathematics: the Polymath Project. Nature 461, 879–881.CrossRefGoogle Scholar
and 2011. Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. International Journal of Forecasting 27, 183–195.CrossRefGoogle Scholar
and 1996. Refining the CRA framework. In , editor. Comparing environmental risks. Resources for the Future, Washington, DC.Google Scholar
, , and 2006. Some pitfalls of an overemphasis on science in environmental risk management decisions. Journal of Risk Research 9, 717–735.CrossRefGoogle Scholar
, and 1993. The role of past states in determining reference points for policy decisions. Organisational Behavior and Human Decision Processes 55, 195–206.CrossRefGoogle Scholar
and 1998. Science in public: communication, culture, and credibility. Plenum Press, New York.Google Scholar
and 2007. Global warming: forecasts by scientists versus scientific forecasting. Energy and Environment 18, 997–1021.CrossRefGoogle Scholar
1958. Experience of clinicians and speech characteristics and statements of clients as variables in clinical judgment. Journal of Consulting Psychology 22, 315–319.CrossRefGoogle ScholarPubMed
and 1996. Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: the clinical-statistical controversy. Psychology, Public Policy, and Law 2, 293–323.CrossRefGoogle Scholar
, , , and 2000. Clinical vs. mechanical prediction: a meta-analysis. Psychological Assessment 12, 19–30.CrossRefGoogle Scholar
2000. The precautionary principle in Australia: policy, law and potential precautionary EIAs. Risk: Health, Safety and Environment 11, 93–124.Google Scholar
, , and 1973. A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance 9, 280–291.CrossRefGoogle Scholar
, , and 2013. Integrating expert judgment in veterinary epidemiology: example guidance for disease freedom surveillance. Preventive Veterinary Medicine 109, 1–9.CrossRefGoogle ScholarPubMed
2001. An epistemologist in the bramble-bush: at the Supreme Court with Mr. Joiner. Journal of Health Politics Policy and Law 26, 217–248.CrossRefGoogle ScholarPubMed
1975. The emergence of probability: a philosophical study of early ideas about probability, induction and statistical inference. Cambridge University Press, London.Google Scholar
, and 2000. Traffic accident statistics and risk perceptions in Japan and the United States. Accident Analysis and Prevention 32, 827–835.CrossRefGoogle ScholarPubMed
and 1986. Assessing uncertainty in physical constants. American Journal of Physics 54, 791–798.CrossRefGoogle Scholar
and 2009. The wisdom of many in one mind. Psychological Science 20, 231–237.CrossRefGoogle ScholarPubMed
1987. Living with uncertainty in resource management. North American Journal of Fisheries Management 7, 1–5.2.0.CO;2>CrossRefGoogle Scholar
and 1997. The ecological detective: confronting models with data. Monographs in Population Biology 28. Princeton University Press, New Jersey.Google Scholar
1990. The dominant view of popularization: conceptual problems, political uses. Social Studies of Science 20, 519–539.CrossRefGoogle Scholar
, , and 2008. Informing risk-mitigation priorities using uncertainty measures derived from heterogeneous expert panels: a demonstration using foodborne pathogens. Reliability Engineering and System Safety 93, 687–698.CrossRefGoogle Scholar
, , and 2002. Representation facilitates reasoning: what natural frequencies are and what they are not. Cognition 84, 343–352.CrossRefGoogle ScholarPubMed
2005. Deciding analytically or trusting your intuition? The advantages and disadvantages of analytic and intuitive thought. In and , editors. The routines of decision making. Lawrence Erlbaum Associates, Mahwah, New Jersey.Google Scholar
and 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proceedings of the National Academy of Sciences of the USA 101, 16385–16389.CrossRefGoogle ScholarPubMed
, and 2011. The impact of gender diversity on the performance of business teams: evidence from a field experiment. Tinbergen Institute Discussion Paper TI 2011–074/3. Amsterdam School of Economics, University of Amsterdam.
1992. Acquisition of expert judgment: examples from risk assessment. Journal of Energy Engineering 118, 136–148.CrossRefGoogle Scholar
2007. Eliciting probabilities from experts (Chapter 8). In , and , editors. Advances in decision analysis. Cambridge University Press.Google Scholar
, , and 1998. Introduction of contagious animal diseases into the Netherlands: elicitation of expert opinions. Livestock Production Science 53, 253–264.CrossRefGoogle Scholar
1993. The Daubert decision: Frye is dead: long live the Federal Rules of Evidence. Trial 29, 60–65.Google Scholar
International Monetary Fund. 2006. Global Financial Stability Report. World Economic and Financial Surveys, Washington.
2005a. Contradicted and initially stronger effects in highly cited clinical research. Journal of the American Medical Association 294, 218–228.Google ScholarPubMed
2005b. Why most published research findings are false. PLoS Medicine 2, e124.CrossRefGoogle ScholarPubMed
2008. Why most discovered true associations are inflated. Epidemiology. 19, 640–648.CrossRefGoogle ScholarPubMed
2001. Constructing the scientific citizen: science and democracy in the biosciences. Public Understanding of Science 10, 1–18.CrossRefGoogle Scholar
2011. How to build the global mathematics brain. New Scientist issue 2811, 4 May 2011, 10–11.
, and 2010. Elicitator: an expert elicitation tool for regression in ecology. Environmental Modelling and Software 25, 129–145.CrossRefGoogle Scholar
, , , and 2006. Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Quality and Safety in Health Care 15, 433–436.CrossRefGoogle ScholarPubMed
Groupthink: Psychological studies of policy decisions and fiascos. Houghton Mifflin, Boston.
2006. Transparency in public science: purposes, reasons, limits. Law and Contemporary Problems 69, 21.Google Scholar
1976. Confidence intervals vs Bayesian intervals. In and , editors. Foundations of probability theory, statistical inference, and statistical theories of science. Volume 2. D. Reidel Publishing, Dordrecht, Holland.Google Scholar
2002. On the definition and age-dependency of the value of a statistical life. Journal of Risk and Uncertainty 25, 251–263.CrossRefGoogle Scholar
and 2001. Calibration of subjective probability judgments in a naturalistic setting. Organizational Behavior and Human Decision Processes 85, 265–290.CrossRefGoogle Scholar
, , , and 1980. Interactive elicitation of opinion for a normal linear model. Journal of the American Statistical Association 75, 815–885.CrossRefGoogle Scholar
2013. Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision Making 8, 407–424.Google Scholar
2015. Climate science communication and the measurement problem. Advances in Political Psychology 36, 1–43.CrossRefGoogle Scholar
and 1979. Prospect theory: an analysis of decision under risk. Econometrica 47, 263–291.CrossRefGoogle Scholar
2002. Software glitch threw off mortality estimates. Science 296, 1945–1946.CrossRefGoogle ScholarPubMed
, , and 2002. Race, gender and environmentalism: the atypical values and beliefs of white men. Race, Gender and Class 9, 1–19.Google Scholar
and 1999. Should we risk it? Exploring environmental, health, and technological problem solving. Princeton University Press, New Jersey.Google Scholar
1992. ‘Expert opinion’ versus ‘expert opinions.’ Another approach to the problem of eliciting/combining/using expert opinion in PRA. Reliability Engineering and System Safety 35, 61–72.CrossRefGoogle Scholar
2006. When should consumers and managers trust their intuition?Journal of Consumer Psychology 16, 20–24.CrossRefGoogle Scholar
and 1991. Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management 38, 191–201.CrossRefGoogle Scholar
1996. A new way to ask the experts: rating radioactive waste risks. Science 274, 913–914.CrossRefGoogle Scholar
and 2004. Group performance and decision making.Annual Review of Psychology 55, 623–655.CrossRefGoogle ScholarPubMed
, , Foldit Contenders Group, Foldit Void Crushers Group, , , , , , , , , and 2011. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nature Structural and Molecular Biology 18, 1175–1177.CrossRefGoogle ScholarPubMed
1995. Daubert v. Merrell Dow Pharmaceuticals: active judicial scrutiny of scientific evidence. Food and Drug Law Journal 50, 213–234.Google Scholar
, , and 2006. Subjective confidence and the sampling of knowledge. In K. Fiedler and P. Juslin, editors. Information sampling and adaptive cognition. Cambridge University Press, New York, pp. 153–182.Google Scholar
and 1997. Types and measures of uncertainty. In , and , editors. Consensus under fuzziness. Kluwer Academic Publishers, Boston.Google Scholar
and 1998. Uncertainty-based information: elements of generalized information theory. Physica-Verlag, Heidelberg.Google Scholar
, and 2007. Uncertainty communication issues and good practice. Copernicus Institute for Sustainable Development and Innovation. Report NWS-E-2007–199. Netherlands Environmental Assessment Agency, Utrecht.
, , and 2010. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environmental Health 9, 19.CrossRefGoogle ScholarPubMed
, , , and 2012. Crowd IQ: measuring the intelligence of crowdsourcing platforms. In Proceedings of the 3rd Annual ACM Web Science Conference, 151–160. ACM.
1993. Earthquake probability in engineering – Part 1: the use and misuse of expert opinion. Engineering Geology 33, 257–288.CrossRefGoogle Scholar
and 2009. Unskilled and unaware: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Psychology 1, 30–46.Google Scholar
1990. The case for motivated reasoning. Psychological Bulletin 108, 480–498.CrossRefGoogle ScholarPubMed
and 1986. Demonstrability and social combination processes on mathematical intellective tasks. Journal of Experimental Social Psychology 22, 177–189.CrossRefGoogle Scholar
, , and 2006. Judgmental forecasting: a review of progress over the last 25 years. International Journal of Forecasting 22, 493–518.CrossRefGoogle Scholar
2003. Amateurs: a 21st-century remake – social commentator Charles Leadbeater looks at the growing number of people in Britain taking a serious approach to pastime pursuits. RSA Journal 150, 22–25.Google Scholar
2012. Groupthink: the brainstorming myth. The New Yorker 30, 12.
, , and 2006. A general, analytic method for generating robust strategies and narrative scenarios. Management Science 52, 514–528.CrossRefGoogle Scholar
and 2000. Knowledge partitioning: context-dependent use of expertise. Memory and Cognition 28, 295–305.CrossRefGoogle Scholar
, and 1987. Member variation, recognition of expertise, and group performance. Journal of Applied Psychology 72, 81–87.CrossRefGoogle Scholar
, and 1982. Calibration of probabilities: the state of the art to 1980. In , and , editors. Judgment under uncertainty: heuristics and biases. Cambridge University Press, New York.Google Scholar
and 2008. A study of expert overconfidence. Reliability Engineering and System Safety 93, 711–721.CrossRefGoogle Scholar
, , , , , , and 2002. Comparison of treadmill scores with physician estimates of diagnosis and prognosis in patients with coronary artery disease. American Heart Journal 143, 650–658.CrossRefGoogle ScholarPubMed
, and 2009. Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. Ecology 90, 265–277.Google Scholar
1950. Implicit hypotheses and clinical predictions. Journal of Abnormal and Social Psychology 5, 756–760.Google Scholar
2008. Collective problem solving: are the many smarter than the few?Durham Anthropology Journal 15, 23–58.Google Scholar
, and 2008. Explosion probability of unexploded ordnance: expert beliefs. Risk Analysis 28, 825–841.Google ScholarPubMed
2001. Decomposition for judgmental forecasting and estimation. In , editor. Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Boston.Google Scholar
and 2006. The Delphi process – an expert-based approach to ecological modelling in data-poor environments. Animal Conservation 9, 11–19.CrossRefGoogle Scholar
and 2005. Can behavioral decision theory explain risk-averse fire management decisions?Forest Ecology and Management 211, 47–58.CrossRefGoogle Scholar
, , , and 2012. An expert panel approach to assessing potential effects of bull trout reintroduction on federally listed salmonids in the Clackamas River, Oregon. North American Journal of Fisheries Management 32, 450–465.CrossRefGoogle Scholar
, , and 2004. Competing approaches to predicting supreme court decision making. Perspectives on Politics 2, 761–767.CrossRefGoogle Scholar
, , , , , , , , , , , and 2010. Guidance note for lead authors of the IPCC Fifth Assessment Report on consistent treatment of uncertainties. Intergovernmental Panel on Climate Change.
, , , , , , , , , , and 2012a. Structured elicitation of expert judgments for threatened species assessment: a case study on a continental scale using email. Methods in Ecology and Evolution 3, 906–920.CrossRefGoogle Scholar
, and 2012b. Evaluating the accuracy and calibration of expert predictions under uncertainty: predicting the outcomes of ecological research. Diversity and Distributions 18, 782–794.CrossRefGoogle Scholar
1954. Clinical versus statistical prediction: a theoretical analysis and review of the evidence. University of Minnesota Press, Minneapolis.CrossRefGoogle Scholar
1986. Causes and effects of my disturbing little book. Journal of Personality Assessment 50, 370–375.CrossRefGoogle ScholarPubMed
2001. Using comparative risk to set pollution prevention priorities in New York State: a formula for inaction: a critical analysis of the New York State Department of Environmental Conservation Comparative Risk Project. A report from the New York State Assembly Legislative Commission on Toxic Substances and Hazardous Wastes. Assemblyman Steve Englebright, Chair. New York.
, , et al. 2015. Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspectives on Psychological Science 10, 267–281.CrossRefGoogle ScholarPubMed
and 1990. Eliciting and analyzing expert judgment: a practical guide. Office of Nuclear Regulatory Research, Division of Systems Research, US Nuclear Regulatory Commission, Washington, DC.
2013. When is consensus knowledge based? Distinguishing shared knowledge from mere agreement. Synthese, 190, 1293–1316.CrossRefGoogle Scholar
and 2008. The trouble with overconfidence. Psychological Review 115, 502–517.CrossRefGoogle ScholarPubMed
, and 2010. Conflict of interest and the intrusion of bias. Judgment and Decision Making 5, 37–53.Google Scholar
2014. Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Sciences of the USA 111, 7176–7184.CrossRefGoogle Scholar
, , and 1996. A proposal for ranking risk within Federal agencies. In , editor. Comparing environmental risks. Resources for the Future, Washington, DC.Google Scholar
and 1990. Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press.CrossRefGoogle Scholar
, and 2012. The market for financial advice: an audit study. National Bureau of Economic Research Working Paper No. 17929. Cambridge, Massachusetts.
2004. Whom to trust? Public concerns, late modern risks, and expert trustworthiness. Journal of Agricultural and Environmental Ethics 17, 113–130.CrossRefGoogle Scholar
1993. What is a good forecast? An essay on the nature of goodness in weather forecasting. American Meterological Society 8, 281–293.Google Scholar
and 1977. Can weather forecasters formulate reliable probability forecasts of precipitation and temperature?National Weather Digest 2, 2–9.Google Scholar
and 1984. Probability forecasting in meteorology. Journal of the American Statistical Association 79, 489–500.Google Scholar
, , , , and 2009. How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rockwallabiesPetrogale penicillata. Journal of Applied Ecology 46, 842–851.Google Scholar
and 2013. Science, policy, and the public discourse of shark ‘attack’: a proposal for reclassifying human-shark interactions. Journal of Environmental Studies and Sciences 3, 65–73.CrossRefGoogle Scholar
, , and 2001. Improving decision making by means of dissent. Journal of Applied Social Psychology 31, 48–58.CrossRefGoogle Scholar
2000. Making better environmental decisions: an alternative to risk assessment. MIT Press, Cambridge, Massachusetts.Google Scholar
2012. Probabilistic uncertainty specification: overview, elaboration techniques and their application to a mechanistic model of carbon flux. Environmental Modelling and Software 36, 35–48.Google Scholar
, , , , , , and 2006. Uncertain judgements: eliciting experts’ probabilities. Wiley, Hoboken, New Jersey.CrossRefGoogle Scholar
and 2004. Probability is perfect, but we can't elicit it perfectly. Reliability Engineering and System Safety 85, 239–248.CrossRefGoogle Scholar
, , , , , and 2009. Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush-tailed rock-wallabyPetrogale penicillata. Environmetrics 20, 379–398.Google Scholar
, , , and 2008. Using expert knowledge to assess uncertainties in future polar bear populations under climate change. Journal of Applied Ecology 45, 1649–1659.Google Scholar
, , and 2003. Professional vs. amateur judgment accuracy: the case of foreign exchange rates. Organizational Behavior and Human Decision Processes 91, 169–185.CrossRefGoogle Scholar
, and 2013. Impact of plain packaging of tobacco products on smoking in adults and children: an elicitation of international experts’ estimates. BMC Public Health 13, 18.CrossRefGoogle Scholar
2005. The precautionary principle in practice: environmental decision-making and scientific uncertainty. Federation Press, Annandale, New South Wales.Google Scholar
2004. The role of self-rated ability in the accuracy of confidence judgements in eyewitness memory and general knowledge. Applied Cognitive Psychology 18, 157–168.CrossRefGoogle Scholar
1998. Comparative risk assessment: where does the public fit in?Science, Technology, and Human Values 23, 221–241.CrossRefGoogle Scholar
, , and 2008. Deliberating the risks of nanotechnologies for energy and health applications in the United States and United Kingdom. Nature Nanotechnology 4, 95–98.Google ScholarPubMed
, , , , , , , and 2003. River flows and water wars: emerging science for environmental decision making. Frontiers in Ecology and the Environment 1, 298–306.CrossRefGoogle Scholar
, , , , and 1999. Subjective versus statistical model assessment of mortality risk in open heart surgical procedures. Annals of Thoracic Surgery 67, 635–640.CrossRefGoogle ScholarPubMed
2006. Measuring observer performance in chest radiology: some experiences. Journal of the American College of Radiology 3, 423–432.CrossRefGoogle ScholarPubMed
, , , , , and 2002. Truth survival in clinical research: an evidence-based requiem. Annals of Internal Medicine 136, 888–895.CrossRefGoogle ScholarPubMed
2003. Science and the law: evaluating evidentiary reliability. Australian Bar Review 23, 263–295.Google Scholar
and 1997. Fisheries stock assessment and decision analysis: the Bayesian approach. Reviews in Fish Biology and Fisheries 7, 35–63.CrossRefGoogle Scholar
, and 2000. A proposal for fuzzy IUCN categories and criteria. Biological Conservation 92, 101–108.CrossRefGoogle Scholar
, and 2002. A taxonomy and treatment of uncertainty for ecology and conservation biology. Ecological Applications 12, 618–628.CrossRefGoogle Scholar
, , and 2001. The currency and tempo of extinction. American Naturalist 157, 1–10.CrossRefGoogle ScholarPubMed
2002. Critical care cardiovascular nurse expert and novice diagnostic cue utilization. Journal of Advanced Nursing 39, 24–34.CrossRefGoogle ScholarPubMed
. and 2009. Assessing authority. Journal of the American Medical Association 301, 1819–1821.Google ScholarPubMed
1994. Risk perception of different societal groups: Australian findings and cross-national comparisons. Australian Journal of Psychology 46, 150–163.CrossRefGoogle Scholar
1998. The risk notion: epistemological and empirical considerations. In and , editors. Integrated risk assessment. Balkema, Rotterdam.Google Scholar
, , and 2010. Future declines of the binational Laurentian Great Lakes fisheries: the importance of environmental and cultural change. Frontiers in Ecology and the Environment 8, 239–244.CrossRefGoogle Scholar
and 2001. Expert opinions in forecasting: the role of the Delphi technique. In , editor. Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Boston.Google Scholar
, , and 2002. The Pacific Salmon wars: what science brings to the challenge of recovering species. Annual Review of Ecology and Systematics 33, 665–706.CrossRefGoogle Scholar
, , and 2004. The Supreme Court forecasting project: legal and political science approaches to predicting Supreme Court decisionmaking. Columbia Law Review 104, 1150–1209.CrossRefGoogle Scholar
, , , and 2006. Group decision making in hidden profile situations: dissent as a facilitator for decision quality. Journal of Personality and Social Psychology 91, 1080–1093.Google Scholar
, and 2012. Why groups perform better than individuals at quantitative judgment tasks: group-to-individual transfer as an alternative to differential weighting. Organizational Behavior and Human Decision Processes 118, 24–36.CrossRefGoogle Scholar
and 2007. Assessing the competence and credibility of human sources of intelligence evidence: contributions from law and probability. Law, Probability and Risk 6, 247–274.CrossRefGoogle Scholar
and 2002. Combination of evidence in Dempster-Shafer theory. SAND Report, SAND2002-0835. Sandia National Laboratories, Albuquerque, New Mexico.
1987. Minimum viable populations: coping with uncertainty. In , editor. Viable populations for conservation.Cambridge University Press.Google Scholar
1992a. Competence in experts: the role of task characteristics. Organizational Behavior and Human Decision Processes 53, 252–266.CrossRefGoogle Scholar
1992b. How much information does an expert use? Is it relevant?Acta Psychologica 81, 75–86.CrossRefGoogle Scholar
1996a. Methodological rules for four classes of scientific uncertainty. In , editor. Scientific uncertainty and environmental problem solving. Blackwell, Cambridge, Massachusetts.Google Scholar
1996b. Value judgments in verifying and validating risk assessment models. In , editor. Handbook for environmental risk decision making: values, perceptions and ethics. CRC Lewis Publishers, Boca Raton.Google Scholar
1959. Theories for decision-making in economic and behavioral science. The American Economic Review 49, 253–283.Google Scholar
1999. Trust, emotion, sex, politics, and science: surveying the risk-assessment battlefield. Risk Analysis 19, 689–701.CrossRefGoogle ScholarPubMed
, and 1984. Perception and acceptability of risk from energy systems. In and , editors. Public reactions to nuclear power: are there critical masses?AAAS/Westview, Boulder, Colorado.Google Scholar
, and 2000. Violence risk assessment and risk communication: the effects of using actual cases, providing instruction, and employing probability versus frequency formats. Law and Human Behavior 24, 271–296.CrossRefGoogle ScholarPubMed
and 2004. Overconfidence in interval estimates. Journal of Experimental Psychology: Learning, Memory, and Cognition 30, 299.Google ScholarPubMed
, , , , , , and 2007. Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge and New York.
, , , , and 2010. Reducing overconfidence in the interval judgments of experts. Risk Analysis 30, 512–523.CrossRefGoogle Scholar
and 1975. Probability encoding in decision analysis. Management Science 22, 340–358.CrossRefGoogle Scholar
, and 1994. Empirical evaluation of prior beliefs about frequencies: methodology and a case study in congenital heart disease. Journal of the American Statistical Association 89, 435–443.CrossRefGoogle Scholar
, , , , and 2014. Roles of scientists as policy advisers on complex issues: a literature review. Environmental Science and Policy 40, 16–25.CrossRefGoogle Scholar
and 1985. Pooling of unshared information in group decision making: biased information sampling during discussion.Journal of Personality and Social Psychology 48, 1467–1478.CrossRefGoogle Scholar
, and 1993. Deriving parameter probability density functions. Reliability Engineering and System Safety 42, 271–291.CrossRefGoogle Scholar
and , editors. 1996. Understanding risk: informing decisions in a democratic society. National Academy Press, Washington, DC.Google Scholar
, and 1993. Practical intelligence: the nature and role of tacit knowledge in work and at school. In and , editors. Mechanisms of everyday cognition. Lawrence Erlbaum Associates, Hillside, New Jersey.Google Scholar
and 1997. Probabilistic risk assessment of engineering systems. Chapman and Hall, London.Google Scholar
2007. A general framework for analysing diversity in science, technology and society. Journal of the Royal Society Interface 4, 707–719.CrossRefGoogle ScholarPubMed
and 2003. Enhancing causal assessment of estuarine fishkills using graphical models. Ecosystems 6, 11–19.CrossRefGoogle Scholar
2005. The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies and nations. Doubleday, New York.Google Scholar
and 1988. Illusion and well-being: a social psychological perspective on mental health. Psychological Bulletin 103, 193–210.CrossRefGoogle ScholarPubMed
and 2005. When 90% confidence intervals are 50% certain: on the credibility of credible intervals. Applied Cognitive Psychology 19, 455–475.CrossRefGoogle Scholar
2005. Expert political opinion, how good is it? How can we know?Princeton University Press, New Jersey.Google Scholar
and 2011. Intelligent management of intelligence agencies. American Psychologist 66, 542–554.CrossRefGoogle ScholarPubMed
and 2007. Environments that make us smart. Current Directions in Psychological Science 16, 167–171.CrossRefGoogle Scholar
1997. Visual explanations: images and quantities, evidence and narrative. Graphics Press, Cheshire, Connecticut.Google Scholar
. and 1971. Belief in the law of small numbers. Psychological Bulletin 76, 105–110.CrossRefGoogle Scholar
and 1974. Judgement under uncertainty: heuristics and biases. Science 185, 1124–1131.CrossRefGoogle ScholarPubMed
and 1985. The framing of decisions and the psychology of choice. In G. Wright, editor. Behavioral decision making, pp. 25–41. Springer, New York.Google Scholar
and 1982a. Belief in the law of small numbers. In , and , editors. Judgement under uncertainty: heuristics and biases. Cambridge University Press, pp. 23–30.Google Scholar
and 1982b. Causal schemata in judgments under uncertainty. In , , , editors. Judgment under uncertainty: heuristics and biases. Cambridge University Press.Google Scholar
, , and 2011. Accuracy and reliability of forensic latent fingerprint decisions. Proceedings of the National Academy of Sciences of the USA 108, 7733–7738.CrossRefGoogle ScholarPubMed
2001. Expert judgment resolution in technically-intensive policy disputes. In and , editors. Assessment and management of environmental risks. Kluwer Academic Publishers, Dordrecht.Google Scholar
2013. Argument mapping. In H. Pashler, editor. Encyclopedia of the mind. Sage, Thousand Oaks, CA.CrossRefGoogle Scholar
1991. Consensus of expertise: the role of consensus of experts in formulating public policy and estimating facts. Journal of Medicine and Philosophy 16, 427–445.CrossRefGoogle ScholarPubMed
2002. Transferring strategies of land management: indigenous land owners and environmental scientists. Research in Science and Technology Studies, Knowledge and Society 13, 155–181.Google Scholar
1996. Quantitative risk analysis: a guide to Monte Carlo simulation modelling. Wiley, Chichester.Google Scholar
2000. Bayesian methods in conservation biology. Conservation Biology 14, 1308–1316.CrossRefGoogle Scholar
and 1986. Does the expert know? The reliability of predictions and confidence ratings of experts. In , and , editors. Intelligent decision support in process environments. Springer-Verlag, Berlin.Google Scholar
1991. Statistical reasoning with imprecise probabilities. Chapman and Hall, London.CrossRefGoogle Scholar
2000. Towards a unified theory of imprecise probability. International Journal of Approximate Reasoning 24, 125–148.CrossRefGoogle Scholar
and 2001. A behavioral model for linguistic uncertainty. Information Sciences 134, 1–37.CrossRefGoogle Scholar
and 1983. Encoding subjective probabilities: a psychological and psychometric review. Management Science 29, 151–173.CrossRefGoogle Scholar
, , and 1997. Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making 10, 243–268.3.0.CO;2-M>CrossRefGoogle Scholar
1997. Appeal to expert opinion: arguments from authority. Pennsylvania State University Press, University Park.Google Scholar
, , and 2002. A domain-specific risk-attitude scale: measuring risk perceptions and risk behaviors. Journal of Behavioral Decision Making 15, 263–290.CrossRefGoogle Scholar
, , and 1991. A novel approach to reducing uncertainty: the Delphi group. Technological Forecasting and Social Change 39, 253–263.CrossRefGoogle Scholar
and 2012. Decloaking the privileged expert. Journal of Management and Organization 18, 300–310.CrossRefGoogle Scholar
, , , , and 2009. The complex relationship between pediatric cardiac surgical case volumes and mortality rates in a national clinical database. Journal of Thoracic and Cardiovascular Surgery 137, 1133–1140.CrossRefGoogle Scholar
2002. Judging the accuracy of a likelihood judgment: the case of smoking risk. Journal of Behavioral Decision Making 15, 19–35.CrossRefGoogle Scholar
and 1993. Evaluating and combining physicians’ probabilities of survival in an intensive care unit. Management Science 39, 1526–1543.CrossRefGoogle Scholar
2013. Making Decisions When Estimates Conflict: Improving Judgements in Environmental Science. PhD Thesis, University of Melbourne.
, , , , , , , , and 2012. The intelligence game: assessing Delphi groups and structured question formats. In , and , editors. Proceedings of the 5th Australian Security and Intelligence Conference. pp. 14–26. Security Research Institute, Edith Cowan University.Google Scholar
, and 2013b. The trial of scientists at L'Aquila: Lessons for risk communication. Risk Analysis (in review).Google Scholar
WMD Commission. 2005. The commission on the intelligence capabilities of the United States regarding weapons of mass destruction. March 31, 2005. US Government 374. In Wheaton, K. and Chido, D. 2007. Evaluating Intelligence. Competitive Intelligence Magazine 10, 2–19.
, , , and 2010. Evidence for a collective intelligence factor in the performance of human groups. Science 330, 686–688.CrossRefGoogle ScholarPubMed
, and 2002. An empirical test of the relative validity of expert and lay judgments of risk. Risk Analysis 22, 1107–1122.CrossRefGoogle Scholar
1996. May the sheep safely graze? A reflexive view of the expert-lay knowledge divide. In , and , editors. Risk, environment and modernity: towards a new ecology. Sage Publications, London.Google Scholar
2000. Making systematic sense of public discontents with expert knowledge: two analytical approaches and a case study. Public Understanding of Science 9, 105–122.CrossRefGoogle Scholar
1986. A simple view of the Dempster–Shafer theory of evidence and its implication for the rule of combination. The AI Magazine 1986, 85–90.Google Scholar