Skip to main content Accessibility help
×
Home
  • Cited by 25
  • Print publication year: 2013
  • Online publication date: March 2013

4 - Quantifying scientific uncertainty from expert judgement elicitation

References

Abrahamson, N. A., Coppersmith, K. J., Koller, M., et al. (2004) Probabilistic seismic hazard analysis for Swiss nuclear power plant sites, PEGASOS Project, Vols 1–6, NAGRA.
ACERA (2007) Eliciting expert judgments: literature review. http://www.acera.unimelb.edu.au/materials/endorsed/0611.pdf (accessed 28 December 2011).
ACERA (2010) Elicitation tool: process manual. http://www.acera.unimelb.edu.au/materials/software.html (accessed 28 December 2011).
Aspinall, W. P. (2006) Structured elicitation of expert judgment for probabilistic hazard and risk assessment in volcanic eruptions. In Statistics in Volcanology, ed. H. M. Mader, S. G. Coles, C. B. Connor and L. J. Connor, London: The Geological Society for IAVCEI, pp. 15–30.
Aspinall, W. (2010) Opinion: a route to more tractable expert advice. Nature 463: 294–295.
Aspinall, W. and Cooke, R. M. (1998) Expert judgment and the Montserrat Volcano eruption. In Proceedings, 4th International Conference on Probabilistic Safety Assessment and Management PSAM4, ed. Ali Mosleh and Robert A. Bari, vol. 3, New York, NY: Springer, pp. 2113–2118.
Aspinall, W. P., Loughlin, S. C., Michael, A. D., et al. (2002) The Montserrat Volcano Observatory; its evolution, organization, role and activities. In The Eruption of Soufrière Hills Volcano, Montserrat from 1995 to 1999, ed. T. H. Druitt and B. P. Kokelaar, London: Geological Society of London, pp. 71–91.
Aspinall, W. P., Woo, G., Voight, B., et al. (2003) Evidence-based volcanology; application to eruption crises. Journal of Volcanology and Geothermal Research: Putting Volcano Seismology in a Physical Context; In Memory of Bruno Martinelli 128: 273–285.
Brockhoff, K. (1975) The performance of forecasting groups in computer dialogue and face to face discussions. In The Delphi Method, Techniques and Applications, ed. H. A. Linstone and M. Turoff, Reading, MA: Addison Wesley, pp. 291–321.
Brown, A. J. and Aspinall, W. P. (2004) Use of expert elicitation to quantify the internal erosion processes in dams. In Proceedings of the British Dam Society Conference: Long-term Benefits and performance of Dams, Canterbury: Thomas Telford, pp. 282–297.
Budnitz, R., Apostolakis, G., Boore, D. M., et al. (1998) Use of technical expert panels: applications to probabilistic seismic hazard analysis. Risk Analysis 18: 463–469.
Budnitz, R. J., C. A. Cornell and Morris, P. A. (2005) Comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants,’ in Engineering Geology, vol. 78: 285–307. Engineering Geology 82: 76–78.
Clemen, R. T. and Reilly, T. (1999) Correlations and copulas for decision and risk analysis. Management Science 45: 208–224.
Clemen, R. T. and Winkler, R. L. (1986) Combining economic forecasts. Journal of Business & Economic Statistics 4 (1): 39–46.
Clemen, R. T. and Winkler, R. L. (1987) Calibrating and combining precipitation probability forecasts. In Probability & Bayesian Statistics, ed. Viertl, R.., New York, NY: Plenum Press, pp. 97–110.
Clemen, R. T. and Winkler, R. L. (1999) Combining probability distributions from experts in risk analysis. Risk Analysis 19: 187–203.
Cooke, R. (1987) A theory of weights for combining expert opinions. Delft University of Technology, Department of Mathematics and Informatics, Report 87–25.
Cooke, R. M. (1991) Experts in Uncertainty:Opinion and Subjective Probability in Science. Oxford: Oxford University Press.
Cooke, R. M. (2008) Special issue on expert judgment. Reliability Engineering & System Safety 93: 655–656.
Cooke, R. M. and Goossens, L. L. H. J. (2008) TU Delft expert judgment data base. Reliability Engineering & System Safety 93: 657–674.
Cooke, R. M., Mendel, M. and Thijs, W. (1988) Calibration and information in expert resolution. Automatica 24: 87–94.
Cooke, R. M., ElSaadany, S. and Huang, X. (2008) On the performance of social network and likelihood-based expert weighting schemes. Reliability Engineering & System Safety 93: 745–756.
Coppersmith, K. J., Youngs, R. R. and Sprecher, C. (2009) Methodology and main results of seismic source characterization for the PEGASOS Project, Switzerland. Swiss Journal of Geosciences 102: 91–105.
De Groot, M. (1974) Reaching a consensus. Journal of the American Statistical Association 69: 118–121.
Delbecq, A. L., Van de Ven, A. and Gusstafson, D. (1975) Group Techniques for Program Planning. Glenview, IL: Scott, Foresman.
EPA (2009) Expert elicitation white paper: external review draft, January. http://www.epa.gov/spc/expertelicitation/index.htm (accessed 6 November 2009).
Fischer, G. (1975) An experimental study of four procedures for aggregating subjective probability assessments. Technical Report 75–7, Decisions and Designs Incl., McLean, Virginia.
Flandoli, F., Giorgi, E., Aspinall, W. P., et al. (2011) Comparison of a new expert elicitation model with the Classical Model, equal weights and single experts, using a cross-validation technique. Reliability Engineering & System Safety 96: 1292–1310.
French, S. (2008) Comments by Prof. French. Reliability Engineering & System Safety Expert Judgment 93: 766–768.
French, S. (2011) Aggregating expert judgment. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas (RACSAM) 105: 181–206.
Gigerenzer, G., Hertwig, R., van den Broek, E., et al. (2005) ‘A 30% chance of rain tomorrow’: how does the public understand probabilistic weather forecasts? Risk Analysis 25: 623–629.
Goossens, L. H. J. and Harper, F. T. (1998) Joint EC/USNRC expert judgment driven radiological protection uncertainty analysis. Journal of Radiological Protection 18: 249–264.
Gough, R. (1975) The effect of group format on aggregate subjective probability distributions. In Utility, Probability and Human Decision Making, ed. D. Wendt and C. Vlek, Dordrecht, Reidel.
Hanks, T. C., Abrahamson, N. A., Boore, D. M., et al. (2009) Implementation of the SSHAC Guidelines for Level 3 and 4 PSHAs: experience gained from actual applications. US Geological Survey Open-File Report 2009–1093. http://pubs.er.usgs.gov/publication/ofr20091093 (accessed 27 July 2012).
Helmer, O. (1966) Social Technology, New York, NY: Basic Books.
Helmer, O. (1968) Analysis of the future: the Delphi method, and The Delphi Method–an illustration. In Technological Forecasting for Industry and Government, ed. J. Bright, Englewood Cliffs, NJ: Prentice Hall.
Hora, S. C. (2004) Probability judgments for continuous quantities: linear combinations and calibration. Management Science 50: 597–604.
Hora, S. C. and Iman, R. L. (1989) Expert opinion in risk analysis: the NUREG-1150 experience. Nuclear Science and Engineering 102: 323–331.
IAEA (2010) Seismic Hazards in Site Evaluation for Nuclear Installations. Vienna: International Atomic Energy Agency.
Kahneman, D. (2011) Thinking, Fast and Slow, London: Allen Lane.
Kahneman, D., Slovic, P. and Tversky, A. (eds) (1982) Judgment Under Uncertainty: Heuristics and Biases, Cambridge: Cambridge University Press.
Kallen, M. J. and Cooke, R. M. (2002) Expert aggregation with dependence. In Probabilistic Safety Assessment and Management, ed. E. J. Bonano, A. L. Camp, M. J. Majors, R. A. Thompson, London: Elsevier, pp. 1287–1294.
KellyG. B., KenneallyR. M. and ChokshiN. C. (1991) Consideration of seismic events in severe accidents. In Proceedings of Probabilistic Safety Assessment and Management 2, pp. 871–876.
Klügel, J.-U. (2005a) Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants. Engineering Geology 78: 285–307.
Klügel, J.-U. (2005b) Reply to the comment of Krinitzsky on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, vol. 78: 285–307. Engineering Geology 82: 69–70.
Klügel, J.-U. (2005c) Reply to the comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, Vol. 78: 285–307, by Budnitz, by, J.U, Klugel. Engineering Geology 82: 79–85.
Klügel, J.-U. (2005d) Reply to the comment on J.U. Klugel’s: ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants,’ Eng. Geol. Vol. 78: 285–307, by Musson et al. Engineering Geology 82: 56–65.
Klügel, J.-U. (2005e) Reply to the comment on J. U. Klugel’s: ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants,’ in Engineering Geology, vol. 78: 285–307, by Lomnitz, by J.U. Klugel. Engineering Geology 82: 74–75.
Klügel, J.-U. (2005f) Reply to the comment on J. U. Klugel’s: Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants, in Engineering Geology, Vol. 78: 285–307, by Wang, by J.U, Klugel. Engineering Geology 82: 89–90.
Klügel, J.-U. (2007) Error inflation in probabilistic seismic hazard analysis. Engineering Geology 90: 186–192.
Klügel, J.-U. (2009) Probabilistic seismic hazard analysis for nuclear power plants: current practice from a European perspective. Nuclear Engineering and Technology 41: 1–12.
Klügel, J. U., Mualchin, L. and Panza, G. F. (2006) A scenario-based procedure for seismic risk analysis. Engineering Geology 88: 22.
Kotra, J. P., Lee, M. P., Eisenberg, N. A., et al. (1996) Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program. NUREG-1563. Washington, DC: US Nuclear Regulatory Commission.
Kraan, B. C. P. and Bedford, T. J. (2005) Probabilistic inversion of expert judgments in the quantification of model uncertainty. Management Science 51 (6): 995–1006.
Krinitzsky, E. L. (2005) Comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, vol. 78: 285–307. Engineering Geology 82: 66–68.
Kurowicka, D. and Cooke, R. (2006) Uncertainty Analysis with High Dimensional Dependence Modelling. Chichester: John Wiley and Sons.
Kynn, M. (2008) The ‘heuristics and biases’ in expert elicitation. Journal of the Royal Statistical Society: Series A 171: 239–264.
Lehrer, K. and Wagner, C. G. (1981) Rational Consensus in Science and Society. Dordrecht: Reidel.
Lin, S.-W. and Bier, V. M. (2008) A study of expert overconfidence. Reliability Engineering & System Safety 93: 775–777.
Lin, S.-W. and Cheng, C.-H. (2008) Can Cooke’s Model sift out better experts and produce well-calibrated aggregated probabilities? 2008 IEEE International Conference on Industrial Engineering and Engineering Management, Singapore: IEEE Singapore Section, pp. 425–429.
Lin, S.-W. and Cheng, C.-H. (2009) The reliability of aggregated probability judgments obtained through Cooke’s classical model. Journal of Modelling in Management 4: 149–161.
Lindley, D. V. (1971) Making Decisions, London: John Wiley and Sons.
Lindley, D. (1983) Reconciliation of probability distributions. Operations Research 31: 866–880.
Lomnitz, C. (2005) Comment on J.U. Klugel’s ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’, in Engineering Geology, vol. 78: 285–307. Engineering Geology 82: 71–73.
Mendel, M. and Sheridan, T. (1989) Filtering information from human experts. IEEE Transactions on Systems, Man and Cybernetics 19 (1): 6–16.
Meyer, M. A. and Booker, J. M. (2001) Eliciting and Analyzing Expert Judgment: A Practical Guide. Philadelphia, PA: ASA-SIAM.
Morgan, M. G. and Henrion, M. (1990) Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. New York, NY: Cambridge University Press.
Morris, P. (1974) Decision analysis expert use. Management Science 20: 1233–1241.
Morris, P. (1977) Combining expert judgments: a Bayesian approach. Management Science 23: 679–693.
Mosleh, A. and Apostolakis, G. (1982) Models for the use of expert opinions. Workshop on low probability high consequence risk analysis, Society for Risk Analysis, Arlington, VA, June.
Musson, R. M. W., Toro, G. R., Coppersmith, K. J., et al. (2005) Evaluating hazard results for Switzerland and how not to do it: A discussion of ‘Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants’ by J-U Klugel. Engineering Geology 82: 43–55.
Neslo, R., Micheli, F., Kappel, C. V., et al. (2009) Modeling stakeholder preferences with probabilistic inversion: application to prioritizing marine ecosystem vulnerabilities. In Real Time and Deliberative Decision Making: Application to Risk Assessment for Non-chemical Stressors, ed. I. Linkov, E. A. Ferguson and V. S. Magar, Amsterdam: Springer: pp. 265–284.
O’Hagan, A., Buck, C. E., Daneshkhah, A., et al. (2006) Uncertain Judgments: Eliciting Experts’ Probabilities. Chichester: John Wiley and Sons.
Parascandola, M. (2010) Epistemic risk: empirical science and the fear of being wrong. Law, Probability and Risk 9: 201–214.
Rothlisberger, J. D., Lodge, D. M., Cooke, R. M., et al. (2009) Frontiers in ecology and the environment. The Ecological Society of America. www.frontiersinecology.org (accessed 27 July 2012).
Sackman, H. (1975) Delphi Critique, Expert Opinion, Forecasting and Group Processes, Lexington, MA: Lexington Books,.
Seaver, D. (1977) How groups can assess uncertainty. Proceedings of the International Conference on Cybernetics and Society, New York: Institute of Electrical and Electronics Engineers, pp. 185–190.
Seaver, D. A. (1978) Assessing probabilities with multiple individuals: group interaction versus mathematical aggregation. Technical Report SSRI-78–3, Social Science Research Institute, University of Southern California, Los Angeles.
Skipp, B. O. and Woo, G. (1993). A question of judgement: expert or engineering? In Risk and Reliability in Ground Engineering, London: Thomas Telford, pp. 29–39.
SSHAC (1995) Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts; prepared by Senior Seismic Hazard Analysis Committee (SSHAC), Livermore, CA: Lawrence Livermore National Laboratory.
SSHAC (1997) [Senior Seismic Hazard Analysis Committee, R. J. Budnitz, Chairman, G.Apostolakis, D. M.Boore, L. S.Cluff, K. J.Coppersmith, C. A. Cornell and P. A. Morris] Recommendations for Probabilistic Seismic Hazard Analysis: Guidance on Uncertainty and Use of Experts, NUREG/CR-6372, Washington, DC: US Nuclear Regulatory Commission.
Stone, J., Aspinall, W. P. and Sparks, R. S. J. (2011) Poster presented at Soufriere Hills Volcano: 15 Years on Conference, Montserrat, West Indies, April.
Tuomisto, J. T., Wilson, A., Evans, J. S., et al. (2008) Uncertainty in mortality response to airborne fine particulate matter: combining European air pollution experts. Reliability Engineering & System Safety Expert Judgment 93: 732–744.
Tversky, A. and Kahneman, D. (2005) Judgment under uncertainty: heuristics and biases. In Social Cognition Key Readings, ed. D. L. Hamilton, New York, NY: Psychology Press, chapter 10.
Tyshenko, G. M., ElSaadany, S., Oraby, T., et al. (2011) Expert elicitation for the judgment of Prion disease risk uncertainties. Journal of Toxicology and Environmental Health, Part A: Current Issues 74: 261–285.
Vick, S. G. (2002) Degrees of Belief: Subjective Probability and Engineering Judgment, Reston, VA: ASCE Press.
Wang, Z. (2005) Comment on J.U. Klugel’s: Problems in the application of the SSHAC probability method for assessing earthquake hazards at Swiss nuclear power plants, in Engineering Geology, vol. 78: 285–307. Engineering Geology 82: 86–88.
Winkler, R. L. (1981) Combining probability distributions from dependent information sources. Management Science 27: 479–488.
Wisse, B., Bedford, T. and Quigley, J. (2008) Expert judgement combination using moment methods. Reliability Engineering & System Safety Expert Judgement 93: 675–686.
Woo, G. (1999) The Mathematics of Natural Catastrophes, River Edge, NJ: Imperial College Press.
Woo, G. (2011) Calculating Catastrophe, River Edge, NJ: Imperial College Press.
Woudenberg, F. (1991) An evaluation of Delphi. Technological Forecasting and Social Change 40: 131–150.
Yunusov, A. R., Cooke, R. M. and Krymsky, V. G. (1999) Rexcalibr-integrated system for processing expert judgment. In Proceedings 9th Annual Conference: Risk Analysis: Facing the New Millenium. Rotterdam, 10–13 October, ed. L. H. J. Goossens, Netherlands: Delft University, pp. 587–589.
Zuidema, P. (2006) PEGASOS: A PSHA for Swiss nuclear power plants–some comments from a managerial point of view. OECD-IAEA Specialist meeting on the Seismic Probabilistic Safety Assessment of Nuclear Facilities, Seogwipo, Jeju Island, Korea.