Skip to main content Accessibility help
×
Home

Model risk: illuminating the black box

  • R. Black, A. Tsanakas, A. D. Smith, M. B. Beck, I. D. Maclugash, J. Grewal, L. Witts, N. Morjaria, R. J. Green and Z. Lim...

Abstract

This paper presents latest thinking from the Institute and Faculty of Actuaries’ Model Risk Working Party and follows on from their Phase I work, Model Risk: Daring to Open the Black Box. This is a more practical paper and presents the contributors’ experiences of model risk gained from a wide range of financial and non-financial organisations with suggestions for good practice and proven methods to reduce model risk. After a recap of the Phase I work, examples of model risk communication are given covering communication: to the Board; to the regulator; and to external stakeholders. We present a practical framework for model risk management and quantification with examples of the key actors, processes and cultural challenge. Lessons learned are then presented from other industries that make extensive use of models and include the weather forecasting, software and aerospace industries. Finally, a series of case studies in practical model risk management and mitigation are presented from the contributors’ own experiences covering primarily financial services.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Model risk: illuminating the black box
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Model risk: illuminating the black box
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Model risk: illuminating the black box
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

Corresponding author

*Correspondence to: Rob Black, Standard Life House, 30 Lothian Road, Edinburgh EH1 2DH, UK E-mail: rob_black@standardlife.com

References

Hide All
Aggarwal, A., Beck, M.B., Cann, M., Ford, T., Georgescu, D., Morjaria, N., Smith, A., Taylor, Y., Tsanakas, A., Witts, L. & Ye, I. (2016). Model risk – daring to open up the black box. British Actuarial Journal, 21(2), 229296.
Bank of England (2015). Governance and the role of Boards – speech by Andrew Bailey, available at http://www.bankofengland.co.uk/publications/Pages/speeches/2015/858.aspx (accessed 30 May 2017).
BBC (2017). Crash was economists’ “Michael Fish” moment, says Andy Haldane, available at http://www.bbc.co.uk/news/uk-politics-38525924 (accessed 5 April 2017).
Beck, M.B. (1991). Forecasting environmental change. Journal of Forecasting, 10(1–2), 319.
Beck, M.B. (2014). Handling uncertainty in environmental models at the science-policy-society interfaces. in Error and Uncertainty in Scientific Practice. ed. M. Boumans, G. Hon & A.C. Petersen London: Pickering & Chatto; pp. 97135.
Board of Governors of the Federal Reserve System (2011). SR 11-7: Guidance on Model Risk Management, available at http://www.federalreserve.gov/bankinforeg/srletters/sr1107a1.pdf (accessed 31 May 2017).
Bovey, R., Wallentin, D., Bullen, S. & Green, J. (2009). Professional Excel Development: The Definitive Guide to Developing Applications Using Microsoft Excel, VBA, and .NET (2nd ed.). Addison Wesley, London.
Burns, T. (1986). The interpretation and use of economic predictions, in Predictability in Science and Society (ed. J. Mason, P. Mathias & J.H. Westcott). London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, pp. 103–125.
Cassidy, J. (2013). The Reinhardt and Rogoff controversy: a summing up New Yorker, available at http://www.newyorker.com/rational-irrationality/the-reinhart-and-rogoff-controversy-a-summing-up (accessed 27 April 2017).
Cohen, G.A. (1986). Historical inevitability and human agency in marxism, in Predictability in Science and Society (ed. J. Mason, P. Mathias & J.H. Westcott). London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, pp. 65–87.
Danielsson, J. & Shin, H.S (2003). Endogenous risk, in Modern Risk Management: A History (ed. P. Field), London: Risk Books, pp. 297316.
Greenspan, A (2013). The Map and the Territory. Risk, Human Nature, and the Future of Forecasting. New York: Penguin.
Gigerenzer, G. & Edwards, A (2003). Simple tools for understanding risks: from innumeracy to insight. British Medical Journal, 327(7417), 741744.
Herndon, T., Ash, M. & Pollin, R (2014). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2), 257279.
HM Treasury (2013). Review of quality assurance of government analytical models, available at https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models (accessed 31 May 2017).
Inquirer (2016). 90 per cent of NHS Trusts are still running Windows XP machines, available at http://www.theinquirer.net/inquirer/news/2479315/90-per-cent-of-nhs-trusts-are-still-running-windows-xp-machines (accessed 31 May 2017).
Lighthill, J. (1986). The recently recognized failure of predictability in Newtonian dynamics, in Predictability in Science and Society (ed. J. Mason, P. Mathias & J.H. Westcott). London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, pp. 35–50.
LMA Exposure Management Working Group (2017). Uncertainty in Catastrophe Modelling for Non-Catastrophe Modellers. New York: Penguin.
Mason, B.J. (1986). Numerical weather prediction, in Predictability in Science and Society (ed. J. Mason, P. Mathias & J.H. Westcott). London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, pp. 51–60.
Mason, B.J., Mathias, P. & Westcott, J.H., (eds) (1986). Predictability in Science and Society. London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, 145 pp.
Met Office (2011). UK NWP Index, available at http://www.metoffice.gov.uk/research/weather/numerical-modelling/verification/uk-nwp-index (accessed 18 April 2017).
Met Office (2017). Continually improving our forecasts, available http://www.metoffice.gov.uk/services/accuracy (accessed 18 April 2017).
Moody’s Analytics (2014). Risk scenario generator, available at http://www.moodysanalytics.com/~/media/Brochures/Enterprise-Risk-Solutions/RiskIntegrity/2014-08-10-Risk-Scenario-Generator.pdf (accessed 9 May 2017).
NASA (1999a). Mars Climate Orbiter Team Finds Likely Cause Of Loss, news release, available at https://mars.jpl.nasa.gov/msp98/news/mco990930.html (accessed 8 May 2017).
NASA (1999b). Mars Climate Orbiter Mishap Investigation Board (1999) “Phase I Report”, available at ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf (accessed 8 May 2017).
NASA (2000a). The Mars Climate Orbiter Mishap Investigation Board “Report on Project Management in NASA”, available at http://www.dcs.gla.ac.uk/~johnson/Mars/MCO_MIB_Report.pdf (accessed 8 May 2017).
NASA (2000b). NASA’s Jet Propulsion Laboratory Website for Mars Climate Orbiter, available at https://mars.jpl.nasa.gov/msp98/orbiter/ (accessed 8 May 2017).
NASA (2009). Lost in translation, System Failure Case Studies 3 (5), available at https:///sma.nasa.gov/docs/default-source/safety-messages/safetymessage-2009-08-01-themarsclimateorbitermishap.pdf?sfvrsn=4 (accessed 8 May 2017).
Pesenti, S., Millossovich, P. & Tsanakas, A. (2017). Reverse sensitivity testing, presented at the 21st International Congress on Insurance: Mathematics and Economics, 3-5 July 2017, Vienna, Austria.
Reinhart, C. & Rogoff, K. (2010). Growth in a time of debt. American Economic Review, 100(2), 573578.
Sandtable (2015). Floods, forecasting and flawed policy, available at http://www.sandtable.com/floods-forecasting-and-flawed-policy/ (accessed 27 April 2017).
Sen, A.K. (1986). Predictability and economic theory, in Predictability in Science and Society (ed. J. Mason, P. Mathias & J.H. Westcott). London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, pp. 3–23.
van Asselt, M.B.A. & Rotmans, J. (1996). Uncertainty in perspective. Global Environmental Change, 6(2), 121157.
Westcott, J.H. (1986). Application of control theory to macro-economic models, in Predictability in Science and Society (ed. J. Mason, P. Mathias & J.H. Westcott). London, Proceedings of Royal Society and British Academy Symposium, Royal Society and British Academy, pp. 89–101.
Williams, D. (2017). How statistics lost their power – and why we should fear what comes next. The Long Read, The Guardian, 19 January, available at https://www.theguardian.com/politics/2017/jan/19/crisis-of-statistics-big-data-democracy (accessed 31 May 2017).
World Meteorological Organisation (WMO) (2000). Guidelines on performance assessment of public weather services. Technical Document TD 1023, WMO, Geneva.
World Meteorological Organisation (WMO) (2017). Public weather services (verification), available at http://www.wmo.int/pages/prog/amp/pwsp/qualityassuranceverification_en.htm (accessed 18 April 2017).

Keywords

Model risk: illuminating the black box

  • R. Black, A. Tsanakas, A. D. Smith, M. B. Beck, I. D. Maclugash, J. Grewal, L. Witts, N. Morjaria, R. J. Green and Z. Lim...

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed