Skip to main content Accessibility help
×
Home

Model risk – daring to open up the black box

  • A. Aggarwal, M. B. Beck, M. Cann, T. Ford, D. Georgescu, N. Morjaria, A. Smith, Y. Taylor, A. Tsanakas, L. Witts and I. Ye...

Abstract

With the increasing use of complex quantitative models in applications throughout the financial world, model risk has become a major concern. Such risk is generated by the potential inaccuracy and inappropriate use of models in business applications, which can lead to substantial financial losses and reputational damage. In this paper, we deal with the management and measurement of model risk. First, a model risk framework is developed, adapting concepts such as risk appetite, monitoring, and mitigation to the particular case of model risk. The usefulness of such a framework for preventing losses associated with model risk is demonstrated through case studies. Second, we investigate the ways in which different ways of using and perceiving models within an organisation both lead to different model risks. We identify four distinct model cultures and argue that in conditions of deep model uncertainty, each of those cultures makes a valuable contribution to model risk governance. Thus, the space of legitimate challenges to models is expanded, such that, in addition to a technical critique, operational and commercial concerns are also addressed. Third, we discuss through the examples of proxy modelling, longevity risk, and investment advice, common methods and challenges for quantifying model risk. Difficulties arise in mapping model errors to actual financial impact. In the case of irreducible model uncertainty, it is necessary to employ a variety of measurement approaches, based on statistical inference, fitting multiple models, and stress and scenario analysis.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Model risk – daring to open up the black box
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Model risk – daring to open up the black box
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Model risk – daring to open up the black box
      Available formats
      ×

Copyright

Corresponding author

*Correspondence to: Nirav Morjaria, 8 Canada Square, London E14 5HQ. E-mail: nirav.morjaria@hsbc.com

References

Hide All
Bauer, D., Bergmann, D. & Reuss, A. (2010). Solvency II and Nested Simulations – A Least-Squares Monte Carlo Approach, Proceedings of the 2010 ICA Congress, available at http://www.ressources-actuarielles.net/EXT/ISFA/1226.nsf/0/3690458b48d11a78c12579ae00695536/ (accessed 26 February 2015).
BBC News (2013a). ‘London Whale’ traders charged in US over $6.2bn loss, 14 August, available at http://www.bbc.co.uk/news/business-23692109 (accessed 26 February 2015).
BBC News (2013b). JP Morgan makes $920m London Whale payout to regulators, 19 September, available at http://www.bbc.co.uk/news/business-24159801 (accessed 26 February 2015).
Beck, M.B. (2014). Handling uncertainty in environmental models at the science-policy-society interfaces, in Error and Uncertainty in Scientific Practice (ed. M. Boumans, G. Hon, and A.C. Petersen), Pickering & Chatto, London, pp. 97–135.
Beck, M.B. & Chen, J. (2000). Assuring the quality of models designed for predictive purposes, in Sensitivity Analysis (ed. A. Saltelli, K. Chan, and E.M. Scott), Wiley, Chichester, pp. 401–420.
Board of Governors of the Federal Reserve System (2011). SR 11-7: guidance on model risk management, available at http://www.federalreserve.gov/bankinforeg/srletters/sr1107a1.pdf (accessed 26 February 2015).
Borges, J.L. (1998). On exactitude in science, in Collected Fictions (transl. Hurley, H.), Penguin, p. 325.
Chartis Research Ltd (2014). The risk enabled enterprise – model risk management, available at http://www.chartis-research.com/research/reports/the-risk-enabled-enterprise-model-risk-management (accessed 26 February 2015).
Department for Transport and Patrick McLoughlin (2012). Report of the Laidlaw inquiry, available at https://www.gov.uk/government/publications/report-of-the-laidlaw-inquiry (accessed 26 February 2015).
Derman, E. & Wilmott, P. (2009). The financial modeller’s manifesto, available at http://www.wilmott.com/blogs/paul/index.cfm/2009/1/8/Financial-Modelers-Manifesto (accessed 26 February 2015).
Edwards, M. & Hoosain, Z. (2012). ‘The Philosophy of Modelling’, presented to the Staple Inn Actuarial Society, 26 June, available at http://www.actuaries.org.uk/events/one-day/sias-meeting-philosophy-modelling (accessed 26 February 2015).
Fisher, E., Pascual, P. & Wagner, W. (2010). Understanding environmental models in their legal and regulatory context. Journal of European Environmental Law, 22(2), 251283.
Funtowicz, S.O. & Ravetz, J.R. (1990). Uncertainty and Quality in Science for Policy. Dordrecht: Kluwer.
Gray, J. (2014). See no evil, available at http://www.bbc.co.uk/news/magazine-25680144 (accessed 26 February 2015).
Haldane, A. & Madouros, V. (2012). ‘The dog and the frisbee’, Federal Reserve Bank of Kansas City’s 36th Economic Policy Symposium, available at http://www.bankofengland.co.uk/publications/Documents/speeches/2012/speech596.pdf (accessed 26 February 2015).
HM Treasury (2013). Review of quality assurance of government analytical models, available at https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models (accessed 26 February 2015).
Ingram, D., Tayler, P. & Thompson, M. (2012). Surprise, surprise: from neoclassical economics to e-life. ASTIN Bulletin, 42(2), 389411.
Jasanoff, S. (1990). The Fifth Branch: Science Advisors as Policymakers. Cambridge, MA: Harvard University Press.
Lowenstein, R. (2008). Long-term capital management: it’s a short-term memory, New York Times, 6 September, p. 13, available at http://www.nytimes.com/2008/09/07/business/worldbusiness/07iht-07ltcm.15941880.html (accessed 26 February 2015).
National Research Council (2007). Models in Environmental Regulatory Decision Making. Washington, DC: National Academies Press.
Pachon, R. & Trefethen, L.N. (2009). Barycentric-Remez algorithms for best polynomial approximation in the chebfun system. BIT Numerical Mathematics, 49(4), 721741.
Palmer, T.N. & Hardaker, P.J. (2011). Handling uncertainty in science. Philosophical Transactions Royal Society A, 369, 46814684.
Richards, S.J., Currie, I.D. & Ritchie, G.P. (2013). A value-at-risk framework for longevity trend risk. British Actuarial Journal, 19(1), 116139.
Runge, C. (1901). Über empirische Funktionen und die Interpolation zwischen äquidistanten Ordinaten. Zeitschrift für Mathematik und Physik, 46, 224243.
Smith, L.A. & Peterson, A.C. (2014). Variations on Reliability: Connecting Climate Predictions to Climate Policy.
Verweij, M. & Thompson, M. eds. (2011). Clumsy Solutions for a Complex World: Governance, Politics and Plural Perceptions . Basingstoke: Palgrave Macmillan.
Wagner, W., Fisher, E. & Pascual, P. (2010). Misunderstanding models in environmental and public health regulation. New York University Law Journal, 18, 293356.
Wood, S.N. (2006). Generalized Additive Models: An Introduction with R. London: Chapman & Hall.

Keywords

Model risk – daring to open up the black box

  • A. Aggarwal, M. B. Beck, M. Cann, T. Ford, D. Georgescu, N. Morjaria, A. Smith, Y. Taylor, A. Tsanakas, L. Witts and I. Ye...

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed