Hostname: page-component-76fb5796d-x4r87 Total loading time: 0 Render date: 2024-04-28T04:50:13.719Z Has data issue: false hasContentIssue false
  • English
  • Français

Aircraft fleet availability optimisation: a reinforcement learning approach

Part of: The 19th and 20th Australian International Aerospace Congresses

Published online by Cambridge University Press:  29 November 2023

K. Vos Open the ORCID record for K. Vos [Opens in a new window] ,
Z. Peng Open the ORCID record for Z. Peng [Opens in a new window] ,
E. Lee  and
W. Wang Open the ORCID record for W. Wang [Opens in a new window]
K. Vos*
Affiliation:
University of New South Wales, Sydney, NSW 2052, Australia
Z. Peng
Affiliation:
University of New South Wales, Sydney, NSW 2052, Australia
E. Lee
Affiliation:
Defence Science and Technology Group, Fishermans Bend, VIC 3207, Australia
W. Wang
Affiliation:
Defence Science and Technology Group, Fishermans Bend, VIC 3207, Australia
*
*Corresponding author: K. Vos; Email: voskilian@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

A fleet of aircraft can be seen as a set of degrading systems that undergo variable loads as they fly missions and require maintenance throughout their lifetime. Optimal fleet management aims to maximise fleet availability while minimising overall maintenance costs. To achieve this goal, individual aircraft, with variable age and degradation paths, need to operate cooperatively to maintain high fleet availability while avoiding mechanical failure by scheduling preventive maintenance actions. In recent years, reinforcement learning (RL) has emerged as an effective method to optimise complex sequential decision-making problems. In this paper, an RL framework to optimise the operation and maintenance of a fleet of aircraft is developed. Three cases studies, with varying number of aircraft in the fleet, are used to demonstrate the ability of the RL policies to outperform traditional operation/maintenance strategies. As more aircraft are added to the fleet, the combinatorial explosion of the number of possible actions is identified as a main computational limitation. We conclude that the RL policy has potential to support fleet management operators and call for greater research on the application of multi-agent RL for fleet availability optimisation.

Keywords

reinforcement learningmarkov decision processfleet operationmaintenance schedulingfleet availability optimisation
Type
Research Article
Information
The Aeronautical Journal , Volume 127 , Special Issue 1318: AIAC 2023 , December 2023 , pp. 2204 - 2218
Check for updates
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of Royal Aeronautical Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This paper is a version of a presentation given at the 20th Australian International Aerospace Congress (AIAC) held in 2023.

References

Painter, M.K., Erraguntla, M., Hogg, G.L. and Beachkofski, B. Using simulation, data mining, and knowledge discovery techniques for optimized aircraft engine fleet management. Proc. – Winter Simul. Conf., 2006, pp 1253–1260. https://doi.org/10.1109/WSC.2006.323221Google Scholar
Khoo, H.L. and Teoh, L.E. An optimal aircraft fleet management decision model under uncertainty. J. Adv. Transp., 2014, 48, pp 798–820. https://doi.org/10.1002/ATR.1228CrossRefGoogle Scholar
Kiran, R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A.A., Yogamani, S. and Pérez, P. Deep reinforcement learning for autonomous driving: A survey. IEEE Trans. Intell. Transport. Syst., 2022, 23, p 4909. https://doi.org/10.1109/TITS.2021.3054625CrossRefGoogle Scholar
Giannoccaro, I. and Pontrandolfo, P. Inventory management in supply chains: A reinforcement learning approach. Int. J. Prod. Econ., 2002, 78, pp 153–161. https://doi.org/10.1016/S0925-5273(00)00156-0CrossRefGoogle Scholar
Silver, D., Hubert, T., Schrittwieser, J., et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 1979, 362, (2018), pp 1140–1144. https://doi.org/10.1126/science.aar6404CrossRefGoogle Scholar
Sutton, R.S., Barto, A.G. Reinforcement Learning, 2nd Edn: The MIT Press, 2015. https://mitpress.mit.edu/books/reinforcement-learning-second-edition (accessed March 2, 2021).Google Scholar
Mattila, V. and Virtanen, K. Maintenance scheduling of a fleet of fighter aircraft through multi-objective simulation-optimization. Simulation, 2014, 90, pp 1023–1040. https://doi.org/10.1177/0037549714540008/FORMAT/EPUBCrossRefGoogle Scholar
Shahmoradi-Moghadam, H., Safaei, N. and Sadjadi, S.J. Robust maintenance scheduling of aircraft fleet: A hybrid simulation-optimization approach. IEEE Access., 2021, 9, pp 17854–17865. https://doi.org/10.1109/ACCESS.2021.3053714CrossRefGoogle Scholar
Torres Sanchez, D., Boyacı, B. and Zografos, K.G. An optimisation framework for airline fleet maintenance scheduling with tail assignment considerations. Transport. Res. Part B., 2020, 133, pp 142–164. https://doi.org/10.1016/j.trb.2019.12.008CrossRefGoogle Scholar
Marlow, D.O., Looker, J.R. and Mukerjee, J. Optimal plans and policies for the management of military aircraft fleets. 19th Australian International Aerospace Congress, Melbourne, Engineers Australia, 2021, pp 22–25. https://search.informit.org/doi/abs/10.3316/informit.063508445265387Google Scholar
Bellani, L., Compare, M., Baraldi, P. and Zio, E. Towards developing a novel framework for practical PHM: A sequential decision problem solved by reinforcement learning and artificial neural networks. Int. J. Progn. Health Manag., 2019, 31, pp 1–15. https://www.researchgate.net/publication/339016560 (accessed March 2, 2021).Google Scholar
Lee, J. and Mitici, M. Deep reinforcement learning for predictive aircraft maintenance using probabilistic remaining-useful-life prognostics. Reliab. Eng. Syst. Saf., 2023, 230. https://doi.org/10.1016/j.ress.2022.108908CrossRefGoogle Scholar
Yousefi, N., Tsianikas, S. and Coit, D.W. Reinforcement learning for dynamic condition-based maintenance of a system with individually repairable components. Qual. Eng., 2020, 32, pp 388–408. https://doi.org/10.1080/08982112.2020.1766692CrossRefGoogle Scholar
Kuhnle, A., Jakubik, J. and Lanza, G. Reinforcement learning for opportunistic maintenance optimization. Prod. Eng., 2019, 13, pp 33–41. https://doi.org/10.1007/s11740-018-0855-7CrossRefGoogle Scholar
Mattila, V. and Virtanen, K. Scheduling fighter aircraft maintenance with reinforcement learning. Proceedings of the 2011 Winter Simulation Conference (WSC), 2011, pp 2535–2546. https://doi.org/10.1109/WSC.2011.6147962CrossRefGoogle Scholar
Virkler, D.A., Hillberry, B.M. and Goel, P.K. Statistical nature of fatigue crack propagation. Tech Rep AFFDL TR Air Force Flight Dyn Lab US TR-43-78, 1978.Google Scholar
Findlay, S.J. and Harrison, N.D. Why aircraft fail. Mater. Today, 2002, 5, pp 18–25. https://doi.org/10.1016/S1369-7021(02)01138-0CrossRefGoogle Scholar
Paris, P. and Erdogan, F. A critical analysis of crack propagation laws. J. Basic Eng., 1963, 85, pp 528–533. https://doi.org/10.1115/1.3656900CrossRefGoogle Scholar
Watkins, C.J.C.H. and Dayan, P. Q-learning. Mach. Learn., 1992, 8, pp 279–292.CrossRefGoogle Scholar
Clifton, J. and Laber, E. Q-learning: Theory and applications. Annu. Rev. Stat. Appl., 2020, 7, pp 279–301. https://doi.org/10.1146/annurev-statistics-031219CrossRefGoogle Scholar
Yousefi, N., Tsianikas, S., Coit, D.W. Dynamic maintenance model for a repairable multi-component system using deep reinforcement learning. Qual. Eng., 2022, 34, pp 16–35. https://doi.org/10.1080/08982112.2021.1977950CrossRefGoogle Scholar
Buşoniu, L., Babuška, R., De Schutter, B. Multi-agent reinforcement learning: An overview. Stud. Comput. Intell., 2010, 310, pp 183–221. https://doi.org/10.1007/978-3-642-14435-6_7/COVERCrossRefGoogle Scholar
Lowe, R., Wu, Y., Tamar, A., Harb, J., Uc, P.A., Openai, B. and Openai, I.M. Multi-agent actor-critic for mixed cooperative-competitive environments. Arxiv Preprint 1706.02275, 2017.Google Scholar