Hostname: page-component-848d4c4894-4rdrl Total loading time: 0 Render date: 2024-07-02T12:37:19.407Z Has data issue: false hasContentIssue false
  • English
  • Français

A novel reinforcement learning framework for disassembly sequence planning using Q-learning technique optimized using an enhanced simulated annealing algorithm

Published online by Cambridge University Press:  01 April 2024

Mirothali Chand Open the ORCID record for Mirothali Chand [Opens in a new window]  and
Chandrasekar Ravi Open the ORCID record for Chandrasekar Ravi [Opens in a new window]
Mirothali Chand
Affiliation:
Department of Computer Science and Engineering, National Institute of Technology Puducherry, Karaikal, India
Chandrasekar Ravi*
Affiliation:
Department of Computer Science and Engineering, National Institute of Technology Puducherry, Karaikal, India
*
Corresponding author: Chandrasekar Ravi; Email: chand191987@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

The increase in Electrical and Electronic Equipment (EEE) usage in various sectors has given rise to repair and maintenance units. Disassembly of parts requires proper planning, which is done by the Disassembly Sequence Planning (DSP) process. Since the manual disassembly process has various time and labor restrictions, it requires proper planning. Effective disassembly planning methods can encourage the reuse and recycling sector, resulting in reduction of raw-materials mining. An efficient DSP can lower the time and cost consumption. To address the challenges in DSP, this research introduces an innovative framework based on Q-Learning (QL) within the domain of Reinforcement Learning (RL). Furthermore, an Enhanced Simulated Annealing (ESA) algorithm is introduced to improve the exploration and exploitation balance in the proposed RL framework. The proposed framework is extensively evaluated against state-of-the-art frameworks and benchmark algorithms using a diverse set of eight products as test cases. The findings reveal that the proposed framework outperforms benchmark algorithms and state-of-the-art frameworks in terms of time consumption, memory consumption, and solution optimality. Specifically, for complex large products, the proposed technique achieves a remarkable minimum reduction of 60% in time consumption and 30% in memory usage compared to other state-of-the-art techniques. Additionally, qualitative analysis demonstrates that the proposed approach generates sequences with high fitness values, indicating more stable and less time-consuming disassembles. The utilization of this framework allows for the realization of various real-world disassembly applications, thereby making a significant contribution to sustainable practices in EEE industries.

Keywords

disassembly planningsimulated annealingmachine learningoptimizationreinforcement learning
Type
Research Article
Information
AI EDAM , Volume 38 , 2024 , e5
Check for updates
Copyright
© The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alshibli, M, El Sayed, A, Kongar, E, Sobh, TM and Gupta, SM (2016) Disassembly sequencing using Tabu search. Journal of Intelligent and Robotic Systems: Theory and Applications 82(1), 69–79. https://doi.org/10.1007/s10846-015-0289-9.CrossRefGoogle Scholar
Anil Kumar, G, Bahubalendruni, MVAR, Prasad, VSS and Sankaranarayanasamy, K (2021) A multi-layered disassembly sequence planning method to support decision making in de-manufacturing. Sadhana - Academy Proceedings in Engineering Sciences 46(2), 1–16. https://doi.org/10.1007/S12046-021-01622-3.Google Scholar
Azab, A, Ziout, A and ElMaraghy, W (2011) Modeling and optimization for disassembly planning. Jordan Journal of Mechanical and Industrial Engineering 5(1), 1–8.Google Scholar
Bahubalendruni, MVAR and Varupala, VP (2021) Disassembly sequence planning for safe disposal of end-of-life waste electric and electronic equipment. National Academy Science Letters 44(3), 243–247. https://doi.org/10.1007/S40009-020-00994-0.CrossRefGoogle Scholar
Beigy, H and Meybodi, MR (2000) Adaptation of parameters of BP algorithm using learning automata. In Proceedings - Brazilian Symposium on Neural Networks, 2000-January, vol 1, pp. 24–31. IEEE, Rio de Janeiro, Brazil. https://doi.org/10.1109/SBRN.2000.889708CrossRefGoogle Scholar
Beigy, H and Meybodi, MR (2001) Backpropagation algorithm adaptation parameters using learning automata. International Journal of Neural Systems 11(3), 219–228. World Scientific, Singapore. https://doi.org/10.1142/S0129065701000655.CrossRefGoogle ScholarPubMed
Chand, M and Ravi, C (2023) A state-of-the-art literature survey on artificial intelligence techniques for disassembly sequence planning. CIRP Journal of Manufacturing Science and Technology 41, 292–310. https://doi.org/10.1016/j.cirpj.2022.11.017.CrossRefGoogle Scholar
Chang, MML, Nee, AYC and Ong, SK (2020) Interactive AR-assisted product disassembly sequence planning (ARDIS). International Journal of Production Research, 58(16), 4916–4931. https://doi.org/10.1080/00207543.2020.1730462CrossRefGoogle Scholar
De Floriani, L and Nagy, G (1989) Graph model for face-to-face assembly. ICRA, 1, 75–78. https://doi.org/10.1109/robot.1989.99970.Google Scholar
Ghandi, S and Masehian, E (2015) Assembly sequence planning of rigid and flexible parts. Journal of Manufacturing Systems 36, 128–146. https://doi.org/10.1016/jjmsy.2015.05.002.CrossRefGoogle Scholar
Giudice, F and Fargione, G (2007) Disassembly planning of mechanical systems for service and recovery: A genetic algorithms based approach. Journal of Intelligent Manufacturing 18(3), 313–329. https://doi.org/10.1007/s10845-007-0025-9.CrossRefGoogle Scholar
Gunji, BM, Pabba, SK, Rajaram, IRS, Sorakayala, PS, Dubey, A, Deepak, BBVL, Biswal, BB and Bahubalendruni, MVAR (2021) Optimal disassembly sequence generation and disposal of parts using stability graph cut-set method for end of life product. Sadhana - Academy Proceedings in Engineering Sciences 46(1), 21. https://doi.org/10.1007/S12046-020-01525-9.Google Scholar
Guo, J, Zhong, J, Li, Y, Du, B and Guo, S (2019) A hybrid artificial fish swam algorithm for disassembly sequence planning considering setup time. Assembly Automation 39(1), 140–153. https://doi.org/10.1108/AA-12-2017-180.CrossRefGoogle Scholar
Han, HJ, Yu, JM and Lee, DH (2013) Mathematical model and solution algorithms for selective disassembly sequencing with multiple target components and sequence-dependent setups. International Journal of Production Research 51(16), 4997–5010. https://doi.org/10.1080/00207543.2013.788794.CrossRefGoogle Scholar
Hui, W, Dong, X and Guanghong, D (2008) A genetic algorithm for product disassembly sequence planning. Neurocomputing 71(13), 2720–2726. https://doi.org/10.1016/j.neucom.2007.11.042.CrossRefGoogle Scholar
Issaoui, L, Aifaoui, N and Benamara, A (2017) Modelling and implementation of geometric and technological information for disassembly simulation in CAD environment. The International Journal of Advanced Manufacturing Technology 89(5), 1731–1741. https://doi.org/10.1007/s00170-016-9128-9.CrossRefGoogle Scholar
Kheder, M, Trigui, M and Aifaoui, N (2017) Optimization of disassembly sequence planning for preventive maintenance. International Journal of Advanced Manufacturing Technology 90(5–8), 1337–1349. https://doi.org/10.1007/s00170-016-9434-2.CrossRefGoogle Scholar
Kim, HW and Lee, DH (2017) An optimal algorithm for selective disassembly sequencing with sequence-dependent set-ups in parallel disassembly environment. International Journal of Production Research 55(24), 7317–7333. https://doi.org/10.1080/00207543.2017.1342879.CrossRefGoogle Scholar
Kirkpatrick, S, Gelatt, CD and Vecchi, MP (1983) Optimization by simulated annealing. Science 220(4598), 671–680. https://doi.org/10.1126/SCIENCE.220.4598.671.CrossRefGoogle ScholarPubMed
Kuo, TC (2013) Waste electronics and electrical equipment disassembly and recycling using Petri net analysis: Considering the economic value and environmental impacts. Computers and Industrial Engineering 65(1), 54–64. https://doi.org/10.1016/j.cie.2011.12.029.CrossRefGoogle Scholar
Liu, J, Zhou, Z, Pham, DT, Xu, W, Ji, C and Liu, Q (2020) Collaborative optimization of robotic disassembly sequence planning and robotic disassembly line balancing problem using improved discrete bees algorithm in remanufacturing ☆. Robotics and Computer-Integrated Manufacturing, 61, 101829. https://doi.org/10.1016/j.rcim.2019.101829.CrossRefGoogle Scholar
Liu, J, Zhou, Z, Pham, DT, Xu, W, Yan, J, Liu, A, Ji, C and Liu, Q (2018) An improved multi-objective discrete bees algorithm for robotic disassembly line balancing problem in remanufacturing. International Journal of Advanced Manufacturing Technology 97(9–12), 3937–3962. https://doi.org/10.1007/s00170-018-2183-7.CrossRefGoogle Scholar
Luo, Y, Peng, Q and Gu, P (2016) Integrated multi-layer representation and ant colony search for product selective disassembly planning. Computers in Industry 75, 13–26. https://doi.org/10.1016/j.compind.2015.10.011.CrossRefGoogle Scholar
Ma, YS, Jun, HB, Kim, HW and Lee, DH (2011) Disassembly process planning algorithms for end-of-life product recovery and environmentally conscious disposal. International Journal of Production Research 49(23), 7007–7027. https://doi.org/10.1080/00207543.2010.495089.CrossRefGoogle Scholar
Meybodi, MR and Beigy, H (2000) A note on learning automata based schemes for adaptation of BP parameters. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 1983, pp. 145–151. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44491-2_22.Google Scholar
Meybodi, MR and Beigy, H (2002) New learning automata based algorithms for adaptation of backpropagation algorithm parameters. International Journal of Neural Systems 12(1), 45–67. World Scientific, Singapore. https://doi.org/10.1142/S012906570200090X.CrossRefGoogle ScholarPubMed
Mitrouchev, P, Wang, C and Chen, J (2016) Virtual disassembly sequences generation and evaluation. Procedia CIRP 44, 347–352. https://doi.org/10.1016/j.procir.2016.02.001.CrossRefGoogle Scholar
Mitrouchev, P, Wang, C and Chen, J (2017) Disassembly process simulation in virtual reality environment. In Eynard, B, Nigrelli, V, Oliveri, SM, Fajarnes, GP and Rizzuti, S (eds.), Advances on Mechanics, Design Engineering and Manufacturing. Lecture Notes in Mechanical Engineering. Springer International Publishing, pp. 631–638. https://doi.org/10.1007/978-3-319-45781-9_63.CrossRefGoogle Scholar
Osti, F, Ceruti, A, Liverani, A and Caligiana, G (2017) Semi-automatic design for disassembly strategy planning: An augmented reality approach. Procedia Manufacturing, 11, 1481–1488. https://doi.org/10.1016/j.promfg.2017.07.279.CrossRefGoogle Scholar
Ottoni, ALC, Nepomuceno, EG, de Oliveira, MS and de Oliveira, DCR (2021) Reinforcement learning for the traveling salesman problem with refueling. Complex & Intelligent Systems, 8, 2001–2015. https://doi.org/10.1007/s40747-021-00444-4.CrossRefGoogle Scholar
Petri, CA and Reisig, W (2008) Petri net. Scholarpedia 3(4), 6477. https://doi.org/10.4249/scholarpedia.6477.CrossRefGoogle Scholar
Ren, Y, Tian, G, Zhao, F, Yu, D and Zhang, C (2017) Selective cooperative disassembly planning based on multi-objective discrete artificial bee colony algorithm. Engineering Applications of Artificial Intelligence, 64, 415–431. https://doi.org/10.1016/j.engappai.2017.06.025.CrossRefGoogle Scholar
Ren, Y, Zhang, C, Zhao, F, Xiao, H and Tian, G (2018) An asynchronous parallel disassembly planning based on genetic algorithm. European Journal of Operational Research 269(2), 647–660. https://doi.org/10.1016/j.ejor.2018.01.055.CrossRefGoogle Scholar
Smith, S, Smith, G and Chen, WH (2012) Disassembly sequence structure graphs: An optimal approach for multiple-target selective disassembly sequence planning. Advanced Engineering Informatics 26(2), 306–316. https://doi.org/10.1016/j.aei.2011.11.003.CrossRefGoogle Scholar
Sutton, RS and Barto, AG (2018) Reinforcement Learning: An Introduction. MIT Press Ltd, Massachusetts.Google Scholar
Syed Shahul Hameed, AS and Rajagopalan, N (2022) SPGD: Search party gradient descent algorithm, a simple gradient-based parallel algorithm for bound-constrained optimization. Mathematics 10(5), 800. https://doi.org/10.3390/math10050800.CrossRefGoogle Scholar
Syed Shahul Hameed, AS and Rajagopalan, N (2023) MABSearch: The bandit way of learning the learning rate—A harmony between reinforcement learning and gradient descent. National Academy Science Letters 1, 1–6. https://doi.org/10.1007/S40009-023-01292-1.Google Scholar
Tian, G, Ren, Y, Feng, Y, Zhou, MC, Zhang, H and Tan, J (2019a) Modeling and planning for dual-objective selective disassembly using and/or graph and discrete artificial bee Colony. IEEE Transactions on Industrial Informatics 15(4), 2456–2468. https://doi.org/10.1109/TII.2018.2884845.CrossRefGoogle Scholar
Tian, Y, Zhang, X, Liu, Z, Jiang, X and Xue, J (2019b) Product cooperative disassembly sequence and task planning based on genetic algorithm. International Journal of Advanced Manufacturing Technology 105(5–6), 2103–2120. https://doi.org/10.1007/s00170-019-04241-9.CrossRefGoogle Scholar
Tian, G, Zhou, M and Chu, J (2013) A chance constrained programming approach to determine the optimal disassembly sequence. IEEE Transactions on Automation Science and Engineering 10(4), 1004–1013. https://doi.org/10.1109/TASE.2013.2249663.CrossRefGoogle Scholar
Tian, G, Zhou, MC and Li, P (2018) Disassembly sequence planning considering fuzzy component quality and varying operational cost. IEEE Transactions on Automation Science and Engineering 15(2), 748–760. https://doi.org/10.1109/TASE.2017.2690802.CrossRefGoogle Scholar
Tseng, HE, Chang, CC, Lee, SC and Huang, YM (2019) Hybrid bidirectional ant colony optimization (hybrid BACO): An algorithm for disassembly sequence planning. Engineering Applications of Artificial Intelligence, 83, 45–56. https://doi.org/10.1016/j.engappai.2019.04.015.CrossRefGoogle Scholar
Tseng, HE and Lee, SC (2018) Disassembly sequence planning using interactive genetic algorithms. In ICNC-FSKD 2018 - 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, pp. 77–84. IEEE, Huangshan, China. https://doi.org/10.1109/FSKD.2018.8686887.Google Scholar
Tseng, YJ, Yu, FY and Huang, FY (2011) A green assembly sequence planning model with a closed-loop assembly and disassembly sequence planning using a particle swarm optimization method. International Journal of Advanced Manufacturing Technology 57(9–12), 1183–1197. https://doi.org/10.1007/s00170-011-3339-x.CrossRefGoogle Scholar
Ullerich, C (2014) Advanced disassembly planning: Flexible, price-quantity dependent, and multi-period planning approaches. In Advanced Disassembly Planning: Flexible, Price- Quantity Dependent, and Multi-Period Planning Approaches. Springer Gabler, Wiesbaden. https://doi.org/10.1007/978-3-658-03118-3.CrossRefGoogle Scholar
Vongbunyong, S, Kara, S and Pagnucco, M (2012) A framework for using cognitive robotics in disassembly automation. In Leveraging Technology for a Sustainable World - Proceedings of the 19th CIRP Conference on Life Cycle Engineering, pp. 173–178. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29069-5_30.CrossRefGoogle Scholar
Vongbunyong, S, Kara, S and Pagnucco, M (2013) Application of cognitive robotics in disassembly of products. CIRP Annals - Manufacturing Technology 62(1), 31–34. https://doi.org/10.1016/j.cirp.2013.03.037.CrossRefGoogle Scholar
Wang, JF, Liu, JH, Li, SQ and Zhong, YF (2003) Intelligent selective disassembly using the ant colony algorithm. Artificial Intelligence for Engineering Design, Analysis and Manufacturing 17(4), 325–333. https://doi.org/10.1017/S0890060403174045.CrossRefGoogle Scholar
Wu, Y j, Cao, Y and Wang, Q f (2019) Assembly sequence planning method based on particle swarm algorithm. Cluster Computing 22(s1), 835–846. https://doi.org/10.1007/s10586-017-1331-4.CrossRefGoogle Scholar
Xing, Y, Wu, D and Qu, L (2021) Parallel disassembly sequence planning using improved ant colony algorithm. International Journal of Advanced Manufacturing Technology 113(7–8), 2327–2342. https://doi.org/10.1007/s00170-021-06753-9.CrossRefGoogle Scholar
Xu, W, Tang, Q, Liu, J, Liu, Z, Zhou, Z and Pham, DT (2020) Disassembly sequence planning using discrete bees algorithm for human-robot collaboration in remanufacturing. Robotics and Computer-Integrated Manufacturing 62, 101860. https://doi.org/10.1016/j.rcim.2019.101860.CrossRefGoogle Scholar
Yeh, W, Lin, C and Wei, S (2012) Disassembly sequencing problems with stochastic processing time using simplified swarm optimization. International Journal of Innovation, Management and Technology Management 3(3), 226–231.Google Scholar
Zhu, B, Sarigecili, MI and Roy, U (2013) Disassembly information model incorporating dynamic capabilities for disassembly sequence generation. Robotics and Computer- Integrated Manufacturing 29(5), 396–409. https://doi.org/10.1016/j.rcim.2013.03.003.CrossRefGoogle Scholar