Hostname: page-component-848d4c4894-8kt4b Total loading time: 0 Render date: 2024-06-28T21:48:09.199Z Has data issue: false hasContentIssue false
  • English
  • Français

Low-cost depth/IMU intelligent sensor fusion for indoor robot navigation

Published online by Cambridge University Press:  06 February 2023

Fares Alkhawaja Open the ORCID record for Fares Alkhawaja [Opens in a new window] ,
Mohammad A. Jaradat  and
Lotfi Romdhane Open the ORCID record for Lotfi Romdhane [Opens in a new window]
Fares Alkhawaja*
Affiliation:
Mechatronics Program, American University of Sharjah, Sharjah, UAE
Mohammad A. Jaradat
Affiliation:
Department of Mechanical Engineering, College of Engineering, American University of Sharjah, Sharjah, UAE
Lotfi Romdhane
Affiliation:
Department of Mechanical Engineering, College of Engineering, American University of Sharjah, Sharjah, UAE
*
*Corresponding author. E-mail: fares.alkhawaja@gmail.com
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper presents a mobile robot platform, which performs both indoor and outdoor localization based on an intelligent low-cost depth–inertial fusion approach. The proposed sensor fusion approach uses depth-based localization data to enhance the accuracy obtained by the inertial measurement unit (IMU) pose data through a depth–inertial fusion. The fusion approach is based on feedforward cascade correlation networks (CCNs). The aim of this fusion approach is to correct the drift accompanied by the use of the IMU sensor, using a depth camera. This approach also has the advantage of maintaining the high frequency of the IMU sensor and the accuracy of the depth camera. The estimated mobile robot dynamic states through the proposed approach are deployed and examined through real-time autonomous navigation. It is shown that using both the planned path and the continuous localization approach, the robot successfully controls its movement toward the destination. Several tests were conducted with different numbers of layers and percentages of the training set. It is shown that the best performance is obtained with 12 layers and 80% of the pose data used as a training set for CCN. The proposed framework is then compared to the solution based on fusing the information given by the XSens IMU–GPS sensor and the Kobuki robot built-in odometry solution. As demonstrated in the results, an enhanced performance was achieved with an average Euclidean error of 0.091 m by the CCN, which is lower than the error achieved by the artificial neural network by 56%.

Keywords

navigationpath planningdepth/inertial localizationindoor localizationpose estimationcascade correlation network
Type
Research Article
Information
Robotica , Volume 41 , Issue 6 , June 2023 , pp. 1689 - 1717
Copyright
© The Author(s), 2023. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Zhu, L., Yang, A., Wu, D. and Liu, L., “Survey of indoor positioning technologies and systems,” Commun. Comput. Inf. Sci. 461, 400–409 (2014). doi: 10.1007/978-3-662-45283-7_41.Google Scholar
Liu, H., Darabi, H., Banerjee, P. and Liu, J., “Survey of wireless indoor positioning techniques and systems,” IEEE Trans. Syst. Man Cybern. C Appl. Rev. 37(6), 1067–1080 (2007). doi: 10.1109/TSMCC.2007.905750.CrossRefGoogle Scholar
Alarifi, A., Al-Salman, A. M., Alsaleh, M., Alnafessah, A., Al-Hadhrami, S., Al-Ammar, M. A. and Al-Khalifa, H. S., “Ultra wideband indoor positioning technologies: Analysis and recent advances,” Sensors 16(5), 707 (2016). doi: 10.3390/S16050707.CrossRefGoogle ScholarPubMed
“BLE Beacon Indoor Positioning Systems Basics by Locatify | Locatify.” https://locatify.com/blog/indoor-positioning-systems-ble-beacons/ (accessed Mar. 11, 2022).Google Scholar
Alkhawaja, F., Jaradat, M. and Romdhane, L., “Techniques of Indoor Positioning Systems (IPS): A Survey,” In: 2019 Advances in Science and Engineering Technology International Conferences, ASET 2019 (May 2019) pp. 1–8. doi:10.1109/ICASET.2019.8714291,CrossRefGoogle Scholar
Hauff, P., Reinhardt, M. and Foster, S., “Ultrasound basics,” Handb Exp. Pharmacol. 185(PART 1), 91–107 (2008). doi: 10.1007/978-3-540-72718-7_5.CrossRefGoogle Scholar
Lukianto, C., Hönniger, C. and Sternberg, H., “Pedestrian Smartphone-Based Indoor Navigation Using Ultra Portable Sensory Equipment,” In: 2010 International Conference on Indoor Positioning and Indoor Navigation, IPIN 2010 - Conference Proceedings (2010) pp. 1–5. doi: 10.1109/IPIN.2010.5646697,CrossRefGoogle Scholar
Nilsson, J. O., Gupta, A. K. and Handel, P., “Foot-Mounted Inertial Navigation Made Easy,” In: IPIN 2014 - 2014 International Conference on Indoor Positioning and Indoor Navigation (2014) pp. 24–29. doi: 10.1109/IPIN.2014.7275464,CrossRefGoogle Scholar
Karamat, T. B., Lins, R. G., Givigi, S. N. and Noureldin, A., “Novel EKF-based vision/inertial system integration for improved navigation,” IEEE Trans. Instrum. Meas. 67(1), 116–125 (2018). doi: 10.1109/TIM.2017.2754678.CrossRefGoogle Scholar
Seitz, J., Vaupel, T., Meyer, S., Boronat, J. G. and Thielecke, J., “A Hidden Markov Model for Pedestrian Navigation,” In: Proceedings of the 2010 7th Workshop on Positioning, Navigation and Communication, WPNC’10 (2010) pp. 120–127. doi: 10.1109/WPNC.2010.5650501,CrossRefGoogle Scholar
“Enhancement of Mobile Robot Navigation and Localization”. https://dspace.aus.edu/xmlui/handle/11073/9152 (accessed Mar. 11, 2022).Google Scholar
Labbe, M. and Michaud, F., “Appearance-based loop closure detection for online large-scale and long-term operation,” IEEE Trans. Robot. 29(3), 734–745 (2013). doi: 10.1109/TRO.2013.2242375.CrossRefGoogle Scholar
Song, H., Choi, W. and Kim, H., “Robust vision-based relative-localization approach using an RGB-depth camera and LiDAR sensor fusion,” IEEE Trans. Ind. Electron. 63(6), 3725–3736 (2016). doi: 10.1109/TIE.2016.2521346.CrossRefGoogle Scholar
Al-Mutib, K. and Sabtio, N., “Autonomous mobile robot localization based on RSSI measurements using an RFID sensor and neural network BPANN,” J. King Saud Univ. Comput. Inf. Sci. 25(2), 137–143 (2013). doi: 10.1016/J.JKSUCI.2012.10.001.Google Scholar
Shareef, A., Zhu, Y. and Musavi, M., “Localization Using Neural Networks in Wireless Sensor Networks”.Google Scholar
Abdelgawad, A., “Auto-localization system for indoor mobile robot using RFID fusion,” Robotica 33(9), 1899–1908 (2015). doi: 10.1017/S0263574714001106.CrossRefGoogle Scholar
Malyavej, V., Kumkeaw, W. and Aorpimai, M., “Indoor Robot Localization by RSSI/IMU Sensor Fusion,” In: 2013 10th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, ECTI-CON 2013 (2013) pp. 1–6. doi: 10.1109/ECTICON.2013.6559517.CrossRefGoogle Scholar
Kelly, J. and Sukhatme, G. S., “Visual-inertial sensor fusion: Localization, mapping and sensor-to-sensor self-calibration,” Int. J. Robot. Res. 30(1), 56–79 (2010). doi: 10.1177/0278364910382802.CrossRefGoogle Scholar
Corrales, J. A., Candelas, F. A. and Torres, F., “Hybrid Tracking of Human Operators Using IMU/UWB Data Fusion by a Kalman Filter,” In: HRI 2008 - Proceedings of the 3rd ACM/IEEE International Conference on Human-Robot Interaction: Living with Robots , (2008) pp. 193–200, 10.1145/1349822.1349848,CrossRefGoogle Scholar
Yousuf, S. and Kadri, M. B., “Information fusion of GPS, INS and odometer sensors for improving localization accuracy of mobile robots in indoor and outdoor applications,” Robotica 39(2), 250–276 (2021). doi: 10.1017/S0263574720000351.CrossRefGoogle Scholar
Saadeddin, K., Abdel-Hafez, M. F., Jaradat, M. A. and Jarrah, M. A., “Optimization of intelligent approach for low-cost INS/GPS navigation system,” J. Intell. Robot. Syst. 73(1), 325–348 (2013). doi: 10.1007/S10846-013-9943-2.CrossRefGoogle Scholar
Jaradat, M. A. K. and Abdel-Hafez, M. F., “Enhanced, delay dependent, intelligent fusion for INS/GPS navigation system,” IEEE Sens. J. 14(5), 1545–1554 (2014). doi: 10.1109/JSEN.2014.2298896.CrossRefGoogle Scholar
Jaradat, M. A. K. and Abdel-Hafez, M. F., “Non-linear autoregressive delay-dependent INS/GPS navigation system using neural networks,” IEEE Sens. J. 17(4), 1105–1115 (2017). doi: 10.1109/JSEN.2016.2642040.CrossRefGoogle Scholar
Ragab, M. M., Ragab, H., Givigi, S. and Noureldin, A., “Performance evaluation of neural network based integration of vision and motion sensors for vehicular navigation,” 15 (2019). doi: 10.1117/12.2521694.CrossRefGoogle Scholar
Abdou, H. A., Tsafack, M. D. D., Ntim, C. G. and Baker, R. D., “Predicting creditworthiness in retail banking with limited scoring data,” Knowl Based Syst. 103, 89–103 (2016). doi: 10.1016/J.KNOSYS.2016.03.023.CrossRefGoogle Scholar
Velusamy, K. and Amalraj, R., “Performance of the Cascade Correlation Neural Network for Predicting the Stock Price,” In: Proceedings of the 2017 2nd IEEE International Conference on Electrical, Computer and Communication Technologies, ICECCT 2017 (2017) pp. 1–6. doi: 10.1109/ICECCT.2017.8117824,CrossRefGoogle Scholar
Abid, F. and Zouari, A., “Financial distress prediction using neural networks,” SSRN Electron. J., (2000). doi: 10.2139/SSRN.355980.Google Scholar
Nachev, A., Hogan, M. and Stoyanov, B., “Cascade-Correlation Neural Networks for Breast Cancer Diagnosis”.Google Scholar
Chen, R. C., Lin, Y. C. and Lin, Y. S., “Indoor Position Location Based on Cascade Correlation Networks,” In: 2011 IEEE International Conference on Systems, Man, and Cybernetics (2011) pp. 2295–2300. doi: 10.1109/ICSMC.2011.6084020,CrossRefGoogle Scholar
Chiang, K. W., Noureldin, A. and El-Sheimy, N., “Constructive neural-networks-based MEMS/GPS integration scheme,” IEEE Trans. Aerosp. Electron. Syst. 44(2), 582–594 (2008). doi: 10.1109/TAES.2008.4560208.CrossRefGoogle Scholar
Siegwart, R., Nourbakhsh, I. R. and Scaramuzza, D., “Introduction to autonomous mobile robots,” Choice Rev. Online 49(3), 49–1492 (2011). doi: 10.5860/choice.49-1492.Google Scholar
Ibrahim, M. and Moselhi, O., “Inertial measurement unit based indoor localization for construction applications,” Autom. Constr. 71, 13–20 (2016). doi: 10.1016/J.AUTCON.2016.05.006.CrossRefGoogle Scholar
Premerlani, W. and Bizard, P.. Direction Cosine Matrix IMU: Theory (USA, DIY DRONE, 2009).Google Scholar
Khan, H., Clark, A., Woodward, G. and Lindeman, R. W., “Improved position accuracy of foot-mounted inertial sensor by discrete corrections from vision-based fiducial marker tracking,” Sensors 20(18), 5031 (2020). doi: 10.3390/S20185031.CrossRefGoogle ScholarPubMed
Kang, S. B., Webb, J., Zitnick, C. L. and Kanade, T., “An Active Multibaseline Stereo System with Real-Time Image Acquisition” (1994).Google Scholar
Ruan, T.,“ORB-Slam Performance for Indoor Environment Using Jackal Mobile Robot,” (2020). doi: 10.25394/PGS.12045543.V1.CrossRefGoogle Scholar
Sappa, A. D., Dornaika, F., Ponsa, D., Gerónimo, D. and López, A., “An efficient approach to onboard stereo vision system pose estimation,” IEEE Trans. Intell. Transp. Syst. 9(3), 476–490 (2008). doi: 10.1109/TITS.2008.928237.CrossRefGoogle Scholar
Gupta, M., Agrawal, A., Veeraraghavan, A. and Narasimhan, S. G., “Structured Light 3D Scanning in the Presence of Global Illumination,” In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2011) pp. 713–720, 10.1109/CVPR.2011.5995321,CrossRefGoogle Scholar
“OpenCV: Harris Corner Detection.” https://docs.opencv.org/3.4/dc/d0d/tutorial_py_features_harris.html (accessed June 17, 2022).Google Scholar
Liu, S., Huang, D. and Wang, Y., “Adaptive NMS: Refining Pedestrian Detection in a Crowd”.Google Scholar
Lowe, D. G., “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis. 60(2), 91–110 (2004).CrossRefGoogle Scholar
“GitHub - IntelRealSense/realsense-ros: Intel(R) RealSense(TM) ROS Wrapper for D400 series, SR300 Camera and T265 Tracking Module.” https://github.com/IntelRealSense/realsense-ros (accessed June 17, 2022).Google Scholar
Labbé, M. and Michaud, F., “Online Global Loop Closure Detection for Large-Scale Multi-Session Graph-Based SLAM,” In: IEEE International Conference on Intelligent Robots and Systems (2014) pp. 2661–2666. doi: 10.1109/IROS.2014.6942926,CrossRefGoogle Scholar
1 X 1 Z 1 N Q N Y 1 X 1 Z 1 N* Q N Q, “x 0 y 0 z RECOVERING THE LOCATION OF MOVING OBJECTS: TRANSLATION q* = p* + q q* = p + q + r”.Google Scholar
Malleswaran, M., Vaidehi, V. and Jebarsi, M., “Peformance Comparison of Autonomous Neural Network Based GPS/INS Integration,” In: 3rd International Conference on Advanced Computing, ICoAC 2011 (2011) pp. 401–406. doi: 10.1109/ICOAC.2011.6165209.CrossRefGoogle Scholar
Chiang, K. W. and Chang, H. W., “Intelligent sensor positioning and orientation through constructive neural network-embedded INS/GPS integration algorithms,” Sensors 10(10), 9252–9285 (2010). doi: 10.3390/S101009252.CrossRefGoogle ScholarPubMed
Malleswaran, M., Vaidehi, V., Saravanaselvan, A. and Mohankumar, M., “Performance analysis of various artificial intelligent neural networks for GPS/INS integration,” Appl. Artif. Intell. 27(5), 367–407 (2013). doi: 10.1080/08839514.2013.785793.CrossRefGoogle Scholar
Bikonis, K. and Demkowicz, J., “Data integration from GPS and inertial navigation systems for Pedestrians in Urban Area,” TransNav 7(3), 401–406 (2013). doi: 10.12716/1001.07.03.12.CrossRefGoogle Scholar
Li, S., Gao, Y., Meng, G., Wang, G. and Guan, L., “Accelerometer-based gyroscope drift compensation approach in a dual-axial stabilization platform,” Electronics 8(5), 594 (2019). doi: 10.3390/ELECTRONICS8050594.CrossRefGoogle Scholar
Mummadi, C. K., Leo, F. P. P., Verma, K. D., Kasireddy, S., Scholl, P. M., Kempfle, J. and Laerhoven, K. V., “Real-time and embedded detection of hand gestures with an IMU-based glove,” Informatics 5(2), 28 (2018). doi: 10.3390/INFORMATICS5020028.CrossRefGoogle Scholar
Grewal, M. S. and Andrews, A. P., “Linear Optimal Filters and Predictors”, In: Kalman Filtering: Theory and Practice Using MATLAB (John Wiley & Sons, Ltd, Hoboken, NJ, 2008) pp. 131–181.Google Scholar
Longman, R. W., Kwon, T. and LeVoci, P. A., “Making the learning control problem well posed - Stabilizing intersample error,” Adv. Astronaut. Sci. 123(II), 1143–1162 (2006).Google Scholar
Takanishi, K., Phan, M. Q. and Longman, R. W., “Multiple-model probabilistic design of robust iterative learning controllers,” Trans. North Am. Manuf. Res. Inst. SME 33, 533–540 (2005).Google Scholar
Haddad, A. H., “Applied optimal estimation,” Proc. IEEE 64(4), 574–575 (2008). doi: 10.1109/proc.1976.10175.CrossRefGoogle Scholar
Brown, R. G. and Hwang, P. Y. C., “Introduction to random signals and applied Kalman filtering,” Int. J. Group Psychother. 60(4), 455–460 (1997). doi: 10.1521/ijgp.2010.60.4.455.Google Scholar
Vanicek, P. and Omerbašic, M., “Does a navigation algorithm have to use a Kalman filter?,” Can. Aeronaut. Space J. 45(3), 292–296 (1999).Google Scholar
Mohammed, A. H.. UCGE Reports Optimizing the Estimation Procedure in INS/GPS Integration for Kinematic Applications (The University of Calgary, 1999).Google Scholar
Borkowf, C. B.. Neural Networks: A Comprehensive Foundation, 2nd edition, (Technometrics, 2002). 10.1198/tech.2002.s718 Google Scholar
Bishop, C. M.. Natural Networks for Pattern Recognition (Oxford University, 2007).Google Scholar
, E. Alpaydin, “Gal: Networks that grow when they learn and shrink when they forget,” Int. J. Pattern Recognit. Artif. Intell. 8(1), 391–414 (1994). doi: 10.1142/s021800149400019x.CrossRefGoogle Scholar
Fahlman, S., “The cascade-correlation learning architecture,” Adv. Neural Inf. Process Syst. 2, 524–532 (1990).Google Scholar
“Robot Operating System,” En.wikipedia.org. https://en.wikipedia.org/wiki/Robot_Operating_System (accessed Jan. 15, 2018).Google Scholar
Renawi, A., Jaradat, M. A. and Abdel-Hafez, M., “ROS Validation for Non-Holonomic Differential Robot Modeling and Control: Case Study: Kobuki Robot Trajectory Tracking Controller,” In: 2017 7th International Conference on Modeling, Simulation, and Applied Optimization, ICMSAO 2017 (2017) pp. 1–5. 10.1109/ICMSAO.2017.7934880,CrossRefGoogle Scholar
“About | KOBUKI.” http://kobuki.yujinrobot.com/about2/ (accessed Mar. 11, 2022).Google Scholar
Joseph, E., “Cohen-Coon PID tuning method: A better option to Ziegler Nichols-Pid tuning method,” Online 9(5), (2018), Accessed: Oct. 01, 2022. [Online]. Available:, www.iiste.org Google Scholar
Musat, A., “AndreeaMusat/Cascade-Correlation-Neural-Network,” GitHub, 2018, https://github.com/AndreeaMusat/Cascade-Correlation-Neural-Network, (accessed May 06, 2019).Google Scholar