Hostname: page-component-76fb5796d-dfsvx Total loading time: 0 Render date: 2024-04-26T08:37:22.276Z Has data issue: false hasContentIssue false
  • English
  • Français

Virtual interaction and manipulation control of a hexacopter through hand gesture recognition from a data glove

Published online by Cambridge University Press:  11 July 2022

Haiming Huang ,
Di’en Wu ,
Zehao Liang ,
Fuchun Sun  and
Mingjie Dong Open the ORCID record for Mingjie Dong [Opens in a new window]
Haiming Huang
Affiliation:
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China
Di’en Wu
Affiliation:
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China
Zehao Liang
Affiliation:
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China
Fuchun Sun
Affiliation:
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China Department of Computer Science and Technology, Tsinghua University, Beijing, China
Mingjie Dong*
Affiliation:
Faculty of Materials and Manufacturing, Beijing University of Technology, Beijing, China
*
*Corresponding author. E-mail: dongmj@bjut.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

The purpose of this study is to realize virtual interaction and manipulation control of a hexacopter based on hand gesture recognition from a designed data glove, to provide an intuitive and visual real-time simulation system for flight control algorithm verification and external control equipment testing. First, the hand gesture recognition from a designed data glove is studied, which can recognize different actions, such as mobile ready, grab, loosen, landing, take-off, and hover. Then, the design of virtual simulation system for hexacopter capture is completed, with the model design of hexacopter and manipulator, and the simulation software design with $CoppeliaSim$ . Finally, virtual simulation experiment of hexacopter grasping and virtual flight control experiment based on data glove are tested, respectively, and quantitatively described. The overall recognition rate is 84.3%, indicating that the data glove produced has the ability to recognize gestures, but its recognition performance is not superior. In gesture recognition, the recognition rate of static gestures is relatively higher than that of dynamic gestures. Among the static gestures, the hover gesture has the highest recognition rate. The average correct rate of static gestures can reach 94%. The lowest recognition rate of dynamic gestures is upward movement, and the average recognition rate of dynamic gestures is 76.1%. The research can be used to remotely operate hexacopter using a data glove in the future and improve the control performance through virtual interaction and manipulation simulation before actual application.

Keywords

virtual simulation controlhexacopterhand gesture recognitiondata gloveCoppeliaSim
Type
Research Article
Information
Robotica , Volume 40 , Issue 12 , December 2022 , pp. 4375 - 4387
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Jiao, R., Chou, W. S., Rong, Y. F. and Dong, M. J., “Anti-disturbance attitude control for quadrotor unmanned aerial vehicle manipulator via fuzzy adaptive sigmoid generalized super-twisting sliding mode observer,” J. Vib. Control 28(11–12), 1251–1266 (2021).CrossRefGoogle Scholar
Xiang, H. and Tian, L., “Development of a low-cost agricultural remote sensing system based on an autonomous unmanned aerial vehicle (UAV),” Biosyst. Eng. 108(2), 174–190 (2011).CrossRefGoogle Scholar
Song, B. D., Park, K. and Kim, J., “Persistent UAV delivery logistics: MILP formulation and efficient heuristic,” Comput. Ind. Eng. 120, 418–428 (2018).Google Scholar
Wu, Q., Zeng, Y. and Zhang, R., “Joint trajectory and communication design for multi-UAV enabled wireless networks,” IEEE. Trans. Wirel. Commun. 17(3), 2109–2121 (2018).CrossRefGoogle Scholar
Jiao, R., Wang, Z. W., Chu, R. H., Dong, M. J., Rong, Y. F. and Chou, W. S., “An intuitive end-to-end human-UAV interaction system for field exploration,” Front. Neurorobot. 13, 1–17.Google Scholar
Kim, S.-J., Lee, D.-Y., Jung, G.-P. and Cho, K.-J., “An origami-inspired, self-locking robotic arm that can be folded flat,” Sci. Robot. 3(16), eaar2915 (2018).CrossRefGoogle ScholarPubMed
Lee, D. J. and Jung, G. P., “Snatcher: A highly mobile chameleon-inspired shooting and rapidly retracting manipulator,” IEEE Robot. Autom. Lett. 5(4), 6097–6104 (2020).CrossRefGoogle Scholar
Liang, J., Cao, J. and Wang, L., “Design of Multi-Mode UAV Human-Computer Interaction System,” 2017 IEEE International Conference on Unmanned Systems (ICUS), Beijing, China (2017) pp. 353–357.Google Scholar
Perera, A. G., Law, Y. W. and Chahl, J., “UAV-GESTURE: A Dataset for UAV Control and Gesture Recognition,” In: Computer Vision - ECCV, 2018 Workshops (Springer, Cham, 2019) pp. 117–128.CrossRefGoogle Scholar
Liu, C. and Sziranyi, T., “Real-time human detection and gesture recognition for on-board UAV rescue,” Sensors 21(6), 1–21 (2021).Google ScholarPubMed
Han, Y., “A low-cost visual motion data glove as an input device to interpret human hand gestures,” IEEE Trans. Consum. Electron. 56(2), 501–509 (2010).CrossRefGoogle Scholar
Fang, B., Sun, F. C., Liu, H. P. and Guo, D., “A novel data glove using inertial and magnetic sensors for motion capture and robotic arm-hand teleoperation,” Ind. Rob. Int. J. 44(2), 155–165 (2017).CrossRefGoogle Scholar
Shi, T., Wang, H. and Zhang, C., “Brain Computer Interface system based on indoor semi-autonomous navigation and motor imagery for Unmanned Aerial Vehicle control,” Expert Syst. Appl. 42(9), 4196–4206 (2015).CrossRefGoogle Scholar
Kumar, P., Verma, J. and Prasad, S., “Hand data glove: A wearable real-time device for human-computer interaction,” Int. J. Adv. Sci. Technol. 43, 15–26 (2012).Google Scholar
Luzhnica, G., Simon, J., Lex, E. and Pammer, V., “A Sliding Window Approach to Natural Hand Gesture Recognition Using a Custom Data Glove,” In: 2016 IEEE Symposium on 3d User Interfaces, Greenville, SC, USA (2016) pp. 81–90.Google Scholar
Dong, M. J., Fang, B., Li, J. F., Sun, F. C. and Liu, H. P., “Wearable sensing devices for upper limbs: A systematic review,” Proc. Inst. Mech. Eng. H. J. Eng. Med. 235(1), 117–130 (2020).CrossRefGoogle ScholarPubMed
Mezzinolu, T. and Karakse, M., “Wearable Glove Based Approach for Human-UAV Interaction,” In: 2020 IEEE International Symposium on Systems Engineering (ISSE), Vienna, Austria (2020) pp. 1–6.Google Scholar
Muezzinoglu, T. and Karakose, M., “An intelligent Human-Unmanned Aerial Vehicle interaction approach in real time based on machine learning using wearable gloves,” Sensors 21(5), 1–24 (2021).CrossRefGoogle ScholarPubMed
Lin, B. S., Lee, I. J., Yang, S. Y., Lo, Y. C., Lee, J. and Chen, J. L., “Design of an inertial-sensor-based data glove for hand function evaluation,” Sensors 18(5), 1–17 (2018).Google ScholarPubMed
Fernando, H. C. T. E., De Silvia, A. T. A., De Zoysa, M. D. C., Dilshan, K. A. D. C. and Munasinghe, S. R., “Modelling, Simulation and Implementation of a Quadrotor UAV,” In: 2013 8th IEEE International Conference on Industrial and Information Systems, Peradeniya, Sri Lanka (2013) pp. 207–212.Google Scholar
Meyer, J., Sendobry, A., Kohlbrecher, S., Klingauf, U. and von Stryk, O., “Comprehensive Simulation of Quadrotor UAVs Using ROS and Gazebo,” In: SIMPAR 2012: Simulation, Modeling, and Programming for Autonomous Robots, Berlin, Heidelberg (2012) pp. 400–411.Google Scholar
Udvardy, P., Beszedes, B., Toth, B., Foldi, A. and Botos, A., “Simulation of Obstacle Avoidance of an UAV,” In: 2020 New Trends in Aviation Development (NTAD), Stary Smokovec, Slovakia (2020) pp. 245–249.CrossRefGoogle Scholar
Li, J., Xu, Y., Ni, J. and Wang, Q., “Glove-based virtual hand grasping for virtual mechanical assembly,” Assembly Autom. 36(5), 349–361 (2016).CrossRefGoogle Scholar