Skip to main content Accessibility help
×
Home

Robust Semantic Mapping in Challenging Environments

  • Jiyu Cheng (a1), Yuxiang Sun (a2) and Max Q.-H. Meng (a1)

Summary

Visual simultaneous localization and mapping (visual SLAM) has been well developed in recent decades. To facilitate tasks such as path planning and exploration, traditional visual SLAM systems usually provide mobile robots with the geometric map, which overlooks the semantic information. To address this problem, inspired by the recent success of the deep neural network, we combine it with the visual SLAM system to conduct semantic mapping. Both the geometric and semantic information will be projected into the 3D space for generating a 3D semantic map. We also use an optical-flow-based method to deal with the moving objects such that our method is capable of working robustly in dynamic environments. We have performed our experiments in the public TUM dataset and our recorded office dataset. Experimental results demonstrate the feasibility and impressive performance of the proposed method.

Copyright

Corresponding author

*Corresponding author. E-mail: max.meng@cuhk.edu.hk

References

Hide All
1.Klein, G. and Murray, D., “Parallel Tracking and Mapping for Small AR Workspaces,” ISMAR 2007 Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan (2007) pp. 225234.
2.Engel, J., Schöps, T. and Cremers, D., “LSD-SLAM: Large-Scale Direct Monocular Slam,” European Conference on Computer Vision, Zurich, Switzerland (2014) pp. 834849.
3.Mur-Artal, R. and Tardós, J. D., “ORB-SLAM2: An open-source slam system for monocular, stereo, and RGB-D cameras,” IEEE Trans. Robot. 33 (5), 12551262 (2017).
4.Whelan, T., Leutenegger, S., Salas-Moreno, R., Glocker, B. and Davison, A., “Elasticfusion: Dense Slam Without a Pose Graph.” Robotics: Science and Systems: A Robotics Conferences, Rome, Italy (2015).
5.Kerl, C., Sturm, J. and Cremers, D., “Dense Visual Slam for RGB-D Cameras,” 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan (2013) pp. 21002106.
6.Wang, C., Meng, L., She, S., Mitchell, I. M., Li, T., Tung, F., Wan, W., Meng, M., de Silva, C. W. and Clarence, W., “Autonomous Mobile Robot Navigation in Uneven and Unstructured Indoor Environments,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada (2017) pp. 109116.
7.Zhu, D., Li, T., Ho, D., Wang, C. and Meng, M. Q.-H., “Deep Reinforcement Learning Supervised Autonomous Exploration in Office Environments,” 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia (2018) pp. 75487555.
8.Sun, Y., Zuo, W. and Liu, M., “Rtfnet: RGB-thermal fusion network for semantic segmentation of urban scenes,” IEEE Robot. Autom. Lett., 25762583 (2019).
9.Wang, C., Cheng, J., Wang, J., Li, X., and Meng, M. Q.-H., “Efficient object search with belief road map using mobile robot,” IEEE Robot. Autom. Lett. 3 (4), 30813088 (2018).
10.Cheng, J., Cheng, H., Meng, M. Q.-H. and Zhang, H., “Autonomous Navigation by Mobile Robots in Human Environments: A Survey,” 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia (2018) pp. 19811986.
11.Raguram, R., Chum, O., Pollefeys, M., Matas, J. and Frahm, J.-M., “USAC: a universal framework for random sample consensus.” IEEE Trans. Pattern Anal. Mach. Intell. 35 (8), 20222038 (2013).
12.Redmon, J., Divvala, S., Girshick, R. and Farhadi, A., “You Only Look Once: Unified, Real-Time Object Detection,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA (2016) pp. 779788.
13.Long, J., Shelhamer, E. and Darrell, T., “Fully Convolutional Networks for Semantic Segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA (2015) pp. 34313440.
14.Mur-Artal, R. and Tardos, J. D., “ORB-SLAM2: an open-source slam system for monocular, stereo and RGB-D cameras,” IEEE Transactions on Robotics 33(5), 12551262 (2017).
15.Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C. and Torr, P. H., “Conditional Random Fields as Recurrent Neural Networks,” Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile (2015) 15291537.
16.Cheng, J., Sun, Y. and Meng, M. Q.-H., “A Dense Semantic Mapping System Based on CRF-RNN Network,” 2017 18th International Conference on Advanced Robotics (ICAR), Hong Kong, China (2017) pp. 589594.
17.Bloesch, M., Omari, S., Hutter, M. and Siegwart, R., “Robust Visual Inertial Odometry using a Direct EKFBased Approach,” 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany (2015) pp. 298304.
18.Usenko, V., Engel, J., Stückler, J. and Cremers, D., “Direct Visual-Inertial Odometry with Stereo Cameras,” 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden (2016) pp. 18851892.
19.Kim, D.-H., Han, S.-B. and Kim, J.-H., “Visual odometry algorithm using an RGB-D sensor and IMU in a highly dynamic environment,” In: Robot Intelligence Technology and Applications 3, (Springer, Cham, 2015) pp. 1126.
20.Sun, Y., Liu, M. and Meng, M. Q.-H., “Active perception for foreground segmentation: An RGB-D databased background modeling method,” IEEE Trans. Autom. Sci. Eng. (2019). Early Access.
21.Kim, D.-H. and Kim, J.-H., “Effective background model-based RGB-D dense visual odometry in a dynamic environment,” IEEE Trans. Robot. 32(6), 15651573 (2016).
22.Sun, Y., Liu, M. and Meng, M. Q.-H., “Improving RGB-D slam in dynamic environments: A motion removal approach,” Robot. Autonom. Syst. 89, 110122 (2017).
23.Li, S. and Lee, D., ‘RGB-D slam in dynamic environments using static point weighting,” IEEE Robot. Autom. Lett. 2(4), 22632270 (2017).
24.Sun, Y., Liu, M. and Meng, M. Q.-H., “Motion removal for reliable RGB-D slam in dynamic environments,” Robot. Autonom. Syst. 108, 115128 (2018).
25.Zou, D. and Tan, P., “Coslam: Collaborative visual slam in dynamic environments,” IEEE Trans. Pattern Anal. Machine Intell. 35 (2), 354366 (2013).
26.Wang, Y. and Huang, S., “Motion Segmentation Based Robust RGB-D Slam,” 2014 11th World Congress on Intelligent Control and Automation (WCICA), Shenyang, China (2014) pp. 31223127.
27.Terashima, T. and Hasegawa, O., “A Visual-Slam for First Person Vision andMobile Robots,” 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), Nagoya, Japan (2017) pp. 7376.
28.Cheng, J., Sun, Y., Chi, W., Wang, C., Cheng, H. and Meng, M. Q.-H., “An Accurate Localization Scheme for Mobile Robots Using Optical Flow in Dynamic Environments,” 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), Kuala Lumpur, Malaysia (2018) pp. 723728.
29.Panchpor, A. A., Shue, S. and Conrad, J. M., “A Survey of Methods for Mobile Robot Localization and Mapping in Dynamic Indoor Environments,” 2018 Conference on Signal Processing and Communication Engineering Systems (SPACES), Vijayawada, India (2018) pp. 138144.
30.Hermans, A., Floros, G. and Leibe, B., “Dense 3d Semantic Mapping of Indoor Scenes from RGB-D Images,” 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China (2014) pp. 26312638.
31.Salas-Moreno, R. F., Newcombe, R. A., Strasdat, H., Kelly, P. H. and Davison, A. J., “Slam++: Simultaneous Localisation and Mapping at the Level of Objects,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, Oregon, USA (2013) 13521359.
32.Sünderhauf, N., Pham, T. T., Latif, Y., Milford, M. and Reid, I., “Meaningful Maps – Object-Oriented Semantic Mapping,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, Canada (2017) pp. 50795085.
33.Bowman, S. L., Atanasov, N., Daniilidis, K. and Pappas, G. J., “Probabilistic Data Association for Semantic Slam,” 2017 IEEE International Conference on Robotics and Automation (ICRA), Marina Bay Sands, Singapore (2017) pp. 17221729.
34.Gan, L., Jadidi, M. G., Parkison, S. A. and Eustice, R. M., “Sparse Bayesian inference for dense semantic mapping,” arXiv preprint arXiv:1709.07973, (2017).
35.Triggs, B., McLauchlan, P. F., Hartley, R. I. and Fitzgibbon, A. W., “Bundle Adjustment – A Modern Synthesis,” International Workshop on Vision Algorithms, Corfu, Greece (1999) pp. 298372.
36.Nistér, D., “An efficient solution to the five-point relative pose problem,” IEEE Trans. Pattern Anal. Machine Intell. 26(6), 756770 (2004).
37.Baker, S. and Matthews, I., “Lucas-kanade 20 years on: A Unifying Framework,” Int. J. Comp. Vision 56(3), 221255 (2004).
38.Sturm, J., Engelhard, N., Endres, F., Burgard, W. and Cremers, D., “A Benchmark for the Evaluation of RGBD Slam Systems,” 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Algarve, Portugal (2012) pp. 573580.
39.Song, S., Lichtenberg, S. P. and Xiao, J., “Sun RGB-D: A RGB-D Scene Understanding Benchmark Suite,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA (2015) pp. 567576.
40.Silberman, N. and Fergus, R., “Indoor Scene Segmentation Using a Structured Light Sensor,” 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain (2011) pp. 601608.
41.Xiao, J., Owens, A., and Torralba, A., “Sun3d: A Database of Big Spaces Reconstructed using SFM and Object Labels,” Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia (2013) pp. 16251632.
42.Song, S. and Xiao, J., “Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA (2016) pp. 808816.
43.Hua, B.-S., Pham, Q.-H., Nguyen, D. T., Tran, M.-K., Yu, L.-F. and Yeung, S.-K., “Scenenn: A Scene Meshes Dataset with Annotations,” 2016 Fourth International Conference on 3D Vision (3DV), Stanford, California, USA (2016) pp. 92101.
44.Geiger, A., Lenz, P. and Urtasun, R., “AreWe Ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Providence, Rhode Island, USA (2012) pp. 33543361.

Keywords

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed