Hostname: page-component-848d4c4894-cjp7w Total loading time: 0 Render date: 2024-06-30T08:45:46.877Z Has data issue: false hasContentIssue false
  • English
  • Français

FabricFolding: learning efficient fabric folding without expert demonstrations

Published online by Cambridge University Press:  04 March 2024

Can He Open the ORCID record for Can He [Opens in a new window] ,
Lingxiao Meng ,
Zhirui Sun Open the ORCID record for Zhirui Sun [Opens in a new window] ,
Jiankun Wang Open the ORCID record for Jiankun Wang [Opens in a new window]  and
Max Q.-H. Meng
Can He
Affiliation:
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China Jiaxing Research Institute, Southern University of Science and Technology Jiaxing, China
Lingxiao Meng
Affiliation:
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China
Zhirui Sun
Affiliation:
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China Jiaxing Research Institute, Southern University of Science and Technology Jiaxing, China
Jiankun Wang*
Affiliation:
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China Jiaxing Research Institute, Southern University of Science and Technology Jiaxing, China
Max Q.-H. Meng*
Affiliation:
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China
*
Corresponding authors: Jiankun Wang; Email: wangjk@sustech.edu.cn, Max Q.-H. Meng; Email: max.meng@ieee.org
Corresponding authors: Jiankun Wang; Email: wangjk@sustech.edu.cn, Max Q.-H. Meng; Email: max.meng@ieee.org
Get access Rights & Permissions [Opens in a new window]

Abstract

Autonomous fabric manipulation is a challenging task due to complex dynamics and potential self-occlusion during fabric handling. An intuitive method of fabric-folding manipulation first involves obtaining a smooth and unfolded fabric configuration before the folding process begins. However, the combination of quasi-static actions like pick & place and dynamic action like fling proves inadequate in effectively unfolding long-sleeved T-shirts with sleeves mostly tucked inside the garment. To address this limitation, this paper introduces an enhanced quasi-static action called pick & drag, specifically designed to handle this type of fabric configuration. Additionally, an efficient dual-arm manipulation system is designed in this paper, which combines quasi-static (including pick & place and pick & drag) and dynamic fling actions to flexibly manipulate fabrics into unfolded and smooth configurations. Subsequently, once it is confirmed that the fabric is sufficiently unfolded and all fabric keypoints are detected, the keypoint-based heuristic folding algorithm is employed for the fabric-folding process. To address the scarcity of publicly available keypoint detection datasets for real fabric, we gathered images of various fabric configurations and types in real scenes to create a comprehensive keypoint dataset for fabric folding. This dataset aims to enhance the success rate of keypoint detection. Moreover, we evaluate the effectiveness of our proposed system in real-world settings, where it consistently and reliably unfolds and folds various types of fabrics, including challenging situations such as long-sleeved T-shirts with most parts of sleeves tucked inside the garment. Specifically, our method achieves a coverage rate of 0.822 and a success rate of 0.88 for long-sleeved T-shirts folding. Supplemental materials and dataset are available on our project webpage at https://sites.google.com/view/fabricfolding.

Keywords

Fabric unfoldheuristic foldingkeypoint detectionself-supervised learning
Type
Research Article
Information
Robotica , Volume 42 , Issue 4 , April 2024 , pp. 1281 - 1296
Copyright
© The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Fukuda, T., Dobashi, H., Nagano, H., Tazaki, Y., Katayama, R. and Yokokohji, Y., “Jigless assembly of an industrial product by a universal robotic hand mounted on an industrial robot,” Robotica 41(8), 2464–2488 (2023).CrossRefGoogle Scholar
Wu, Y., Fu, Y. and Wang, S., “Global motion planning and redundancy resolution for large objects manipulation by dual redundant robots with closed kinematics,” Robotica 40(4), 1125–1150 (2022).CrossRefGoogle Scholar
Bejjani, W., Leonetti, M. and Dogar, M. R., “Learning image-based receding horizon planning for manipulation in clutter,” Robot. Auton. Syst. 138, 103730 (2021).CrossRefGoogle Scholar
Ren, K., Kavraki, L. E. and Hang, K., “Rearrangement-based Manipulation via Kinodynamic Planning and Dynamic Planning Horizons,” In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2022) pp. 1145–1152.Google Scholar
Triantafyllou, D., Mariolis, I., Kargakos, A., Malassiotis, S. and Aspragathos, N., “A geometric approach to robotic unfolding of garments,” Robot. Auton. Syst. 75, 233–243 (2016).CrossRefGoogle Scholar
Sun, L., Aragon-Camarasa, G., Cockshott, P., Rogers, S. and Siebert, J. P., “A Heuristic-based Approach for Flattening Wrinkled Clothes,” Towards Autonomous Robotic Systems: 14th Annual Conference, TAROS 2013, Oxford, UK, August 28-30, 2013, Revised Selected Papers 14 (2014) pp. 148–160.Google Scholar
Doumanoglou, A., Stria, J., Peleka, G., Mariolis, I., Petrik, V., Kargakos, A., Wagner, L., Hlaváč, V., Kim, T.-K. and Malassiotis, S., “Folding clothes autonomously: A complete pipeline,” IEEE Trans. Robot. 32(6), 1461–1478 (2016).CrossRefGoogle Scholar
Wu, Y., Yan, W., Kurutach, T., Pinto, L. and Abbeel, P., “Learning to Manipulate Deformable Objects without Demonstrations,” In: Proceedings of the 2020 Robotics: Science and Systems (RSS), (2020).CrossRefGoogle Scholar
Lee, R., Ward, D., Dasagi, V., Cosgun, A., Leitner, J. and Corke, P., “Learning Arbitrary-goal Fabric Folding with One Hour of Real Robot Experience,” In: Proceedings of the 2020 Conference on Robot Learning, PMLR 155, pp. 2317–2327 (2021).Google Scholar
Ha, H. and Song, S., “Flingbot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding,” In: Proceedings of the 2022 Conference on Robot Learning, PMLR 164, pp. 24–33 (2022).Google Scholar
Avigal, Y., Berscheid, L., Asfour, T., Kröger, T. and Goldberg, K., “Speedfolding: Learning Efficient Bimanual Folding of Garments,” In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2022) pp. 1–8.Google Scholar
Canberk, A., Chi, C., Ha, H., Burchfiel, B., Cousineau, E., Feng, S. and Song, S., “Cloth Funnels: Canonicalized-Alignment for Multi-purpose Garment Manipulation,” In: 2023 IEEE International Conference of Robotics and Automation (ICRA), (2023) pp. 5872–5879.Google Scholar
Yuba, H., Arnold, S. and Yamazaki, K., “Unfolding of a rectangular cloth from unarranged starting shapes by a dual-armed robot with a mechanism for managing recognition error and uncertainty,” Adv. Robot. 31(10), 544–556 (2017).CrossRefGoogle Scholar
Maitin-Shepard, J., Cusumano-Towner, M., Lei, J. and Abbeel, P., “Cloth Grasp Point Detection based on Multiple-view Geometric Cues with Application to Robotic Towel Folding,” In: 2010 IEEE International Conference on Robotics and Automation (ICRA), (2010) pp. 2308–2315.Google Scholar
Seita, D., Jamali, N., Laskey, M., Berenstein, R., Tanwani, A. K., Baskaran, P., Iba, S., Canny, J. and Goldberg, K.. “Deep transfer learning of pick points on fabric for robot bed-making,” In: 2019 The International Symposium of Robotics Research (ISRR), Springer (2019) pp. 275–290.Google Scholar
Seita, D., Ganapathi, A., Hoque, R., Hwang, M., Cen, E., Tanwani, A. K., Balakrishna, A., Thananjeyan, B., Ichnowski, J., Jamali, N., et al., “Deep Imitation Learning of Sequential Fabric Smoothing from an Algorithmic Supervisor,” In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2020) pp. 9651–9658.Google Scholar
Ganapathi, A., Sundaresan, P., Thananjeyan, B., Balakrishna, A., Seita, D., Grannen, J., Hwang, M., Hoque, R., Gonzalez, J. E., Jamali, N., et al., “Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics,” In: 2021 IEEE International Conference on Robotics and Automation (ICRA), (2021) pp. 11515–11522.Google Scholar
Morrison, D., Tow, A. W., Mctaggart, M., Smith, R., Kelly-Boxall, N., Wade-Mccue, S., Erskine, J., Grinover, R., Gurman, A., Hunn, T., et al., “Cartman: The Low-cost Cartesian Manipulator that Won the Amazon Robotics Challenge,” In: 2018 IEEE International Conference on Robotics and Automation (ICRA), (2018) pp. 7757–7764.Google Scholar
Lim, V., Huang, H., Chen, L. Y., Wang, J., Ichnowski, J., Seita, D., Laskey, M. and Goldberg, K., “Real2sim2real: Self-supervised Learning of Physical Single-step Dynamic Actions for Planar Robot Casting,” In: 2022 IEEE International Conference on Robotics and Automation (ICRA), (2022) pp. 8282–8289.Google Scholar
Wang, C., Wang, S., Romero, B., Veiga, F. and Adelson, E., “Swingbot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation,” In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2020) pp. 5633–5640.Google Scholar
Jangir, R., Alenya, G. and Torras, C., “Dynamic Cloth Manipulation with Deep Reinforcement Learning,” In: 2020 IEEE International Conference on Robotics and Automation (ICRA), (2020) pp. 4630–4636.Google Scholar
Xu, Z., Chi, C., Burchfiel, B., Cousineau, E., Feng, S. and Song, S., “Dextairity: Deformable Manipulation can be a Breeze,” In: Proceedings of the 2022 Robotics: Science and Systems (RSS), (2022).Google Scholar
Cusumano-Towner, M., Singh, A., Miller, S., O’Brien, J. F. and Abbeel, P., “Bringing Clothing into Desired Configurations with Limited Perception,” In: 2011 IEEE International Conference on Robotics and Automation, (2011) pp. 3893–3900.Google Scholar
Camacho, I. G., Sol, J. B.às and Ribas, G. A.à, “Knowledge Representation to Enable High-Level Planning in Cloth Manipulation Tasks,” In: Proceeding of the 2022 ICAPS Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), (2022) p. 9.Google Scholar
Hietala, J., Blanco-Mulero, D., Alcan, G. and Kyrki, V., “Learning Visual Feedback Control for Dynamic Cloth Folding,” In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (2022) pp. 1455–1462.Google Scholar
Matas, J., James, S. and Davison, A. J., “Sim-to-real Reinforcement Learning for Deformable Object Manipulation,” In: Proceedings of the 2018 Conference on Robot Learning, PMLR 87, pp. 734–743 (2018).Google Scholar
Yang, P.-C., Sasaki, K., Suzuki, K., Kase, K., Sugano, S. and Ogata, T., “Repeatable folding task by humanoid robot worker using deep learning,” IEEE Robot. Autom. Lett. 2(2), 397–403 (2016).CrossRefGoogle Scholar
Hoque, R., Seita, D., Balakrishna, A., Ganapathi, A., Tanwani, A. K., Jamali, N., Yamane, K., Iba, S. and Goldberg, K., “Visuospatial foresight for physical sequential fabric manipulation,” Auton. Robot., 1–25 (2022).Google Scholar
Huang, Z., Lin, X. and Held, D., “Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation,” In: Proceedings of the 2022 Robotics: Science and Systems (RSS), (2022).CrossRefGoogle Scholar
Mo, K., Xia, C., Wang, X., Deng, Y., Gao, X. and Liang, B., “Foldsformer: Learning sequential multi-step cloth manipulation with space-time attention,” IEEE Robot. Autom. Lett. 8(2), 760–767 (2022).CrossRefGoogle Scholar
Thach, B., Cho, B. Y., Kuntz, A. and Hermans, T., “Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-view Point Clouds,” In: 2022 IEEE International Conference on Robotics and Automation (ICRA), (2022) pp. 8274–8281.Google Scholar
Weng, T., Bajracharya, S. M., Wang, Y., Agrawal, K. and Held, D., “Fabricflownet: Bimanual Cloth Manipulation with a Flow-based Policy,” In: Proceedings of the 2022 Conference on Robot Learning, PMLR 164, pp.192–202 (2022).Google Scholar
Lips, T., De Gusseme, V.-L., et al., “Learning keypoints from synthetic data for robotic cloth folding,” In: 2nd Workshop on Representing and Manipulating Deformable Objects - ICRA (2022).Google Scholar
Ronneberger, O., Fischer, P. and Brox, T., “U-net: Convolutional Networks for Biomedical Image Segmentation,” In: Proceedings of the 2015 Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer (2015) pp. 234–241.Google Scholar
Wu, J., Sun, X., Zeng, A., Song, S., Lee, J., Rusinkiewicz, S. and Funkhouser, T., “Spatial Action Maps for Mobile Manipulation,” In: Proceedings of the 2020 Robotics: Science and Systems (RSS), (2020).CrossRefGoogle Scholar
Lin, X., Wang, Y., Olkin, J. and Held, D., “Softgym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation,” In: Proceedings of the 2021 Conference on Robot Learning, PMLR 155, pp. 432–448 (2021).Google Scholar
Payer, C., Štern, D., Bischof, H. and Urschler, M., “Integrating spatial configuration into heatmap regression based cnns for landmark localization,” Med. Image Anal. 54, 207–219 (2019).CrossRefGoogle ScholarPubMed