{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T18:22:25Z","timestamp":1732040545052},"reference-count":46,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T00:00:00Z","timestamp":1621900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"Soft robotic grippers are increasingly desired in applications that involve grasping of complex and deformable objects. However, their flexible nature and non-linear dynamics makes the modelling and control difficult. Numerical techniques such as Finite Element Analysis (FEA) present an accurate way of modelling complex deformations. However, FEA approaches are computationally expensive and consequently challenging to employ for real-time control tasks. Existing analytical techniques simplify the modelling by approximating the deformed gripper geometry. Although this approach is less computationally demanding, it is limited in design scope and can lead to larger estimation errors. In this paper, we present a learning based framework that is able to predict contact forces as well as stress distribution from soft Fin Ray Effect (FRE) finger images in real-time. These images are used to learn internal representations for deformations using a deep neural encoder, which are further decoded to contact forces and stress maps using separate branches. The entire network is jointly learned in an end-to-end fashion. In order to address the challenge of having sufficient labelled data for training, we employ FEA to generate simulated images to supervise our framework. This leads to an accurate prediction, faster inference and availability of large and diverse data for better generalisability. Furthermore, our approach is able to predict a detailed stress distribution that can guide grasp planning, which would be particularly useful for delicate objects. Our proposed approach is validated by comparing the predicted contact forces to the computed ground-truth forces from FEA as well as real force sensor. We rigorously evaluate the performance of our approach under variations in contact point, object material, object shape, viewing angle, and level of occlusion.<\/jats:p>","DOI":"10.3389\/frobt.2021.631371","type":"journal-article","created":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T15:35:19Z","timestamp":1621956919000},"update-policy":"http:\/\/dx.doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["A Deep Learning Method for Vision Based Force Prediction of a Soft Fin Ray Gripper Using Simulation Data"],"prefix":"10.3389","volume":"8","author":[{"given":"Daniel","family":"De Barrie","sequence":"first","affiliation":[]},{"given":"Manjari","family":"Pandya","sequence":"additional","affiliation":[]},{"given":"Harit","family":"Pandya","sequence":"additional","affiliation":[]},{"given":"Marc","family":"Hanheide","sequence":"additional","affiliation":[]},{"given":"Khaled","family":"Elgeneidy","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2021,5,25]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"1619","DOI":"10.1109\/TMECH.2018.2844327","article-title":"Grasp and stress analysis of an underactuated finger for proprioceptive tactile sensing","volume":"23","author":"Abdeetedal","year":"2018","journal-title":"IEEE\/ASME Trans. Mechatron"},{"key":"B2","doi-asserted-by":"publisher","first-page":"4469","DOI":"10.3390\/s20164469","article-title":"Dynamic-vision-based force measurements using convolutional recurrent neural networks","volume":"20","author":"Baghaei Naeini","year":"2020","journal-title":"Sensors"},{"key":"B3","unstructured":"BannaschR.\n KnieseL.\n Manipulator tool and holding and\/or expanding tool with at least one manipulator tool. US Patent: US83334172012"},{"key":"B4","doi-asserted-by":"crossref","first-page":"713","DOI":"10.1109\/ICCA.2019.8899497","article-title":"Geometric conformity study of a fin ray gripper utilizing active haptic control","volume-title":"2019 IEEE 15th International Conference on Control and Automation (ICCA)","author":"Basson","year":"2019"},{"key":"B5","doi-asserted-by":"publisher","first-page":"128","DOI":"10.7166\/29-1-1874","article-title":"Testing flexible grippers for geometric and surface grasping conformity in reconfigurable assembly systems","volume":"29","author":"Basson","year":"2018","journal-title":"South Afr. J. Indus. Eng"},{"key":"B6","doi-asserted-by":"publisher","first-page":"18809","DOI":"10.1073\/pnas.1003250107","article-title":"Universal robotic gripper based on the jamming of granular material","volume":"107","author":"Brown","year":"2010","journal-title":"Proc. Natl. Acad. Sci. U.S.A"},{"key":"B7","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1109\/RoboSoft48309.2020.9115970","article-title":"Soft robotic module for sensing and controlling contact force","volume-title":"2020 3rd IEEE International Conference on Soft Robotics","author":"Buso","year":"2020"},{"key":"B8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/frobt.2016.00070","article-title":"Fin ray\u00aeeffect inspired soft robotic gripper: from the robosoft grand challenge toward optimization","volume":"3","author":"Crooks","year":"2016","journal-title":"Front. Robot. AI"},{"key":"B9","doi-asserted-by":"crossref","first-page":"779","DOI":"10.1109\/RoboSoft48309.2020.9115969","article-title":"Structural optimization of adaptive soft fin ray fingers with variable stiffening capability","volume-title":"2020 3rd IEEE International Conference on Soft Robotics (RoboSoft)","author":"Elgeneidy","year":"2020"},{"key":"B10","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1109\/ROBOSOFT.2019.8722715","article-title":"Characterising 3D-printed soft fin ray robotic fingers with layer jamming capability for delicate grasping","volume-title":"2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)","author":"Elgeneidy","year":"2019"},{"key":"B11","first-page":"43","article-title":"Optimising soft fin ray robotic fingers using finite element analysis to reduce object slippage","volume-title":"UK RAS","author":"Emerson","year":"2020"},{"key":"B12","year":"2014","journal-title":"MultiChoiceGripper One Gripper for the Widest Variety of Shapes"},{"key":"B13","year":"2020","journal-title":"The Fin Ray-Paper Kit"},{"key":"B14","doi-asserted-by":"publisher","first-page":"692","DOI":"10.3390\/s18030692","article-title":"Enhancing perception with tactile object recognition in adaptive grippers for human-robot interaction","volume":"18","author":"Gandarias","year":"2018","journal-title":"Sensors"},{"key":"B15","doi-asserted-by":"publisher","first-page":"873","DOI":"10.1109\/LRA.2018.2792684","article-title":"Use of deep learning for characterization of microfluidic soft sensors","volume":"3","author":"Han","year":"2018","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B16","first-page":"6109","article-title":"Universal soft pneumatic robotic gripper with variable effective length","volume-title":"Chinese Control Conference, CCC","author":"Hao","year":"2016"},{"key":"B17","first-page":"3619","article-title":"Design and development of a bio-inspired, under-actuated soft gripper","volume-title":"Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS","author":"Hassan","year":"2015"},{"key":"B18","unstructured":"A robot for harvesting sweet-pepper in greenhouses\n HemmingJ.\n BacC. W.\n TuijlB.\n BarthR.\n BontsemaJ.\n PekkerietE.\n Proceedings International Conference of Agricultural Engineering2014"},{"key":"B19","first-page":"12627","article-title":"Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"James","year":"2019"},{"key":"B20","doi-asserted-by":"publisher","first-page":"717","DOI":"10.1109\/ROBOSOFT.2019.8722804","article-title":"Dynamically closed-loop controlled soft robotic arm using a reduced order finite element model with state observer","author":"Katzschmann","year":"2019","journal-title":"RoboSoft 2019 - 2019 IEEE International Conference on Soft Robotics"},{"key":"B21","first-page":"3883","article-title":"Development of a soft robotic glove with high gripping force using force distributing compliant structures","volume-title":"IEEE International Conference on Intelligent Robots and Systems","author":"Kim","year":"2017"},{"key":"B22","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"B23","doi-asserted-by":"publisher","first-page":"469","DOI":"10.1109\/LRA.2019.2890897","article-title":"Stiffness control of deformable robots using finite element modeling","volume":"4","author":"Koehler","year":"2019","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B24","doi-asserted-by":"publisher","first-page":"2550","DOI":"10.1109\/ICRA.2015.7139541","article-title":"Real-time control of soft-robots using asynchronous finite element modeling","author":"Largilliere","year":"2015","journal-title":"Proceedings - IEEE International Conference on Robotics and Automation"},{"key":"B25","doi-asserted-by":"publisher","first-page":"20170844","DOI":"10.1098\/rsif.2017.0844","article-title":"A deep learning approach to estimate stress distribution: a fast and accurate surrogate of finite-element analysis","volume":"15","author":"Liang","year":"2018","journal-title":"J. R. Soc. Interface"},{"key":"B26","first-page":"5","article-title":"Real-time curvature estimation of printable soft gripper using electro-conductive yarn","volume-title":"2017 IEEE International Conference on Real-Time Computing and Robotics, RCAR 2017","author":"Matsuno","year":"2018"},{"key":"B27","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1108\/IR-12-2012-452","article-title":"Development of a new type of passively adaptive compliant gripper","volume":"40","author":"Petkovi\u0107","year":"2013","journal-title":"Indus. Robot"},{"key":"B28","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1016\/j.asoc.2014.04.037","article-title":"Adaptive neuro-fuzzy prediction of grasping object weight for passively compliant gripper","volume":"22","author":"Petkovi\u0107","year":"2014","journal-title":"Appl. Soft Comput. J"},{"key":"B29","doi-asserted-by":"publisher","first-page":"104","DOI":"10.24352\/UB.OVGU-2018-010","article-title":"Experimental determination of elastic and rupture properties of printed Ninjaflex","volume":"38","author":"Reppel","year":"2018","journal-title":"Tech. Mech"},{"key":"B30","first-page":"234","article-title":"U-net: Convolutional networks for biomedical image segmentation","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Ronneberger","year":"2015"},{"key":"B31","doi-asserted-by":"publisher","first-page":"467","DOI":"10.1038\/nature14543","article-title":"Design, fabrication and control of soft robots","volume":"521","author":"Rus","year":"2015","journal-title":"Nature"},{"key":"B32","first-page":"504","article-title":"Online shape estimation based on tactile sensing and deformation modeling for robot manipulation","volume-title":"IEEE International Conference on Intelligent Robots and Systems","author":"Sanchez","year":"2018"},{"key":"B33","doi-asserted-by":"crossref","first-page":"4389","DOI":"10.1109\/IROS45743.2020.9341285","article-title":"Learning the sense of touch in simulation: a sim-to-real strategy for vision-based tactile sensing","volume-title":"2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS)","author":"Sferrazza","year":"2020"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/0278364920913926","article-title":"Modeling and analysis of soft robotic fingers using the fin ray effect","volume":"39","author":"Shan","year":"2020","journal-title":"Int. J. Robot. Res"},{"key":"B35","doi-asserted-by":"crossref","first-page":"10075","DOI":"10.1109\/ICRA40945.2020.9197369","article-title":"Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing","volume-title":"2020 IEEE International Conference on Robotics and Automation (ICRA)","author":"She","year":"2020"},{"key":"B36","doi-asserted-by":"publisher","first-page":"30","DOI":"10.3389\/frobt.2019.00030","article-title":"Design considerations for 3D printed, soft, multimaterial resistive sensors for soft robotics","volume":"6","author":"Shih","year":"2019","journal-title":"Front. Robot"},{"key":"B37","doi-asserted-by":"publisher","first-page":"eaaz9239","DOI":"10.1126\/scirobotics.aaz9239","article-title":"Electronic skins and machine learning for intelligent soft robots","volume":"5","author":"Shih","year":"2020","journal-title":"Sci. Robot"},{"key":"B38","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1101\/2020.06.15.151670","article-title":"Fem-based mechanics modeling of bio-inspired compliant mechanisms for medical applications","volume":"2","author":"Sun","year":"2020","journal-title":"IEEE Transactions on Medical Robotics and Bionics"},{"key":"B39","doi-asserted-by":"publisher","first-page":"eaav1488","DOI":"10.1126\/scirobotics.aav1488","article-title":"Soft robot perception using embedded soft sensors and recurrent neural networks","volume":"4","author":"Thuruthel","year":"2019","journal-title":"Sci. Robot"},{"key":"B40","article-title":"Soft robot optimal control via reduced order finite element models","author":"Tonkens","year":"2020","journal-title":"arXiv: abs\/2011.02092"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1800541","DOI":"10.1002\/advs.201800541","article-title":"Toward perceptive soft robots: progress and challenges","volume":"5","author":"Wang","year":"2018","journal-title":"Adv. Sci"},{"key":"B42","doi-asserted-by":"publisher","first-page":"624","DOI":"10.1109\/LRA.2017.2650149","article-title":"Soft gripper dynamics using a line-segment model with an optimization-based parameter identification method","volume":"2","author":"Wang","year":"2017","journal-title":"IEEE Robot. Autom. Lett"},{"key":"B43","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1089\/soro.2017.0052","article-title":"The TacTip family: soft optical tactile sensors with 3D-printed biomimetic morphologies","volume":"5","author":"Ward-Cherrier","year":"2018","journal-title":"Soft Robot"},{"key":"B44","first-page":"532","article-title":"Deceptionnet: Network-driven domain randomization","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Zakharov","year":"2019"},{"key":"B45","first-page":"5509","article-title":"Kinematic modeling and observer based control of soft robot using real-time finite element method","volume-title":"IEEE International Conference on Intelligent Robots and Systems, 2016","author":"Zhang","year":"2016"},{"key":"B46","doi-asserted-by":"publisher","first-page":"3020","DOI":"10.3390\/app10093020","article-title":"Control-based 4D printing: adaptive 4D-printed systems","volume":"10","author":"Zolfagharian","year":"2020","journal-title":"Appl. Sci"}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2021.631371\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T15:35:43Z","timestamp":1621956943000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2021.631371\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,25]]},"references-count":46,"alternative-id":["10.3389\/frobt.2021.631371"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2021.631371","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,25]]}}}