{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T11:41:57Z","timestamp":1744458117898},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,8]]},"abstract":"\n We present a deep learning method for composite and task-driven motion control for physically simulated characters. In contrast to existing data-driven approaches using reinforcement learning that imitate full-body motions, we learn decoupled motions for specific body parts from multiple reference motions simultaneously and directly by leveraging the use of multiple discriminators in a GAN-like setup. In this process, there is no need of any manual work to produce composite reference motions for learning. Instead, the control policy explores by itself how the composite motions can be combined automatically. We further account for multiple task-specific rewards and train a single, multi-objective control policy. To this end, we propose a novel framework for multi-objective learning that adaptively balances the learning of disparate motions from multiple sources and multiple goal-directed control objectives. In addition, as composite motions are typically augmentations of simpler behaviors, we introduce a sample-efficient method for training composite control policies in an incremental manner, where we reuse a pre-trained policy as the meta policy and train a cooperative policy that adapts the meta one for new composite tasks. We show the applicability of our approach on a variety of challenging multi-objective tasks involving both composite motion imitation and multiple goal-directed control. Code is available at\n https:\/\/motion-lab.github.io\/CompositeMotion<\/jats:italic>\n .\n <\/jats:p>","DOI":"10.1145\/3592447","type":"journal-article","created":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T14:29:21Z","timestamp":1690381761000},"page":"1-16","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Composite Motion Learning with Task Control"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-7851-3971","authenticated-orcid":false,"given":"Pei","family":"Xu","sequence":"first","affiliation":[{"name":"Clemson University, Charleston, United States of America"},{"name":"Roblox, Charleston, United States of America"}]},{"ORCID":"http:\/\/orcid.org\/0009-0003-4833-0674","authenticated-orcid":false,"given":"Xiumin","family":"Shang","sequence":"additional","affiliation":[{"name":"University of California, Merced, Merced, United States of America"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-7309-7013","authenticated-orcid":false,"given":"Victor","family":"Zordan","sequence":"additional","affiliation":[{"name":"Roblox, San Mateo, United States of America"},{"name":"Clemson University, San Mateo, United States of America"}]},{"ORCID":"http:\/\/orcid.org\/0009-0000-4315-6556","authenticated-orcid":false,"given":"Ioannis","family":"Karamouzas","sequence":"additional","affiliation":[{"name":"Clemson University, Charleston, United States of America"}]}],"member":"320","published-online":{"date-parts":[[2023,7,26]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 249--258","author":"Abe Yeuhi","year":"2007","unstructured":"Yeuhi Abe, Marco Da Silva, and Jovan Popovi\u0107. 2007. Multiobjective control with frictional contacts. In ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 249--258."},{"key":"e_1_2_2_2_1","volume-title":"Generating Upper-Body Motion for Real-Time Characters Making their Way through Dynamic Environments. Computer Graphics Forum 41, 8","author":"Alvarado Eduardo","year":"2022","unstructured":"Eduardo Alvarado, Damien Rohmer, and Marie-Paule Cani. 2022. Generating Upper-Body Motion for Real-Time Characters Making their Way through Dynamic Environments. Computer Graphics Forum 41, 8 (2022)."},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356536"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274247.3274506"},{"key":"e_1_2_2_5_1","volume-title":"Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555","author":"Chung Junyoung","year":"2014","unstructured":"Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275048"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1781156"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13096"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2008.01134.x"},{"key":"e_1_2_2_10_1","volume-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems. 5755--5762","author":"Lasa Martin De","year":"2009","unstructured":"Martin De Lasa and Aaron Hertzmann. 2009. Prioritized optimization for task-space control. In IEEE\/RSJ International Conference on Intelligent Robots and Systems. 5755--5762."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1781157"},{"key":"e_1_2_2_12_1","volume-title":"Improved training of wasserstein gans. Advances in Neural Information Processing Systems 30","author":"Gulrajani Ishaan","year":"2017","unstructured":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. Advances in Neural Information Processing Systems 30 (2017)."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2767002"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICHR.2004.1442140"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392480"},{"key":"e_1_2_2_16_1","volume-title":"Generative adversarial imitation learning. Advances in Neural Information Processing Systems 29","author":"Ho Jonathan","year":"2016","unstructured":"Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in Neural Information Processing Systems 29 (2016)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3516429"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-007-0200-1"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-30353-1_31"},{"key":"e_1_2_2_20_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/1921427.1921447"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983616"},{"key":"e_1_2_2_23_1","volume-title":"Deep Compliant Control. In ACM SIGGRAPH 2022 Conference Proceedings. Association for Computing Machinery.","author":"Lee Seunghwan","year":"2022","unstructured":"Seunghwan Lee, Phil Sik Chang, and Jehee Lee. 2022. Deep Compliant Control. In ACM SIGGRAPH 2022 Conference Proceedings. Association for Computing Machinery."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459774"},{"key":"e_1_2_2_25_1","first-page":"1","article-title":"Scalable muscle-actuated human simulation and control","volume":"38","author":"Lee Seunghwan","year":"2019","unstructured":"Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. 2019. Scalable muscle-actuated human simulation and control. ACM Transactions on Graphics 38, 4 (2019), 1--13.","journal-title":"ACM Transactions on Graphics"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1781155"},{"key":"e_1_2_2_27_1","volume-title":"Geometric GAN. arXiv preprint arXiv:1705.02894","author":"Lim Jae Hyun","year":"2017","unstructured":"Jae Hyun Lim and Jong Chul Ye. 2017. Geometric GAN. arXiv preprint arXiv:1705.02894 (2017)."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392422"},{"key":"e_1_2_2_29_1","first-page":"1","article-title":"Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning","volume":"37","author":"Liu Libin","year":"2018","unstructured":"Libin Liu and Jessica Hodgins. 2018. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions on Graphics 37, 4 (2018), 1--14.","journal-title":"ACM Transactions on Graphics"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982424"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12571"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2366145.2366173"},{"key":"e_1_2_2_33_1","volume-title":"ACM SIGGRAPH 2010 papers. 1--10","author":"Liu Libin","unstructured":"Libin Liu, KangKang Yin, Michiel van de Panne, Tianjia Shao, and Weiwei Xu. 2010. Sampling-based contact-rich motion control. In ACM SIGGRAPH 2010 papers. 1--10."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1576246.1531386"},{"key":"e_1_2_2_35_1","volume-title":"Isaac Gym: High performance GPU-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470","author":"Makoviychuk Viktor","year":"2021","unstructured":"Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. 2021. Isaac Gym: High performance GPU-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470 (2021)."},{"key":"e_1_2_2_36_1","volume-title":"Neural Probabilistic Motor Primitives for Humanoid Control. In International Conference on Learning Representations.","author":"Merel Josh","year":"2019","unstructured":"Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. 2019. Neural Probabilistic Motor Primitives for Humanoid Control. In International Conference on Learning Representations."},{"key":"e_1_2_2_37_1","volume-title":"Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201","author":"Merel Josh","year":"2017","unstructured":"Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. 2017. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201 (2017)."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392474"},{"key":"e_1_2_2_39_1","volume-title":"Robotics: Science and Systems","volume":"4","author":"Mordatch Igor","year":"2014","unstructured":"Igor Mordatch and Emo Todorov. 2014. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, Vol. 4."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185539"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1576246.1531387"},{"key":"e_1_2_2_42_1","volume-title":"Multiagent manipulation via locomotion using hierarchical sim2real. arXiv preprint arXiv:1908.05224","author":"Nachum Ofir","year":"2019","unstructured":"Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. 2019. Multiagent manipulation via locomotion using hierarchical sim2real. arXiv preprint arXiv:1908.05224 (2019)."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356501"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275014"},{"key":"e_1_2_2_45_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3072959.3073602","article-title":"Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning","volume":"36","author":"Peng Xue Bin","year":"2017","unstructured":"Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel van de Panne. 2017. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics 36, 4 (2017), 1--13.","journal-title":"ACM Transactions on Graphics"},{"key":"e_1_2_2_46_1","first-page":"3686","article-title":"MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies","volume":"32","author":"Peng Xue Bin","year":"2019","unstructured":"Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. Advances in Neural Information Processing Systems 32 (2019), 3686--3697.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530110"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459670"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359566.3360071"},{"key":"e_1_2_2_50_1","volume-title":"Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347","author":"Schulman John","year":"2017","unstructured":"John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2945078.2945107"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1275808.1276511"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3355089.3356505"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459881"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2011.30"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1002\/cav.260"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386025"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2014.6907001"},{"key":"e_1_2_2_59_1","volume-title":"Learning values across many orders of magnitude. Advances in Neural Information Processing Systems 29","author":"van Hasselt Hado P","year":"2016","unstructured":"Hado P van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver. 2016. Learning values across many orders of magnitude. Advances in Neural Information Processing Systems 29 (2016)."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601097.2601192"},{"key":"e_1_2_2_61_1","volume-title":"Unicon: Universal neural controller for physics-based character motion. arXiv preprint arXiv:2011.15119","author":"Wang Tingwu","year":"2020","unstructured":"Tingwu Wang, Yunrong Guo, Maria Shugrina, and Sanja Fidler. 2020. Unicon: Universal neural controller for physics-based character motion. arXiv preprint arXiv:2011.15119 (2020)."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392381"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459761"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530067"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272127.3275023"},{"key":"e_1_2_2_66_1","volume-title":"ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 113--118","author":"Wu Chun-Chih","year":"2010","unstructured":"Chun-Chih Wu and Victor Zordan. 2010. Goal-directed stepping with momentum control. In ACM SIGGRAPH\/Eurographics Symposium on Computer Animation. 113--118."},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14115"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3480148"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3451254"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/CW.2015.26"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/1833349.1778811"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2009.01625.x"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276509"},{"key":"e_1_2_2_74_1","volume-title":"The surprising effectiveness of PPO in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955","author":"Yu Chao","year":"2021","unstructured":"Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. 2021. The surprising effectiveness of PPO in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 (2021)."},{"key":"e_1_2_2_75_1","first-page":"1","article-title":"Learning symmetric and low-energy locomotion","volume":"37","author":"Yu Wenhao","year":"2018","unstructured":"Wenhao Yu, Greg Turk, and C Karen Liu. 2018. Learning symmetric and low-energy locomotion. ACM Transactions on Graphics 37, 4 (2018), 1--12.","journal-title":"ACM Transactions on Graphics"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2330610"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/545261.545276"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3592447","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T17:48:48Z","timestamp":1717436928000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592447"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,26]]},"references-count":77,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.1145\/3592447"],"URL":"https:\/\/doi.org\/10.1145\/3592447","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,26]]},"assertion":[{"value":"2023-07-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}