{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,22]],"date-time":"2025-02-22T00:38:06Z","timestamp":1740184686201,"version":"3.37.3"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"18","license":[{"start":{"date-parts":[[2018,4,25]],"date-time":"2018-04-25T00:00:00Z","timestamp":1524614400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01GM097576"],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["P30CA008748"],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,9,15]]},"abstract":"Abstract<\/jats:title>\n \n Motivation<\/jats:title>\n Cell movement in the early phase of Caenorhabditis elegans development is regulated by a highly complex process in which a set of rules and connections are formulated at distinct scales. Previous efforts have demonstrated that agent-based, multi-scale modeling systems can integrate physical and biological rules and provide new avenues to study developmental systems. However, the application of these systems to model cell movement is still challenging and requires a comprehensive understanding of regulatory networks at the right scales. Recent developments in deep learning and reinforcement learning provide an unprecedented opportunity to explore cell movement using 3D time-lapse microscopy images.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n We present a deep reinforcement learning approach within an agent-based modeling system to characterize cell movement in the embryonic development of C.elegans. Our modeling system captures the complexity of cell movement patterns in the embryo and overcomes the local optimization problem encountered by traditional rule-based, agent-based modeling that uses greedy algorithms. We tested our model with two real developmental processes: the anterior movement of the Cpaaa cell via intercalation and the rearrangement of the superficial left-right asymmetry. In the first case, the model results suggested that Cpaaa\u2019s intercalation is an active directional cell movement caused by the continuous effects from a longer distance (farther than the length of two adjacent cells), as opposed to a passive movement caused by neighbor cell movements. In the second case, a leader-follower mechanism well explained the collective cell movement pattern in the asymmetry rearrangement. These results showed that our approach to introduce deep reinforcement learning into agent-based modeling can test regulatory mechanisms by exploring cell migration paths in a reverse engineering perspective. This model opens new doors to explore the large datasets generated by live imaging.<\/jats:p>\n <\/jats:sec>\n \n Availability and implementation<\/jats:title>\n Source code is available at https:\/\/github.com\/zwang84\/drl4cellmovement.<\/jats:p>\n <\/jats:sec>\n \n Supplementary information<\/jats:title>\n Supplementary data are available at Bioinformatics online.<\/jats:p>\n <\/jats:sec>","DOI":"10.1093\/bioinformatics\/bty323","type":"journal-article","created":{"date-parts":[[2018,4,24]],"date-time":"2018-04-24T19:15:06Z","timestamp":1524597306000},"page":"3169-3177","source":"Crossref","is-referenced-by-count":24,"title":["Deep reinforcement learning of cell movement in the early stage of C.elegans<\/i> embryogenesis"],"prefix":"10.1093","volume":"34","author":[{"given":"Zi","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6806-5108","authenticated-orcid":false,"given":"Dali","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN, USA"},{"name":"Environmental Science Division, Oak Ridge National Laboratory, Oak Ridge, TN, USA"}]},{"given":"Chengcheng","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8163-3920","authenticated-orcid":false,"given":"Yichi","family":"Xu","sequence":"additional","affiliation":[{"name":"Developmental Biology Program, Sloan-Kettering Institute, New York, NY, USA"}]},{"given":"Husheng","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville, TN, USA"}]},{"given":"Zhirong","family":"Bao","sequence":"additional","affiliation":[{"name":"Developmental Biology Program, Sloan-Kettering Institute, New York, NY, USA"}]}],"member":"286","published-online":{"date-parts":[[2018,4,25]]},"reference":[{"key":"2023012704181782000_bty323-B1","doi-asserted-by":"crossref","DOI":"10.1101\/pdb.prot065599","article-title":"Mounting Caenorhabditis elegans embryos for live imaging of embryogenesis","volume":"2011","author":"Bao","year":"2011","journal-title":"Cold Spring Harbor Protoc"},{"key":"2023012704181782000_bty323-B2","doi-asserted-by":"crossref","first-page":"2707","DOI":"10.1073\/pnas.0511111103","article-title":"Automated cell lineage tracing in Caenorhabditis elegans","volume":"103","author":"Bao","year":"2006","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023012704181782000_bty323-B3","doi-asserted-by":"crossref","first-page":"275.","DOI":"10.1186\/1471-2105-7-275","article-title":"Acetree: a tool for visual analysis of Caenorhabditis elegans embryogenesis","volume":"7","author":"Boyle","year":"2006","journal-title":"BMC Bioinformatics"},{"key":"2023012704181782000_bty323-B4","doi-asserted-by":"crossref","first-page":"156.","DOI":"10.1109\/TSMCC.2007.913919","article-title":"A comprehensive survey of multiagent reinforcement learning","volume":"38","author":"Busoniu","year":"2008","journal-title":"IEEE Trans. Syst. Man Cybern. C: Appl. Rev"},{"key":"2023012704181782000_bty323-B5","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.cell.2013.11.046","article-title":"De novo inference of systems-level mechanistic models of development from live-imaging-based phenotype analysis","volume":"156","author":"Du","year":"2014","journal-title":"Cell"},{"key":"2023012704181782000_bty323-B6","doi-asserted-by":"crossref","first-page":"592","DOI":"10.1016\/j.devcel.2015.07.014","article-title":"The regulatory landscape of lineage differentiation in a metazoan embryo","volume":"34","author":"Du","year":"2015","journal-title":"Dev. Cell"},{"key":"2023012704181782000_bty323-B7","first-page":"1","volume-title":"Stanford CS231n Course Report","author":"Egorov","year":"2016"},{"key":"2023012704181782000_bty323-B8","doi-asserted-by":"crossref","first-page":"4271","DOI":"10.1242\/dev.086256","article-title":"Quantitative semi-automated analysis of morphogenesis with single-cell resolution in complex embryos","volume":"139","author":"Giurumescu","year":"2012","journal-title":"Development"},{"key":"2023012704181782000_bty323-B9","doi-asserted-by":"crossref","first-page":"948","DOI":"10.1016\/j.cell.2013.05.008","article-title":"Forces in tissue morphogenesis and patterning","volume":"153","author":"Heisenberg","year":"2013","journal-title":"Cell"},{"key":"2023012704181782000_bty323-B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ydbio.2009.06.014","article-title":"Spatio-temporal reference model of Caenorhabditis elegans embryogenesis with cell contact maps","volume":"333","author":"Hench","year":"2009","journal-title":"Dev. Biol"},{"key":"2023012704181782000_bty323-B11","doi-asserted-by":"crossref","first-page":"2641","DOI":"10.1093\/bioinformatics\/btq437","article-title":"A cell-based simulation software for multi-cellular systems","volume":"26","author":"Hoehme","year":"2010","journal-title":"Bioinformatics"},{"key":"2023012704181782000_bty323-B12","first-page":"3675","volume-title":"Advances in Neural Information Processing Systems","author":"Kulkarni","year":"2016"},{"key":"2023012704181782000_bty323-B13","doi-asserted-by":"crossref","first-page":"D732","DOI":"10.1093\/nar\/gks1107","article-title":"Wddd: worm developmental dynamics database","volume":"41","author":"Kyoda","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2023012704181782000_bty323-B14","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1242\/dev.00211","article-title":"Mechanisms of cell positioning during C. elegans gastrulation","volume":"130","author":"Lee","year":"2003","journal-title":"Development"},{"year":"2015","author":"Lillicrap","key":"2023012704181782000_bty323-B15"},{"key":"2023012704181782000_bty323-B16","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/S0006-3495(00)76279-5","article-title":"Cell movement is guided by the rigidity of the substrate","volume":"79","author":"Lo","year":"2000","journal-title":"Biophys. J"},{"year":"2013","author":"Mnih","key":"2023012704181782000_bty323-B17"},{"key":"2023012704181782000_bty323-B18","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"2023012704181782000_bty323-B19","first-page":"1928","volume-title":"International Conference on Machine Learning","author":"Mnih","year":"2016"},{"key":"2023012704181782000_bty323-B20","doi-asserted-by":"crossref","first-page":"3266","DOI":"10.1242\/dev.096040","article-title":"Systematic quantification of developmental phenotypes at single-cell resolution during embryogenesis","volume":"140","author":"Moore","year":"2013","journal-title":"Development"},{"key":"2023012704181782000_bty323-B21","doi-asserted-by":"crossref","first-page":"1282","DOI":"10.1101\/gr.131920.111","article-title":"Multidimensional regulation of gene expression in the C. elegans embryo","volume":"22","author":"Murray","year":"2012","journal-title":"Genome Res"},{"key":"2023012704181782000_bty323-B22","doi-asserted-by":"crossref","first-page":"3798","DOI":"10.1093\/bioinformatics\/btw551","article-title":"Virtual exploration of early stage atherosclerosis","volume":"32","author":"Olivares","year":"2016","journal-title":"Bioinformatics"},{"key":"2023012704181782000_bty323-B23","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1016\/j.devcel.2010.08.014","article-title":"Chiral forces organize left-right patterning in C. elegans by uncoupling midline and anteroposterior axis","volume":"19","author":"Pohl","year":"2010","journal-title":"Dev. Cell"},{"key":"2023012704181782000_bty323-B24","doi-asserted-by":"crossref","first-page":"217.","DOI":"10.1186\/1471-2105-15-217","article-title":"A semi-local neighborhood-based framework for probabilistic cell lineage tracing","volume":"15","author":"Santella","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2023012704181782000_bty323-B25","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1006\/dbio.1997.8509","article-title":"Assessing normal embryogenesis in Caenorhabditis elegans using a 4d microscope: variability of development and regional specification","volume":"184","author":"Schnabel","year":"1997","journal-title":"Dev. Biol"},{"key":"2023012704181782000_bty323-B26","doi-asserted-by":"crossref","first-page":"2022","DOI":"10.1093\/bioinformatics\/bts307","article-title":"Multi-scale computational modeling of developmental biology","volume":"28","author":"Setty","year":"2012","journal-title":"Bioinformatics"},{"key":"2023012704181782000_bty323-B27","doi-asserted-by":"crossref","first-page":"20374","DOI":"10.1073\/pnas.0808725105","article-title":"Four-dimensional realistic modeling of pancreatic organogenesis","volume":"105","author":"Setty","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023012704181782000_bty323-B28","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1016\/j.mod.2003.06.005","article-title":"Mechanisms, mechanics and function of epithelial\u2013mesenchymal transitions in early development","volume":"120","author":"Shook","year":"2003","journal-title":"Mech. Dev"},{"key":"2023012704181782000_bty323-B29","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/0012-1606(83)90201-4","article-title":"The embryonic cell lineage of the nematode Caenorhabditis elegans","volume":"100","author":"Sulston","year":"1983","journal-title":"Dev. Biol"},{"volume-title":"Reinforcement Learning: An Introduction","year":"1998","author":"Sutton","key":"2023012704181782000_bty323-B30"},{"key":"2023012704181782000_bty323-B31","doi-asserted-by":"crossref","first-page":"e0172395.","DOI":"10.1371\/journal.pone.0172395","article-title":"Multiagent cooperation and competition with deep reinforcement learning","volume":"12","author":"Tampuu","year":"2017","journal-title":"PloS One"},{"key":"2023012704181782000_bty323-B32","doi-asserted-by":"crossref","first-page":"e0166551.","DOI":"10.1371\/journal.pone.0166551","article-title":"An observation-driven agent-based modeling and analysis framework for C. elegans embryogenesis","volume":"11","author":"Wang","year":"2016","journal-title":"PloS One"},{"key":"2023012704181782000_bty323-B33","doi-asserted-by":"crossref","first-page":"70","DOI":"10.4236\/jcc.2017.512008","article-title":"Visualization of 3-dimensional vectors in a dynamic embryonic system wormguides","volume":"5","author":"Wang","year":"2017","journal-title":"J. Comput. Commun"},{"year":"2017","author":"Wang","key":"2023012704181782000_bty323-B34"},{"year":"2012","author":"Zeiler","key":"2023012704181782000_bty323-B35"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/34\/18\/3169\/48919468\/bioinformatics_34_18_3169.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/34\/18\/3169\/48919468\/bioinformatics_34_18_3169.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T04:40:23Z","timestamp":1674794423000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/34\/18\/3169\/4986416"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2018,4,25]]},"references-count":35,"journal-issue":{"issue":"18","published-print":{"date-parts":[[2018,9,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/bty323","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"type":"print","value":"1367-4803"},{"type":"electronic","value":"1367-4811"}],"subject":[],"published-other":{"date-parts":[[2018,9,15]]},"published":{"date-parts":[[2018,4,25]]}}}