{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T09:10:03Z","timestamp":1723021803065},"reference-count":36,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2019,7,25]],"date-time":"2019-07-25T00:00:00Z","timestamp":1564012800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2019,7,25]]},"abstract":"The rapid evolution of deep neural networks is demanding deep learning (DL) frameworks not only to satisfy the requirement of quickly executing large computations, but also to support straightforward programming models for quickly implementing and experimenting with complex network structures. However, existing frameworks fail to excel in both departments simultaneously, leading to diverged efforts for optimizing performance and improving usability.<\/jats:p>\n This paper presents JANUS, a system that combines the advantages from both sides by transparently converting an imperative DL program written in Python, a de-facto scripting language for DL, into an efficiently executable symbolic dataflow graph. JANUS can convert various dynamic features of Python, including dynamic control flow, dynamic types, and impure functions, into elements of a symbolic dataflow graph. Our experiments show that JANUS can achieve fast DL training by exploiting the techniques imposed by symbolic graph-based DL frameworks, while maintaining the simple and flexible programmability of imperative DL frameworks at the same time.<\/jats:p>","DOI":"10.1145\/3352020.3352025","type":"journal-article","created":{"date-parts":[[2019,7,26]],"date-time":"2019-07-26T13:17:18Z","timestamp":1564147038000},"page":"26-33","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Speculative Symbolic Graph Execution of Imperative Deep Learning Programs"],"prefix":"10.1145","volume":"53","author":[{"given":"Eunji","family":"Jeong","sequence":"first","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]},{"given":"Sungwoo","family":"Cho","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]},{"given":"Gyeong-In","family":"Yu","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]},{"given":"Joo Seong","family":"Jeong","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]},{"given":"Dong-Jin","family":"Shin","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]},{"given":"Taebum","family":"Kim","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]},{"given":"Byung-Gon","family":"Chun","sequence":"additional","affiliation":[{"name":"Seoul National University, Seoul, South Korea"}]}],"member":"320","published-online":{"date-parts":[[2019,7,25]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Mart\u00edn Abadi et al. 2016. TensorFlow: A System for Large-scale Machine Learning. In OSDI. Mart\u00edn Abadi et al. 2016. TensorFlow: A System for Large-scale Machine Learning. In OSDI."},{"key":"e_1_2_1_2_1","volume-title":"CoRR abs\/1606.01540","author":"Brockman Greg","year":"2016","unstructured":"Greg Brockman , Vicki Cheung , Ludwig Pettersson , Jonas Schneider , John Schulman , Jie Tang , and Wojciech Zaremba . 2016. Open AI Gym . CoRR abs\/1606.01540 ( 2016 ). arXiv:1606.01540 http:\/\/arxiv.org\/abs\/1606.01540 Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. CoRR abs\/1606.01540 (2016). arXiv:1606.01540 http:\/\/arxiv.org\/abs\/1606.01540"},{"key":"e_1_2_1_4_1","volume-title":"MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. In Workshop on Machine Learning Systems in NIPS.","author":"Chen Tianqi","year":"2015","unstructured":"Tianqi Chen , Mu Li , Yutian Li , Min Lin , Naiyan Wang , Minjie Wang , Tianjun Xiao , Bing Xu , Chiyuan Zhang , and Zheng Zhang . 2015 . MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. In Workshop on Machine Learning Systems in NIPS. Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. In Workshop on Machine Learning Systems in NIPS."},{"key":"e_1_2_1_5_1","volume-title":"TVM: An Automated End-to- End Optimizing Compiler for Deep Learning. In OSDI.","author":"Chen Tianqi","year":"2018","unstructured":"Tianqi Chen , Thierry Moreau , Ziheng Jiang , Lianmin Zheng , Eddie Yan , Haichen Shen , Meghan Cowan , LeyuanWang, Yuwei Hu , Luis Ceze , Carlos Guestrin , and Arvind Krishnamurthy . 2018 . TVM: An Automated End-to- End Optimizing Compiler for Deep Learning. In OSDI. Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, LeyuanWang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to- End Optimizing Compiler for Deep Learning. In OSDI."},{"key":"e_1_2_1_6_1","volume-title":"Sherjil Ozair, Aaron Courville, and Yoshua Bengio.","author":"Goodfellow Ian","year":"2014","unstructured":"Ian Goodfellow , Jean Pouget-Abadie , Mehdi Mirza , Bing Xu , David Warde- Farley , Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014 . Generative Adversarial Nets. In NIPS. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde- Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In NIPS."},{"key":"e_1_2_1_7_1","unstructured":"Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR."},{"key":"e_1_2_1_8_1","volume-title":"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML.","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy . 2015 . Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML. Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML."},{"key":"e_1_2_1_9_1","volume-title":"Efros","author":"Isola Phillip","year":"2017","unstructured":"Phillip Isola , Jun-Yan Zhu , Tinghui Zhou , and Alexei A . Efros . 2017 . Imageto- Image Translation with Conditional Adversarial Networks. In CVPR. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Imageto- Image Translation with Conditional Adversarial Networks. In CVPR."},{"key":"e_1_2_1_10_1","volume-title":"Dong-Jin Shin, and Byung-Gon Chun.","author":"Jeong Eunji","year":"2019","unstructured":"Eunji Jeong , Sungwoo Cho , Gyeong-In Yu , Joo Seong Jeong , Dong-Jin Shin, and Byung-Gon Chun. 2019 . JANUS : Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs. In NSDI. Eunji Jeong, Sungwoo Cho, Gyeong-In Yu, Joo Seong Jeong, Dong-Jin Shin, and Byung-Gon Chun. 2019. JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs. In NSDI."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190508.3190530"},{"key":"e_1_2_1_12_1","volume-title":"Exploring the Limits of Language Modeling. CoRR abs\/1602.02410","author":"J\u00f3zefowicz Rafal","year":"2016","unstructured":"Rafal J\u00f3zefowicz , Oriol Vinyals , Mike Schuster , Noam Shazeer , and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. CoRR abs\/1602.02410 ( 2016 ). arXiv:1602.02410 http:\/\/arxiv.org\/abs\/1602.02410 Rafal J\u00f3zefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. CoRR abs\/1602.02410 (2016). arXiv:1602.02410 http:\/\/arxiv.org\/abs\/1602.02410"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_1_14_1","unstructured":"Yann LeCun and Corinna Cortes. The MNIST Database of handwritten digits. http:\/\/yann.lecun.com\/exdb\/mnist\/. Yann LeCun and Corinna Cortes. The MNIST Database of handwritten digits. http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"e_1_2_1_15_1","unstructured":"Moshe Looks Marcello Herreshoff DeLesley Hutchins and Peter Norvig. 2017. Deep Learning with Dynamic Computation Graphs. In ICLR. Moshe Looks Marcello Herreshoff DeLesley Hutchins and Peter Norvig. 2017. Deep Learning with Dynamic Computation Graphs. In ICLR."},{"key":"e_1_2_1_16_1","volume-title":"Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.","author":"Mnih Volodymyr","year":"2016","unstructured":"Volodymyr Mnih , Adri\u00e0 Puigdom\u00e8nech Badia , Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016 . Asynchronous Methods for Deep Reinforcement Learning. In ICML. Volodymyr Mnih, Adri\u00e0 Puigdom\u00e8nech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In ICML."},{"key":"e_1_2_1_17_1","volume-title":"Wiltschko","author":"Moldovan Dan","year":"2019","unstructured":"Dan Moldovan , James M. Decker , Fei Wang , Andrew A. Johnson , Brian K. Lee , Zachary Nado , D. Sculley , Tiark Rompf , and Alexander B . Wiltschko . 2019 . AutoGraph: Imperative-style Coding with Graph-based Performance. In SysML. Dan Moldovan, James M. Decker, Fei Wang, Andrew A. Johnson, Brian K. Lee, Zachary Nado, D. Sculley, Tiark Rompf, and Alexander B. Wiltschko. 2019. AutoGraph: Imperative-style Coding with Graph-based Performance. In SysML."},{"key":"e_1_2_1_18_1","unstructured":"MXNet Developers. Gluon. http:\/\/gluon.mxnet.io\/. MXNet Developers. Gluon. http:\/\/gluon.mxnet.io\/."},{"key":"e_1_2_1_19_1","volume-title":"Pydron: Semi-Automatic Parallelization for Multi-Core and the Cloud. In OSDI.","author":"M\u00fcller Stefan C.","year":"2014","unstructured":"Stefan C. M\u00fcller , Gustavo Alonso , and Adam Amara Andr\u00e9 Csillaghy . 2014 . Pydron: Semi-Automatic Parallelization for Multi-Core and the Cloud. In OSDI. Stefan C. M\u00fcller, Gustavo Alonso, and Adam Amara Andr\u00e9 Csillaghy. 2014. Pydron: Semi-Automatic Parallelization for Multi-Core and the Cloud. In OSDI."},{"key":"e_1_2_1_20_1","unstructured":"Graham Neubig et al. 2017. DyNet: The Dynamic Neural Network Toolkit. CoRR abs\/1701.03980 (2017). Graham Neubig et al. 2017. DyNet: The Dynamic Neural Network Toolkit. CoRR abs\/1701.03980 (2017)."},{"key":"e_1_2_1_21_1","volume-title":"Automatic Differentiation in PyTorch. In Autodiff Workshop in NIPS.","author":"Paszke Adam","year":"2017","unstructured":"Adam Paszke , Sam Gross , Soumith Chintala , Gregory Chanan , Edward Yang , Zachary DeVito , Zeming Lin , Alban Desmaison , Luca Antiga , and Adam Lerer . 2017 . Automatic Differentiation in PyTorch. In Autodiff Workshop in NIPS. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic Differentiation in PyTorch. In Autodiff Workshop in NIPS."},{"key":"e_1_2_1_22_1","unstructured":"Python Software Foundation. Python programming language. https: \/\/www.python.org\/. Python Software Foundation. Python programming language. https: \/\/www.python.org\/."},{"key":"e_1_2_1_23_1","unstructured":"Pytorch Developers. PyTorch JIT. https:\/\/github.com\/pytorch\/pytorch\/ tree\/master\/torch\/csrc\/jit. Pytorch Developers. PyTorch JIT. https:\/\/github.com\/pytorch\/pytorch\/ tree\/master\/torch\/csrc\/jit."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_2_1_25_1","volume-title":"Proximal Policy Optimization Algorithms. CoRR abs\/1707.06347","author":"Schulman John","year":"2017","unstructured":"John Schulman , Filip Wolski , Prafulla Dhariwal , Alec Radford , and Oleg Klimov . 2017. Proximal Policy Optimization Algorithms. CoRR abs\/1707.06347 ( 2017 ). John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. CoRR abs\/1707.06347 (2017)."},{"key":"e_1_2_1_26_1","volume-title":"Andrew Y. Ng, and Christopher D. Manning.","author":"Socher Richard","year":"2011","unstructured":"Richard Socher , Cliff Chiung-Yu Lin , Andrew Y. Ng, and Christopher D. Manning. 2011 . Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML. Richard Socher, Cliff Chiung-Yu Lin, Andrew Y. Ng, and Christopher D. Manning. 2011. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In ICML."},{"key":"e_1_2_1_27_1","unstructured":"Richard Socher Alex Perelygin Jean Wu Jason Chuang Christopher D Manning Andrew Ng and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank. In EMNLP. Richard Socher Alex Perelygin Jean Wu Jason Chuang Christopher D Manning Andrew Ng and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank. In EMNLP."},{"key":"e_1_2_1_28_1","unstructured":"Swift Project Authors. Swift for TensorFlow. https:\/\/github.com\/tensorflow\/ swift. Swift Project Authors. Swift for TensorFlow. https:\/\/github.com\/tensorflow\/ swift."},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jon Shlens and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR. Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jon Shlens and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In CVPR.","DOI":"10.1109\/CVPR.2016.308"},{"key":"e_1_2_1_30_1","unstructured":"Kai Sheng Tai Richard Socher and Christopher D Manning. 2015. Improved Semantic Representations from Tree-structured Long Short-term Memory Networks. In ACL. Kai Sheng Tai Richard Socher and Christopher D Manning. 2015. Improved Semantic Representations from Tree-structured Long Short-term Memory Networks. In ACL."},{"key":"e_1_2_1_31_1","unstructured":"TensorFlow Developers. Eager Execution. https:\/\/www.tensorflow.org\/ programmers_guide\/eager. TensorFlow Developers. Eager Execution. https:\/\/www.tensorflow.org\/ programmers_guide\/eager."},{"key":"e_1_2_1_32_1","unstructured":"TensorFlow Developers. tf.contrib.eager.defun. https:\/\/www.tensorflow. org\/versions\/r1.8\/api_docs\/python\/tf\/contrib\/eager\/defun. TensorFlow Developers. tf.contrib.eager.defun. https:\/\/www.tensorflow. org\/versions\/r1.8\/api_docs\/python\/tf\/contrib\/eager\/defun."},{"key":"e_1_2_1_33_1","unstructured":"TensorFlow Developers. XLA Overview. https:\/\/www.tensorflow.org\/ performance\/xla\/. TensorFlow Developers. XLA Overview. https:\/\/www.tensorflow.org\/ performance\/xla\/."},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Radim Tylecek and Radim \u00e1ra. 2013. Spatial Pattern Templates for Recognition of Objects with Regular Structure. In GCPR. Radim Tylecek and Radim \u00e1ra. 2013. Spatial Pattern Templates for Recognition of Objects with Regular Structure. In GCPR.","DOI":"10.1007\/978-3-642-40602-7_39"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2011.37"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190508.3190551"},{"key":"e_1_2_1_37_1","volume-title":"Recurrent Neural Network Regularization. CoRR abs\/1409.2329","author":"Zaremba Wojciech","year":"2014","unstructured":"Wojciech Zaremba , Ilya Sutskever , and Oriol Vinyals . 2014. Recurrent Neural Network Regularization. CoRR abs\/1409.2329 ( 2014 ). Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent Neural Network Regularization. CoRR abs\/1409.2329 (2014)."}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3352020.3352025","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T06:36:24Z","timestamp":1672554984000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3352020.3352025"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,25]]},"references-count":36,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,7,25]]}},"alternative-id":["10.1145\/3352020.3352025"],"URL":"https:\/\/doi.org\/10.1145\/3352020.3352025","relation":{},"ISSN":["0163-5980"],"issn-type":[{"value":"0163-5980","type":"print"}],"subject":[],"published":{"date-parts":[[2019,7,25]]},"assertion":[{"value":"2019-07-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}