{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T13:10:29Z","timestamp":1740143429521,"version":"3.37.3"},"reference-count":26,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T00:00:00Z","timestamp":1585180800000},"content-version":"vor","delay-in-days":366,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000185","name":"DARPA","doi-asserted-by":"crossref","award":["HR0011-13-C-0052"],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Emerg. Technol. Comput. Syst."],"published-print":{"date-parts":[[2019,4,30]]},"abstract":"\n We provide here a novel method, called hypercolumn sparsification, to achieve high recognition performance for convolutional neural networks (CNNs) despite low-precision weights and activities during both training and test phases. This method is applicable to any CNN architecture that operates on signal patterns (e.g., audio, image, video) to extract information such as class membership. It operates on the stack of feature maps in each of the cascading feature matching and pooling layers through the processing hierarchy of the CNN by an explicit competitive process (\n k<\/jats:italic>\n -WTA, winner take all) that generates a sparse feature vector at each spatial location. This principle is inspired by local brain circuits, where neurons tuned to respond to different patterns in the incoming signals from an upstream region inhibit each other using interneurons, such that only the ones that are maximally activated survive the quenching threshold. We show this process of sparsification is critical for probabilistic learning of low-precision weights and bias terms, thereby making pattern recognition amenable for energy-efficient hardware implementations. Further, we show that hypercolumn sparsification could lead to more data-efficient learning as well as having an emergent property of significantly pruning down the number of connections in the network. A theoretical account and empirical analysis are provided to understand these effects better.\n <\/jats:p>","DOI":"10.1145\/3304104","type":"journal-article","created":{"date-parts":[[2019,3,28]],"date-time":"2019-03-28T12:23:24Z","timestamp":1553775804000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Hypercolumn Sparsification for Low-Power Convolutional Neural Networks"],"prefix":"10.1145","volume":"15","author":[{"given":"Praveen K.","family":"Pilly","sequence":"first","affiliation":[{"name":"Center for Autonomy Computing, Information and Systems Sciences Laboratory, HRL Laboratories, LLC, USA"}]},{"given":"Nigel D.","family":"Stepp","sequence":"additional","affiliation":[{"name":"Center for Autonomy Computing, Information and Systems Sciences Laboratory, HRL Laboratories, LLC, USA"}]},{"given":"Yannis","family":"Liapis","sequence":"additional","affiliation":[{"name":"Center for Autonomy Computing, Information and Systems Sciences Laboratory, HRL Laboratories, LLC, USA"}]},{"given":"David W.","family":"Payton","sequence":"additional","affiliation":[{"name":"Center for Autonomy Computing, Information and Systems Sciences Laboratory, HRL Laboratories, LLC, USA"}]},{"given":"Narayan","family":"Srinivasa","sequence":"additional","affiliation":[{"name":"Center for Autonomy Computing, Information and Systems Sciences Laboratory, HRL Laboratories, LLC, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,3,26]]},"reference":[{"unstructured":"Matthieu Courbariaux Yoshua Bengio and Jean-Pierre David. 2014. Training deep neural networks with low precision multiplications. (2014). arXiv:arXiv:1412.7024 Matthieu Courbariaux Yoshua Bengio and Jean-Pierre David. 2014. Training deep neural networks with low precision multiplications. (2014). arXiv:arXiv:1412.7024","key":"e_1_2_1_1_1"},{"unstructured":"Matthieu Courbariaux Yoshua Bengio and Jean-Pierre David. 2015. BinaryConnect: Training deep neural networks with binary weights during propagations. (2015). arXiv:arXiv:1511.00363 Matthieu Courbariaux Yoshua Bengio and Jean-Pierre David. 2015. BinaryConnect: Training deep neural networks with binary weights during propagations. (2015). arXiv:arXiv:1511.00363","key":"e_1_2_1_2_1"},{"key":"e_1_2_1_3_1","volume-title":"Thomas","author":"Cover Thomas M.","year":"2012","unstructured":"Thomas M. Cover and Joy A . Thomas . 2012 . Elements of Information Theory. John Wiley 8 Sons, New York, NY. Thomas M. Cover and Joy A. Thomas. 2012. Elements of Information Theory. John Wiley 8 Sons, New York, NY."},{"key":"e_1_2_1_4_1","volume-title":"Stork","author":"Duda Richard O.","year":"1973","unstructured":"Richard O. Duda , Peter E. Hart , and David G . Stork . 1973 . Pattern Classification, Vol . 2. Wiley Press , New York, NY. Richard O. Duda, Peter E. Hart, and David G. Stork. 1973. Pattern Classification, Vol. 2. Wiley Press, New York, NY."},{"doi-asserted-by":"publisher","key":"e_1_2_1_5_1","DOI":"10.1146\/annurev.neuro.23.1.441"},{"unstructured":"Benjamin Graham. 2013. Sparse arrays of signatures for online character recognition. (2013). arXiv:arXiv:1308.0371 Benjamin Graham. 2013. Sparse arrays of signatures for online character recognition. (2013). arXiv:arXiv:1308.0371","key":"e_1_2_1_6_1"},{"unstructured":"Benjamin Graham. 2014. Spatially-sparse convolutional neural networks. (2014). arXiv:arXiv:1409.6070 Benjamin Graham. 2014. Spatially-sparse convolutional neural networks. (2014). arXiv:arXiv:1409.6070","key":"e_1_2_1_7_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_8_1","DOI":"10.1109\/CVPR.2016.90"},{"doi-asserted-by":"publisher","key":"e_1_2_1_9_1","DOI":"10.1109\/72.143374"},{"unstructured":"Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv and Yoshua Bengio. 2016. Quantized neural networks: Training neural networks with low precision weights and activations. (2016). arXiv:arXiv:1609.07061 Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv and Yoshua Bengio. 2016. Quantized neural networks: Training neural networks with low precision weights and activations. (2016). arXiv:arXiv:1609.07061","key":"e_1_2_1_10_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_11_1","DOI":"10.1109\/ICPR.2014.416"},{"key":"e_1_2_1_12_1","volume-title":"Hinton","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E . Hinton . 2012 . ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems ( 2012). 1097--1105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (2012). 1097--1105."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Liu Baoyuan","year":"2015","unstructured":"Baoyuan Liu , Min Wang , Hassan Foroosh , Marshall Tappen , and Marianna Pensky . 2015 . Sparse convolutional neural networks . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015). 806--814. Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. 2015. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015). 806--814."},{"doi-asserted-by":"publisher","key":"e_1_2_1_14_1","DOI":"10.1145\/1553374.1553463"},{"unstructured":"Alireza Makhzani and Brendan Frey. 2014. k-sparse autoencoders. (2014). arXiv:arXiv:1312.5663 Alireza Makhzani and Brendan Frey. 2014. k -sparse autoencoders. (2014). arXiv:arXiv:1312.5663","key":"e_1_2_1_15_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_16_1","DOI":"10.5555\/2029556.2029563"},{"unstructured":"Daisuke Miyashita Edward H Lee and Boris Murmann. 2016. Convolutional neural networks using logarithmic data representation. (2016). arXiv:arXiv:1603.01025 Daisuke Miyashita Edward H Lee and Boris Murmann. 2016. Convolutional neural networks using logarithmic data representation. (2016). arXiv:arXiv:1603.01025","key":"e_1_2_1_17_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_18_1","DOI":"10.1109\/CVPR.2006.200"},{"doi-asserted-by":"publisher","key":"e_1_2_1_19_1","DOI":"10.1007\/s11263-007-0118-0"},{"doi-asserted-by":"crossref","unstructured":"Mohammad Rastegari Vicente Ordonez Joseph Redmon and Ali Farhadi. 2016. XNOR-Net: ImageNet classification using binary convolutional neural networks. (2016). arXiv:arXiv:1603.05279 Mohammad Rastegari Vicente Ordonez Joseph Redmon and Ali Farhadi. 2016. XNOR-Net: ImageNet classification using binary convolutional neural networks. (2016). arXiv:arXiv:1603.05279","key":"e_1_2_1_20_1","DOI":"10.1007\/978-3-319-46493-0_32"},{"unstructured":"Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla and Michael Bernstein. 2014. ImageNet large scale visual recognition challenge. (2014). arXiv:arXiv:1409.0575 Olga Russakovsky Jia Deng Hao Su Jonathan Krause Sanjeev Satheesh Sean Ma Zhiheng Huang Andrej Karpathy Aditya Khosla and Michael Bernstein. 2014. ImageNet large scale visual recognition challenge. (2014). arXiv:arXiv:1409.0575","key":"e_1_2_1_21_1"},{"key":"e_1_2_1_22_1","volume-title":"Overfeat: Integrated recognition, localization and detection using convolutional networks.","author":"Sermanet Pierre","year":"2013","unstructured":"Pierre Sermanet , David Eigen , Xiang Zhang , Micha\u00ebl Mathieu , Rob Fergus , and Yann LeCun . 2013 . Overfeat: Integrated recognition, localization and detection using convolutional networks. (2013). arXiv:arXiv:1312.6229 Pierre Sermanet, David Eigen, Xiang Zhang, Micha\u00ebl Mathieu, Rob Fergus, and Yann LeCun. 2013. Overfeat: Integrated recognition, localization and detection using convolutional networks. (2013). arXiv:arXiv:1312.6229"},{"doi-asserted-by":"publisher","key":"e_1_2_1_23_1","DOI":"10.1073\/pnas.0700622104"},{"key":"e_1_2_1_24_1","volume-title":"Advances in Neural Information Processing Systems (2016)","author":"Wen Wei","year":"2016","unstructured":"Wei Wen , Chunpeng Wu , Yandan Wang , Yiran Chen , and Hai Li . 2016 . Learning structured sparsity in deep neural networks . In Advances in Neural Information Processing Systems (2016) , 2074--2082. Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2016. Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (2016), 2074--2082."},{"key":"e_1_2_1_25_1","volume-title":"McDonnell","author":"Wong Sebastien C.","year":"2017","unstructured":"Sebastien C. Wong , Victor Stamatescu , Adam Gatt , David Kearney , Ivan Lee , and Mark D . McDonnell . 2017 . Track everything: Limiting prior knowledge in online multi-object recognition. (2017). arXiv:arXiv:1704.06415 Sebastien C. Wong, Victor Stamatescu, Adam Gatt, David Kearney, Ivan Lee, and Mark D. McDonnell. 2017. Track everything: Limiting prior knowledge in online multi-object recognition. (2017). arXiv:arXiv:1704.06415"},{"unstructured":"Shuchang Zhou Zekun Ni Xinyu Zhou He Wen Yuxin Wu and Yuheng Zou. 2016. Training low bitwidth convolutional neural networks with low bitwidth gradients. (2016). arXiv:arXiv:1606.06160 Shuchang Zhou Zekun Ni Xinyu Zhou He Wen Yuxin Wu and Yuheng Zou. 2016. Training low bitwidth convolutional neural networks with low bitwidth gradients. (2016). arXiv:arXiv:1606.06160","key":"e_1_2_1_26_1"}],"container-title":["ACM Journal on Emerging Technologies in Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3304104","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3304104","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T08:53:05Z","timestamp":1672563185000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3304104"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,26]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,4,30]]}},"alternative-id":["10.1145\/3304104"],"URL":"https:\/\/doi.org\/10.1145\/3304104","relation":{},"ISSN":["1550-4832","1550-4840"],"issn-type":[{"type":"print","value":"1550-4832"},{"type":"electronic","value":"1550-4840"}],"subject":[],"published":{"date-parts":[[2019,3,26]]},"assertion":[{"value":"2018-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}