Abstract
In this paper, we analyze a deep neural network model from the viewpoint of singularities. First, we show that there exist a large number of critical points introduced by a hierarchical structure in the deep neural network as straight lines. Next, we derive sufficient conditions for the deep neural network having no critical points introduced by a hierarchical structure.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)
Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)
Fukumizu, K., Amari, S.: Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Netw. 13(3), 317–327 (2000)
Montúfar, G., Pascanu, R., Cho, K., Bengio, Y.: On the number of linear regions of deep neural networks. In: Advances in Neural Information Processing Systems 27 (NIPS 2014), pp. 2924–2932 (2014)
Szymanski, L., McCane, B.: Deep networks are effective encoders of periodicity. IEEE Trans. Neural Netw. Learn. Syst. 25(10), 1816–1827 (2014)
Nitta, T.: Resolution of singularities introduced by hierarchical structure in deep neural networks. IEEE Trans. Neural Netw. Learn. Syst. (accepted). doi:10.1109/TNNLS.2016.2580741
Acknowledgments
The author would like to give special thanks to the anonymous reviewers for valuable comments. This work was supported by the Japan Society for the Promotion of Science through the Grants-in-Aid for Scientific Research (KAKENHI) under Grant JP16K00347.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Nitta, T. (2016). On the Singularity in Deep Neural Networks. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds) Neural Information Processing. ICONIP 2016. Lecture Notes in Computer Science(), vol 9950. Springer, Cham. https://doi.org/10.1007/978-3-319-46681-1_47
Download citation
DOI: https://doi.org/10.1007/978-3-319-46681-1_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-46680-4
Online ISBN: 978-3-319-46681-1
eBook Packages: Computer ScienceComputer Science (R0)