Abstract
Artificial neural networks (ANNs) have achieved significant success in tackling classical and modern machine learning problems. As learning problems grow in scale and complexity, and expand into multi-disciplinary territory, a more modular approach for scaling ANNs will be needed. Modular neural networks (MNNs) are neural networks that embody the concepts and principles of modularity. MNNs adopt a large number of different techniques for achieving modularization. Previous surveys of modularization techniques are relatively scarce in their systematic analysis of MNNs, focusing mostly on empirical comparisons and lacking an extensive taxonomical framework. In this review, we aim to establish a solid taxonomy that captures the essential properties and relationships of the different variants of MNNs. Based on an investigation of the different levels at which modularization techniques act, we attempt to provide a universal and systematic framework for theorists studying MNNs, also trying along the way to emphasise the strengths and weaknesses of different modularization approaches in order to highlight good practices for neural network practitioners.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Achard S, Bullmore E (2007) Efficiency and cost of economical brain functional networks. PLoS Comput Biol 3(2):0174–0183. https://doi.org/10.1371/journal.pcbi.0030017
Aguirre C, Huerta R, Corbacho F, Pascual P (2002) Analysis of biologically inspired small-world networks. In: International conference on artificial neural networks. Springer, pp 27–32
Allen F, Almasi G, Andreoni W, Beece D, Berne BJ, Bright A, Brunheroto J, Cascaval C, Castanos J, Coteus P, Crumley P, Curioni A, Denneau M, Donath W, Eleftheriou M, Flitch B, Fleischer B, Georgiou CJ, Germain R, Giampapa M, Gresh D, Gupta M, Haring R, Ho H, Hochschild P, Hummel S, Jonas T, Lieber D, Martyna G, Maturu K, Moreira J, Newns D, Newton M, Philhower R, Picunko T, Pitera J, Pitman M, Rand R, Royyuru A, Salapura V, Sanomiya A, Shah R, Sham Y, Singh S, Snir M, Suits F, Swetz R, Swope WC, Vishnumurthy N, Ward TJC, Warren H, Zhou R (2001) Blue Gene: a vision for protein science using a petaflop supercomputer. IBM Syst J 40(2):310–327. https://doi.org/10.1147/sj.402.0310
Almasri MN, Kaluarachchi JJ (2005) Modular neural networks to predict the nitrate distribution in ground water using the on-ground nitrogen loading and recharge data. Environ Model Softw 20(7):851–871
Aminian M, Aminian F (2007) A modular fault-diagnostic system for analog electronic circuits using neural networks with wavelet transform as a preprocessor. IEEE Trans Instrum Meas 56(5):1546–1554
Anand R, Mehrotra K, Mohan C, Ranka S (1995) Efficient classification for multiclass problems using modular neural networks. IEEE Trans Neural Netw 6(1):117–124. https://doi.org/10.1109/72.363444
Anderson A, Shaffer K, Yankov A, Corley CD, Hodas NO (2016) Beyond fine tuning: a modular approach to learning on small data. arXiv:1611.01714v1
Andreas J, Rohrbach M, Darrell T, Klein D (2016a) Neural module networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 39–48
Andreas J, Rohrbach M, Darrell T, Klein D (2016b) Learning to compose neural networks for question answering. arXiv:1601.01705
Angelucci a, Clascá F, Bricolo E, Cramer KS, Sur M (1997) Experimentally induced retinal projections to the ferret auditory thalamus: development of clustered eye-specific patterns in a novel target. J Neurosci Off J Soc Neurosci 17(6):2040–2055
Auda G, Kamel M (1998) Modular neural network classifiers: a comparative study. J Intell Robot Syst 21:117–129. https://doi.org/10.1023/A:1007925203918
Auda G, Kamel M (1999) Modular neural networks: a survey. Int J Neural Syst 9(2):129–51
Azam F (2000) Biologically inspired modular neural networks. https://vtechworks.lib.vt.edu/handle/10919/27998. Accessed 23 Dec 2018
Ba J, Caruana R (2014) Do deep nets really need to be deep? In: Advances in neural information processing systems. pp 2654–2662
Babaei S, Geranmayeh A, Seyyedsalehi SA (2010) Protein secondary structure prediction using modular reciprocal bidirectional recurrent neural networks. Comput Methods Programs Biomed 100(3):237–247. https://doi.org/10.1016/j.cmpb.2010.04.005
Battaglia PW, Hamrick JB, Bapst V, Sanchez-Gonzalez A, Zambaldi V, Malinowski M, Tacchetti A, Raposo D, Santoro A, Faulkner R, Gulcehre C, Song F, Ballard A, Gilmer J, Dahl G, Vaswani A, Allen K, Nash C, Langston V, Dyer C, Heess N, Wierstra D, Kohli P, Botvinick M, Vinyals O, Li Y, Pascanu R (2018) Relational inductive biases, deep learning, and graph networks. arXiv:1806.01261
Bender G, Kindermans PJ, Zoph B, Vasudevan V, Le Q (2018) Understanding and simplifying one-shot architecture search. http://proceedings.mlr.press/v80/bender18a. Accessed 5 Dec 2018
Bengio Y, Louradour J, Collobert R, Weston J (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning—ICML ’09. ACM Press, New York, New York, USA, pp 1–8. https://doi.org/10.1145/1553374.1553380, http://portal.acm.org/citation.cfm?doid=1553374.1553380
Bengio S, Vinyals O, Jaitly N, Shazeer N (2015) Scheduled sampling for sequence prediction with recurrent neural networks. http://papers.nips.cc/paper/5956-scheduled-sampling-for-sequence-prediction-with-recurrent-neural-networks. Accessed 12 Mar 2018
Bhende C, Mishra S, Panigrahi B (2008) Detection and classification of power quality disturbances using S-transform and modular neural network. Electr Power Syst Res 78(1):122–128. https://doi.org/10.1016/j.epsr.2006.12.011
Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05424
Bohland JW, Minai AA (2001) Efficient associative memory using small-world architecture. Neurocomputing 38:489–496. https://doi.org/10.1016/S0925-2312(01)00378-2
Brandes U, Delling D, Gaertler M, Gorke R, Hoefer M, Nikoloski Z, Wagner D (2008) On modularity clustering. IEEE Trans Knowl Data Eng 20(2):172–188. https://doi.org/10.1109/TKDE.2007.190689
Braylan A, Hollenbeck M, Meyerson E, Miikkulainen R (2015) Reuse of neural modules for general video game playing. arXiv:1512.01537
Bullmore ET, Bassett DS (2011) Brain graphs: graphical models of the human brain connectome. Annu Rev Clin Psychol 7(1):113–140. https://doi.org/10.1146/annurev-clinpsy-040510-143934
Bullmore E, Sporns O (2009) Complex brain networks: graph theoretical analysis of structural and functional systems. Nat Rev Neurosci 10(3):186–198. https://doi.org/10.1038/nrn2575
Buxhoeveden DP (2002) The minicolumn hypothesis in neuroscience. Brain 125(5):935–951. https://doi.org/10.1093/brain/awf110
Caelli T, Guan L, Wen W (1999) Modularity in neural computing. Proc IEEE 87(9):1497–1518. https://doi.org/10.1109/5.784227
Calabretta R, Nolfi S, Parisi D, Wagner GP (2000) Duplication of modules facilitates the evolution of functional specialization. Artif Life 6(1):69–84
Chen ZJ, He Y, Rosa-Neto P, Germann J, Evans AC (2008) Revealing modular architecture of human brain structural networks by using cortical thickness from MRI. Cereb Cortex 18(10):2374–2381. https://doi.org/10.1093/cercor/bhn003
Chiang CC, Fu HC (1994) A divide-and-conquer methodology for modular supervised neural network design. In: Neural networks, 1994. IEEE world congress on computational intelligence, 1994 IEEE international conference on. IEEE, vol 1, pp 119–124
Chihaoui M, Elkefi A, Bellil W, Ben Amar C (2016) A survey of 2D face recognition techniques. Computers 5(4):21. https://doi.org/10.3390/computers5040021
Chollet F (2016) Xception: deep learning with depthwise separable convolutions. arXiv:1610.02357
Chris Tseng H, Almogahed B (2009) Modular neural networks with applications to pattern profiling problems. Neurocomputing 72(10–12):2093–2100. https://doi.org/10.1016/J.NEUCOM.2008.10.020
Ciregan D, Meier U, Schmidhuber J (2012) Multi-column deep neural networks for image classification. In: Computer vision and pattern recognition (CVPR), 2012 IEEE conference on. IEEE, pp 3642–3649
Clune J, Mouret JB, Lipson H (2013) The evolutionary origins of modularity. Proc Biol Sci R Soc 280(1755):20122863. https://doi.org/10.1098/rspb.2012.2863. arXiv:1207.2743v1
de Nardi R, Togelius J, Holland O, Lucas S (2006) Evolution of neural networks for helicopter control: Why modularity matters. In: 2006 IEEE international conference on evolutionary computation. IEEE, pp 1799–1806. https://doi.org/10.1109/CEC.2006.1688525
Di Ferdinando A, Calabretta R, Parisi D (2001) Evolving modular architectures for neural networks. Proc Sixth Neural Comput Psychol Workshop Evol Learn Dev 12(5):253–262
Douglas RJ, Martin KAC (2007) Recurrent neuronal circuits in the neocortex. Curr Biol CB 17(13):R496–500. https://doi.org/10.1016/j.cub.2007.04.024
Eppel S (2017) Hierarchical semantic segmentation using modular convolutional neural networks. arXiv:1710.05126v1
Eyben F, Weninger F, Squartini S, Schuller B (2013) Real-life voice activity detection with LSTM recurrent neural networks and an application to Hollywood movies. In: ICASSP, IEEE international conference on acoustics, speech and signal processing—proceedings, pp 483–487. https://doi.org/10.1109/ICASSP.2013.6637694
Fernando C, Banarse D, Blundell C, Zwols Y, Ha D, Rusu AA, Pritzel A, Wierstra D (2017) PathNet: evolution channels gradient descent in super neural networks. arXiv:1701.08734
Ferreira MD, Corrêa DC, Nonato LG, de Mello RF (2018) Designing architectures of convolutional neural networks to solve practical problems. Expert Syst Appl 94:205–217. https://doi.org/10.1016/J.ESWA.2017.10.052
Franco L, Cannas SA (2001) Generalization properties of modular networks: implementing the parity function. IEEE Trans Neural Netw 12(6):1306–1313. https://doi.org/10.1109/72.963767
Freddolino PL, Liu F, Gruebele M, Schulten K (2008) Ten-microsecond molecular dynamics simulation of a fast-folding WW domain. Biophys J 94(10):L75–L77. https://doi.org/10.1529/biophysj.108.131565
Fritsch J (1996) Modular neural networks for speech recognition (No. CMU-CS-96-203). Carnegie-Mellon Univ Pittsburgh PA Dept of Computer Science
Fu HC, Lee YP, Chiang CC, Pao HT (2001) Divide-and-conquer learning and modular perceptron networks. IEEE Trans Neural Netw 12(2):250–263. https://doi.org/10.1109/72.914522
Fukushima K, Miyake S, Ito T (1983) Neocognitron: a neural network model for a mechanism of visual pattern recognition. IEEE Trans Syst Man Cybern SMC–13(5):826–834. https://doi.org/10.1109/TSMC.1983.6313076
Garcia-Pedrajas N, Hervas-Martinez C, Munoz-Perez J (2003) COVNET: a cooperative coevolutionary model for evolving artificial neural networks. IEEE Trans Neural Netw 14(3):575–596. https://doi.org/10.1109/TNN.2003.810618
Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE (2017) Neural message passing for quantum chemistry. arXiv:1704.01212
Gollisch T, Meister M (2010) Eye smarter than scientists believed: neural computations in circuits of the retina. https://doi.org/10.1016/j.neuron.2009.12.009
Goltsev A, Gritsenko V (2015) Modular neural networks with radial neural columnar architecture. Biol Inspir Cognit Archit 13:63–74. https://doi.org/10.1016/J.BICA.2015.06.001
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. http://papers.nips.cc/paper/5423-generative-adversarial-nets. Accessed 23 Dec 2018
Gradojevic N, Gençay R, Kukolj D (2009) Option pricing with modular neural networks. IEEE Trans Neural Netw Publ IEEE Neural Netw Council 20(4):626–637. https://doi.org/10.1109/TNN.2008.2011130
Guan SU, Li S (2002) Parallel growing and training of neural networks using output parallelism. IEEE Trans Neural Netw 13(3):542–550
Happel BLM, Murre JMJ (1994) Design and evolution of modular neural network architectures. Neural Netw 7(6–7):985–1004. https://doi.org/10.1016/S0893-6080(05)80155-8
Haykin S (1994) Neural networks: a comprehensive foundation. Prentice Hall PTR, Upper Saddle River
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 770–778
Hidalgo D, Castillo O, Melin P (2009) Type-1 and type-2 fuzzy inference systems as integration methods in modular neural networks for multimodal biometry and its optimization with genetic algorithms. Inf Sci 179(13):2123–2145
Hochreiter S, Urgen Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hu R, Rohrbach M, Andreas J, Darrell T, Saenko K (2016) Modeling relationships in referential expressions with compositional modular networks. arXiv:1611.09978
Huang G-B (2003) Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Trans Neural Netw 14(2):274–281. https://doi.org/10.1109/TNN.2003.809401
Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. In: European conference on computer vision. Springer, pp 646–661
Huizinga J, Mouret JB, Clune J (2014) Evolving neural networks that are both modular and regular: HyperNeat plus the connection cost technique. Gecco, pp 697–704, https://doi.org/10.1145/2576768.2598232
Hüsken M, Igel C, Toussaint M (2002) Task-dependent evolution of modularity in neural networks. Connect Sci 14(3):219–229
Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE (1991) Adaptive mixtures of local experts. Neural Comput 3(1):79–87. https://doi.org/10.1162/neco.1991.3.1.79
Wei Jiang, Kong Seong G (2007) Block-based neural networks for personalized ECG signal classification. IEEE Trans Neural Netw 18(6):1750–1761. https://doi.org/10.1109/TNN.2007.900239
Kacprzyk J, Pedrycz W (2015) Springer handbook of computational intelligence. Springer, Berlin
Kaiser M, Hilgetag CC (2010) Optimal hierarchical modular topologies for producing limited sustained activation of neural networks. Front Neuroinform 4:8
Karami M, Safabakhsh R, Rahmati M (2013) Modular cellular neural network structure for wave-computing-based image processing. ETRI J 35(2):207–217. https://doi.org/10.4218/etrij.13.0112.0107
Kashtan N, Alon U (2005) Spontaneous evolution of modularity and network motifs. Proc Natl Acad Sci USA 102(39):13773–8. https://doi.org/10.1073/pnas.0503610102
Kastellakis G, Cai DJ, Mednick SC, Silva AJ, Poirazi P (2015) Synaptic clustering within dendrites: an emerging theory of memory formation. https://doi.org/10.1016/j.pneurobio.2014.12.002
Kim T, Cha M, Kim H, Lee JK, Kim J (2017) Learning to discover cross-domain relations with generative adversarial networks. arXiv:1703.05192
Larsson G, Maire M, Shakhnarovich G (2016) FractalNet: ultra-deep neural networks without residuals. arXiv:1605.07648
Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436
Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint p 10, https://doi.org/10.1109/ASRU.2015.7404828, arXiv:1312.4400
Liu H, Simonyan K, Vinyals O, Fernando C, Kavukcuoglu K (2017) Hierarchical representations for efficient architecture search. arXiv:1711.00436
Lodato S, Arlotta P (2015) Generating neuronal diversity in the mammalian cerebral cortex. Annu Rev Cell Dev Biol 31(1):699–720. https://doi.org/10.1146/annurev-cellbio-100814-125353
López-Muñoz F, Boya J, Alamo C (2006) Neuron theory, the cornerstone of neuroscience, on the centenary of the Nobel Prize award to Santiago Ramón y Cajal. Brain Res Bull 70(4–6):391–405. https://doi.org/10.1016/j.brainresbull.2006.07.010
Melin P, Mancilla A, Lopez M, Mendoza O (2007) A hybrid modular neural network architecture with fuzzy sugeno integration for time series forecasting. Appl Soft Comput 7(4):1217–1226
Melin P, Mendoza O, Castillo O (2011) Face recognition with an improved interval type-2 fuzzy logic sugeno integral and modular neural networks. IEEE Trans Syst Man Cybern Part A Syst Hum 41(5):1001–1012
Mendoza O, Melin P, Licea G (2009a) A hybrid approach for image recognition combining type-2 fuzzy logic, modular neural networks and the Sugeno integral. Inf Sci 179(13):2078–2101. https://doi.org/10.1016/j.ins.2008.11.018
Mendoza O, Melín P, Castillo O (2009b) Interval type-2 fuzzy logic and modular neural networks for face recognition applications. Appl Soft Comput 9(4):1377–1387. https://doi.org/10.1016/j.asoc.2009.06.007
Meunier D, Lambiotte R, Bullmore ET (2010) Modular and hierarchically modular organization of brain networks. Front Neurosci. https://doi.org/10.3389/fnins.2010.00200
Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan A, Duffy N, Hodjat B (2017) Evolving deep neural networks. arXiv:1703.00548
Montufar GF, Pascanu R, Cho K, Bengio Y (2014) On the number of linear regions of deep neural networks. http://papers.nips.cc/paper/5422-on-the-number-of-linear-regions-of-deep-neural-networks. Accessed 24 Dec 2018
Moon S-W, Kong S-G (2001) Block-based neural networks. IEEE Trans Neural Netw 12(2):307–317. https://doi.org/10.1109/72.914525
Mountcastle VB (1997) The columnar organization of the neocortex. Brain J Neurol. https://doi.org/10.1093/brain/120.4.701
Mouret JB, Doncieux S (2009) Evolving modular neural-networks through exaptation. In: 2009 IEEE congress on evolutionary computation, CEC 2009. pp 1570–1577. https://doi.org/10.1109/CEC.2009.4983129
Mouret JB, Doncieux S (2008) MENNAG: a modular, regular and hierarchical encoding for neural-networks based on attribute grammars. Evolut Intell 1(3):187–207. https://doi.org/10.1007/s12065-008-0015-7
Newman MEJ (2004) Detecting community structure in networks. Eur Phys J B 38:321–330. https://doi.org/10.1140/epjb/e2004-00124-y
Newman MEJ (2006) Modularity and community structure in networks. Proc Natl Acad Sci USA 103(23):8577–82. https://doi.org/10.1073/pnas.0601602103
Newman MEJ (2016) Community detection in networks: modularity optimization and maximum likelihood are equivalent. 1:1–8. https://doi.org/10.1103/PhysRevE.94.052315, arXiv:1606.02319
Oh IS, Suen CY (2002) A class-modular feedforward neural network for handwriting recognition. Pattern Recognit 35(1):229–244. https://doi.org/10.1016/S0031-3203(00)00181-3
Ortín S, Gutiérrez J, Pesquera L, Vasquez H (2005) Nonlinear dynamics extraction for time-delay systems using modular neural networks synchronization and prediction. Physica A Stat Mech Appl 351(1):133–141. https://doi.org/10.1016/j.physa.2004.12.015
Ou G, Murphey YL (2007) Multi-class pattern classification using neural networks. Pattern Recognit 40(1):4–18. https://doi.org/10.1016/j.patcog.2006.04.041
Pan P, Xu Z, Yang Y, Wu F, Zhuang Y (2016) Hierarchical recurrent neural encoder for video representation with application to captioning. In: The IEEE conference on computer vision and pattern recognition (CVPR)
Phan KT, Maul TH, Tuong TV (2015) A parallel circuit approach for improving the speed and generalization properties of neural networks. In: 2015 11th international conference on natural computation (ICNC). IEEE, pp 1–7. https://doi.org/10.1109/ICNC.2015.7377956
Phan KT, Maul TH, Vu TT, Lai WK (2016) Improving neural network generalization by combining parallel circuits with dropout. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), vol 9949 LNCS, pp 572–580. https://doi.org/10.1007/978-3-319-46675-0_63, arXiv:1612.04970
Phan KT, Maul TH, Vu TT, Lai WK (2017) Dropcircuit: a modular regularizer for parallel circuit networks. Neural Process Lett 47:1–18
Phaye SSR, Sikka A, Dhall A, Bathula D (2018) Dense and diverse capsule networks: making the capsules learn better. arXiv:1805.04001
Radicchi F, Castellano C, Cecconi F, Loreto V, Parisi D (2004) Defining and identifying communities in networks. Proc Natl Acad Sci USA 101(9):2658–2663
Reisinger J, Stanley KO, Miikkulainen R (2004) Evolving reusable neural modules. In: Genetic and evolutionary computation conference. Springer, pp 69–81
Ronco E, Gawthrop P (1995) Modular neural networks: a state of the art. Rapport Technique CSC-95026, center of system and control, University of Glasgow. http://www.mech.gla.ac.uk/control/report.html
Ronen M, Shabtai Y, Guterman H (2002) Hybrid model building methodology using unsupervised fuzzy clustering and supervised neural networks. Biotechnol Bioeng 77(4):420–429
Rudasi L, Zahorian S (1991) Text-independent talker identification with neural networks. In: [Proceedings] ICASSP 91: 1991 international conference on acoustics, speech, and signal processing. IEEE, vol 1, pp 389–392. https://doi.org/10.1109/ICASSP.1991.150358
Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. http://papers.nips.cc/paper/6975-dynamic-routing-between-capsules. Accessed 28 Feb 2018
San PP, Ling SH, Nguyen HT (2011) Block based neural network for hypoglycemia detection. In: 2011 annual international conference of the IEEE engineering in medicine and biology society. IEEE, pp 5666–5669. https://doi.org/10.1109/IEMBS.2011.6091371
Santoro A, Raposo D, Barrett DGT, Malinowski M, Pascanu R, Battaglia P, Lillicrap T (2017) A simple neural network module for relational reasoning. arXiv:1706.01427
Schwarz AJ, Gozzi A, Bifone A (2008) Community structure and modularity in networks of correlated brain activity. Magn Reson Imaging 26(7):914–920. https://doi.org/10.1016/j.mri.2008.01.048
Serban IV, Sordoni A, Bengio Y, Courville A, Pineau J (2016) Building end-to-end dialogue systems using generative hierarchical neural network models. AAAI p 8. https://doi.org/10.1017/CBO9781107415324.004, arXiv:1507.04808
Sharkey AJC (1996) On combining artificial neural nets. Connect Sci 8(3–4):299–313. https://doi.org/10.1080/095400996116785
Shetty R, Laaksonen J (2015) Video captioning with recurrent networks based on frame- and video-level features and visual content classification. arXiv:1512.02949
Singh S, Hoiem D, Forsyth D (2016) Swapout: Learning an ensemble of deep architectures. In: Advances in neural information processing systems. pp 28–36
Song L, Zhang Y, Wang Z, Gildea D (2018) A graph-to-sequence model for AMR-to-text generation. arXiv:1805.02473
Soutner D, Müller L (2013) Application of lstm neural networks in language modelling. In: International conference on text, speech and dialogue. Springer, pp 105–112
Sporns O (2011) The human connectome: a complex network. Ann N Y Acad Sci. https://doi.org/10.1111/j.1749-6632.2010.05888.x
Sporns O, Zwi JD (2004) The small world of the cerebral cortex. Neuroinformatics 2(2):145–162. https://doi.org/10.1385/NI:2:2:145
Srivastava RK, Masci J, Kazerounian S, Gomez F, Schmidhuber J (2013) Compete to compute. Nips pp 2310–2318
Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958. https://doi.org/10.1214/12-AOS1000. arXiv:1102.4807
Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:1505.00387 [cs]
Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evolut Comput 10(2):99–127. https://doi.org/10.1162/106365602320169811
Stanley KO, D’Ambrosio DB, Gauci J (2009) A hypercube-based encoding for evolving large-scale neural networks. Artif Life 15(2):185–212. https://doi.org/10.1162/artl.2009.15.2.15202
Stollenga MF, Byeon W, Liwicki M, Schmidhuber J (2015) Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates Inc, Red Hook, pp 2998–3006
Subirats JL, Jerez JM, Gómez I, Franco L (2010) Multiclass pattern recognition extension for the new C-Mantec constructive neural network algorithm. Cognit Comput 2(4):285–290. https://doi.org/10.1007/s12559-010-9051-6
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. vol 07-12-June, pp 1–9. https://doi.org/10.1109/CVPR.2015.7298594, arXiv:1409.4842
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2818–2826
Terekhov AV, Montone G, O’Regan JK (2015) Knowledge transfer in deep block-modular neural networks. Springer, Cham, pp 268–279. https://doi.org/10.1007/978-3-319-22979-9_27
Tyler JR, Wilkinson DM, Huberman BA (2005) E-Mail as spectroscopy: automated discovery of community structure within organizations. Inf Soc 21(2):143–153. https://doi.org/10.1080/01972240590925348
Veit A, Wilber MJ, Belongie S (2016) Residual networks behave like ensembles of relatively shallow networks. In: Advances in neural information processing systems. pp 550–558
Verbancsics P, Stanley KO (2011) Constraining connectivity to encourage modularity in HyperNEAT. In: Proceedings of the 13th annual conference on Genetic and evolutionary computation—GECCO ’11. p 1483. https://doi.org/10.1145/2001576.2001776
Vlahogianni EI, Karlaftis MG, Golias JC (2007) Spatio-temporal short-term urban traffic volume forecasting using genetically optimized modular networks. Comput Aided Civ Infrastruct Eng 22(5):317–325
Waibel A (1989) Modular construction of time-delay neural networks for speech recognition. Neural Comput 1(1):39–46. https://doi.org/10.1162/neco.1989.1.1.39
Wang M (2015) Multi-path convolutional neural networks for complex image classification. arXiv:1506.04701
Wang SJ, Hilgetag CC, Zhou C (2011) Sustained activity in hierarchical modular neural networks: self-organized criticality and oscillations. Front Comput Neurosci 5:30
Wang T, Wu DJ, Coates A, Ng AY (2012) End-to-end text recognition with convolutional neural networks. In: Pattern recognition (ICPR), 2012 21st international conference on. IEEE, pp 3304–3308
Watanabe C, Hiramatsu K, Kashino K (2018) Modular representation of layered neural networks. Neural Netw 97:62–73. https://doi.org/10.1016/J.NEUNET.2017.09.017
Watts DJ (1999) Networks, dynamics, and the smallworld phenomenon. Am J Sociol 105(2):493–527. https://doi.org/10.1086/210318
Weston J, Chopra S, Bordes A (2014) Memory networks. arXiv:1410.3916
Xie S, Girshick R, Dollár P, Tu Z, He K (2016) Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431
Xu L, Krzyzak A, Suen C (1992) Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Trans Syst Man Cybern 22(3):418–435. https://doi.org/10.1109/21.155943
Yu L, Lin Z, Shen X, Yang J, Lu X, Bansal M, Berg TL (2018) MAttNet: modular attention network for referring expression comprehension. arXiv:1801.08186v2
Yu H, Wang J, Huang Z, Yang Y, Xu W (2016) Video paragraph captioning using hierarchical recurrent neural networks. In: The IEEE conference on computer vision and pattern recognition (CVPR)
Yuan M, Lin Y (2006) Model selection and estimation in regression with grouped variables. J R Stat Soc Ser B (Stat Methodol) 68(1):49–67. https://doi.org/10.1111/J.1467-9868.2005.00532.X@10.1111/(ISSN)1467-9868.TOP_SERIES_B_RESEARCH, https://rss.onlinelibrary.wiley.com/doi/full/10.1111/j.1467-9868.2005.00532.x%4010.1111/%28ISSN%291467-9868.TOP_SERIES_B_RESEARCH
Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). LNCS, vol 8689 pp 834–849. https://doi.org/10.1007/978-3-319-10590-1_54, arXiv:1407.3867
Zhang F, Leitner J, Milford M, Corke P (2016) Modular deep Q networks for sim-to-real transfer of visuo-motor policies. arXiv:1610.06781v4
Zhang C, Ren M, Urtasun R (2018) Graph HyperNetworks for neural architecture search. arXiv:1810.05749
Zheng W, Lee DH, Shi Q (2006) Short-term freeway traffic flow prediction: Bayesian combined neural network approach. J Transp Eng 132(2):114–121. https://doi.org/10.1061/(ASCE)0733-947X(2006)132:2(114)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Amer, M., Maul, T. A review of modularization techniques in artificial neural networks. Artif Intell Rev 52, 527–561 (2019). https://doi.org/10.1007/s10462-019-09706-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-019-09706-7