Neural modularity helps organisms evolve to learn new skills without forgetting old skills - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 Apr 2;11(4):e1004128.
doi: 10.1371/journal.pcbi.1004128. eCollection 2015 Apr.

Neural modularity helps organisms evolve to learn new skills without forgetting old skills

Kai Olav Ellefsen  1 , Jean-Baptiste Mouret  2 , Jeff Clune  3
Affiliations

Neural modularity helps organisms evolve to learn new skills without forgetting old skills

Kai Olav Ellefsen et al. PLoS Comput Biol. .

Abstract

A long-standing goal in artificial intelligence is creating agents that can learn a variety of different skills for different problems. In the artificial intelligence subfield of neural networks, a barrier to that goal is that when agents learn a new skill they typically do so by losing previously acquired skills, a problem called catastrophic forgetting. That occurs because, to learn the new task, neural learning algorithms change connections that encode previously acquired skills. How networks are organized critically affects their learning dynamics. In this paper, we test whether catastrophic forgetting can be reduced by evolving modular neural networks. Modularity intuitively should reduce learning interference between tasks by separating functionality into physically distinct modules in which learning can be selectively turned on or off. Modularity can further improve learning by having a reinforcement learning module separate from sensory processing modules, allowing learning to happen only in response to a positive or negative reward. In this paper, learning takes place via neuromodulation, which allows agents to selectively change the rate of learning for each neural connection based on environmental stimuli (e.g. to alter learning in specific locations based on the task at hand). To produce modularity, we evolve neural networks with a cost for neural connections. We show that this connection cost technique causes modularity, confirming a previous result, and that such sparsely connected, modular networks have higher overall performance because they learn new skills faster while retaining old skills more and because they have a separate reinforcement learning module. Our results suggest (1) that encouraging modularity in neural networks may help us overcome the long-standing barrier of networks that cannot learn new skills without forgetting old ones, and (2) that one benefit of the modularity ubiquitous in the brains of natural animals might be to alleviate the problem of catastrophic forgetting.

PubMed Disclaimer

Conflict of interest statement

The authors have declared that no competing interests exist.

Figures

Fig 1
Fig 1. Two hypotheses for how neural modularity can improve learning.
Hypothesis 1: Evolving non-modular networks leads to the forgetting of old skills as new skills are learned. Evolving networks with a pressure to minimize connection costs leads to modular solutions that can retain old skills as new skills are learned. Hypothesis 2: Evolving modular networks makes reward-based learning easier, because it allows a clear separation of reward signals and learned skills. We present evidence for both hypotheses in this paper.
Fig 2
Fig 2. The environment for one individual’s lifetime.
A lifetime lasts 3 years. Each year has 2 seasons: winter and summer. Each season consists of 5 days. In each day, each individual sees all food items available in that season (only two are shown) in a random order.
Fig 3
Fig 3. Randomizing food associations between generations.
To ensure that agents learn associations within their lifetimes instead of genetically hardcoding associations, whether each food item is nutritious or poisonous is randomized each generation. There are four food items per season (two are depicted).
Fig 4
Fig 4. The addition of a cost for network connections, which is present only in the P&CC treatment, significantly increases performance and modularity.
Modularity is measured via a widely used approximation of the standard Q modularity score [23, 57, 65, 67] (Methods). For each treatment, the median from 100 independent evolution experiments is shown ± 95% bootstrapped confidence intervals of the median (Methods). Asterisks below each plot indicate statistically significant differences at p < 0.01 according to the Mann-Whitney U test, which is the default statistical test throughout this paper unless otherwise specified.
Fig 5
Fig 5. Performance each day for evolved agents from both treatments.
Plotted is median performance per day (± 95% bootstrapped confidence intervals of the median) measured across 100 organisms (the highest-performing organism from each experiment per treatment) tested in 80 new environments (lifetimes) with random associations (Methods). P&CC networks significantly outperform PA networks on every day (asterisks). Eating no items or all items produces a score of 0.5; eating all and only nutritious food items achieves the maximum score of 1.0.
Fig 6
Fig 6. PA networks are visually non-modular whereas P&CC networks tend to create a separate module for learning (red and orange neurons), as hypothesized in Fig. 1 (bottom).
Dark blue nodes are inputs that encode which type of food has been encountered. Light blue nodes indicate internal, non-modulatory neurons. Red nodes are reward or punishment inputs that indicate if a nutritious or poisonous item has been eaten. Orange neurons are neuromodulatory neurons that regulate learning. P&CC networks tend to separate the reward/punishment inputs and neuromodulatory neurons into a separate module that applies learning to downstream neurons that determine which actions to take. For each treatment, the highest-performing network from each of the nine highest-performing evolution experiments are shown (all are shown in the Supporting Information). In each panel, the left number reports performance and the right number reports modularity. We follow the convention from [23] of placing nodes in the way that minimizes the total connection length.
Fig 7
Fig 7. Performance is correlated with sparsity and modularity.
Black dots represent the highest-performing network from each of the 100 experiments from both the PA and P&CC treatments. Both the sparsity (p = 1.08 × 10−16) and modularity (p = 1.19 × 10−5) of networks significantly correlates with their performance. Performance was measured in 80 randomly generated environments (Methods). Significance was calculated by a t-test of the hypothesis that the correlation is zero. Notice that many of the lowest-performing networks are close to the maximum of 150 connections.
Fig 8
Fig 8. Comparing the retention and forgetting of networks from the two treatments.
P&CC networks, which are more modular, are better at retaining associations learned on a previous task (winter associations) while learning a new task (summer associations), better at learning new (summer) associations, and significantly better when measuring performance on both the associations for the original task (winter) and the new task (summer). Note that networks were evolved with five days per season, so the results during those first five days are the most informative regarding the evolutionary mitigation of catastrophic forgetting: we show additional days to reveal longer-term consequences of the evolved architectures. Solid lines show median performance and shaded areas indicate 95% bootstrapped confidence intervals of the median. The retention scores (left panel) are normalized relative to the original performance before training on the new task (an unnormalized version is provided as Supp. S6 Fig). During all performance measurements, learning was disabled to prevent such measurements from changing an individual’s known associations (Methods).
Fig 9
Fig 9. P&CC networks significantly outperform PA networks in both learning and retention.
P&CC individuals learn significantly more associations, whether counting only when the associations for both seasons are known (“Perfect” knowledge) or separately counting knowledge of either season’s association (total “Known”). P&CC networks also forget fewer associations, defined as associations known in one season and then forgotten in the next, which is significant when looking at the percent of known associations forgotten (“% Forgotten”). P&CC networks also retain significantly more associations, meaning they did not forget one season’s association when learning the next season’s association. See text for more information about the “Perfect”, “Known”, “Forgotten,” and “Retained” metrics. During all performance measurements, learning was disabled to prevent such measurements from changing an individual’s known associations (Methods). Bars show median performance, whiskers show the 95% bootstrapped confidence interval of the median. Two asterisks indicate p < 0.01, three asterisks indicate p < 0.001.
Fig 10
Fig 10. Forcing individuals to forget what they have learned in the past eliminates the performance benefits of adding a connection cost.
With forced forgetting, P&CC does not significantly outperform PA: P&CC 0.91 [95% CI: 0.91, 0.91] vs. PA 0.91 [0.90, 0.91], p > 0.05. In the default treatment where remembering is possible, P&CC significantly outperforms PA: P&CC 0.94 [0.92, 0.94] vs. PA 0.78 [0.78, 0.81], p = 8.08 × 10−6.
Fig 11
Fig 11. The effect of neuromodulation and connection costs when evolving solutions for catastrophic forgetting.
Connection costs and neuromodulatory dynamics interact to evolve forgetting-resistant solutions. Without neuromodulation, neither treatment performs well, suggesting that neuromodulation is a prerequisite for solving these types of problems, a result that is consistent with previous research showing that neuromodulation is required to solve challenging learning tasks [25]. However, even in the non-neuromodulatory (pure Hebbian) experiments, P&CC is more modular (0.33 [95% CI: 0.33, 0.33] vs PA 0.26 [0.22, 0.31], p = 1.16 × 10−12) and performs significantly better (0.72 [95% CI: 0.71, 0.72] vs. PA 0.70 [0.69, 0.71], p = 0.003). That said, because both treatments perform poorly without neuromodulation, and because natural animal brains contain neuromodulated learning [28], it is most interesting to see the additional impact of modularity against the backdrop of neuromodulation. Against that backdrop, neural modularity improves performance to a much larger degree (P&CC 0.94 [0.92, 0.94] vs. PA 0.78 [0.78, 0.81], p = 8.08 × 10−6), in part by reducing catastrophic forgetting (see text).
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. French R (1999) Catastrophic forgetting in connectionist networks. Trends Cogn Sci 3: 128–135. 10.1016/S1364-6613(99)01294-2 - DOI - PubMed
    1. Mermillod M, Bugaiska A, Bonin P (2013) The Stability-Plasticity Dilemma: Investigating the Continuum from Catastrophic Forgetting to Age-Limited Learning Effects. Front Psychol 4: 504 10.3389/fpsyg.2013.00504 - DOI - PMC - PubMed
    1. Ajemian R, D’Ausilio A, Moorman H, Bizzi E (2013) A theory for how sensorimotor skills are learned and retained in noisy and nonstationary neural circuits. Proc Natl Acad Sci U S A 110: E5078–87. 10.1073/pnas.1320116110 - DOI - PMC - PubMed
    1. Haykin SS (2009) Neural networks and learning machines. New York: Prentice Hall, 3 edition.
    1. Floreano D, Mattiussi C (2008) Bio-inspired artificial intelligence: theories, methods, and technologies. The MIT Press, 659 pp.

Publication types

Grants and funding

KOE and JC have no specific financial support for this work. JBM is supported by an ANR young researchers grant (Creadapt, ANR-12-JS03-0009). URL: http://www.agence-nationale-recherche.fr/en/funding-opportunities/documents/aap-en/generic-call-for-proposals-2015-2015/nc/. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.