Cognitive control over learning: creating, clustering, and generalizing task-set structure - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2013 Jan;120(1):190-229.
doi: 10.1037/a0030852.

Cognitive control over learning: creating, clustering, and generalizing task-set structure

Anne G E Collins  1 , Michael J Frank
Affiliations

Cognitive control over learning: creating, clustering, and generalizing task-set structure

Anne G E Collins et al. Psychol Rev. 2013 Jan.

Abstract

Learning and executive functions such as task-switching share common neural substrates, notably prefrontal cortex and basal ganglia. Understanding how they interact requires studying how cognitive control facilitates learning but also how learning provides the (potentially hidden) structure, such as abstract rules or task-sets, needed for cognitive control. We investigate this question from 3 complementary angles. First, we develop a new context-task-set (C-TS) model, inspired by nonparametric Bayesian methods, specifying how the learner might infer hidden structure (hierarchical rules) and decide to reuse or create new structure in novel situations. Second, we develop a neurobiologically explicit network model to assess mechanisms of such structured learning in hierarchical frontal cortex and basal ganglia circuits. We systematically explore the link between these modeling levels across task demands. We find that the network provides an approximate implementation of high-level C-TS computations, with specific neural mechanisms modulating distinct C-TS parameters. Third, this synergism yields predictions about the nature of human optimal and suboptimal choices and response times during learning and task-switching. In particular, the models suggest that participants spontaneously build task-set structure into a learning problem when not cued to do so, which predicts positive and negative transfer in subsequent generalization tests. We provide experimental evidence for these predictions and show that C-TS provides a good quantitative fit to human sequences of choices. These findings implicate a strong tendency to interactively engage cognitive control and learning, resulting in structured abstract representations that afford generalization opportunities and, thus, potentially long-term rather than short-term optimality.

PubMed Disclaimer

Figures

Figure 1
Figure 1. Task-set clustering vs perceptual category clustering
A task-set defines a set of (potentially probabilistic) stimulus-action-outcome (S-A-O) contingencies, depicted here with deterministic binary outcomes for simplicity. To identify similarity between disparate contexts pointing to the same latent task-set (left), the agent has to actively sample and experience multiple distinct S-A-O contingencies across trials (only one S and one A from the potentially much larger set is observable in a single trial). In contrast, in perceptual category learning, clustering is usually built from similarity among perceptual dimensions (shown simplistically here as color grouping, right), with all (or most) relevant dimensions observed at each trial. Furthermore, from the experimenter perspective, subject beliefs about category labels are observed directly by their actions; in contrast, abstract task-sets remain hidden to the experimenter (e.g., the same action can apply to multiple task-sets and a single task-set consists of multiple S-A contingencies).
Figure 2
Figure 2. Paradigms used to assess task-set clustering as a function of clustering parameter α
Top: Initial clustering benefit task: demonstration of advantage to clustering during a learning task in which there are 16 “redundant” contexts, signifying just two distinct TS (see protocol on the left). Speeded learning is observed for structured model (with low Dirichlet parameter α = 1, thus high prior for clustering), compared to flat learning model (high α, so that state-action-outcome mappings are learned separately for each context). Bottom: Structure transfer task: Effect of clustering on subsequent transfer, when there is no advantage to clustering during initial learning (protocol on left). Bottom middle. Proportion of correct responses as a function of clustering α parameter in the first 10 trials, for C3 transfer (blue) and C4 new (green) test conditions. Large α’s indicate a strong prior to assign a new hidden state to a new context, thus leading to no performance difference between conditions. Low α’s indicate a strong prior to re-use existing hidden states in new contexts, leading to positive transfer for C3, but negative transfer for C4, due to the ambiguity of the new task-set. Bottom right. Example of learning curves for C3 and C4, and error repartition pattern (Inset). NC: neglect C errors, NS: neglect S errors, NA: neglect all errors.
Figure 3
Figure 3. Neural network models
Top: Schematic representation of a single loop corticostriatal network. Here, input features are represented in two separate input layers. Bottom: schematic representation of the two loop corticostriatal gating network. Color context serves as input for learning to select the TS in the first PFC loop. The PFC TS representation is multiplexed with the shape stimulus in the parietal cortex, the representation of which acts as input to the second motor loop. Before the TS has been selected, multiple candidate TS representations are active in PFC. This TS-conflict results in greater excitation of the subthalamic nucleus in the motor loop (due to a diagonal projection), thus making it more difficult to select motor actions until TS conflict is resolved. PMC: Premotor cortex; STN: subthalamic nucleus; Str: Striatum; Thal: Thalamus; GPe: Globus Pallidus external segment; GPi: Globus Pallidus internal segment.
Figure 4
Figure 4. Neural network model
Top: Detailed representation of the two-loop network. See text for detailed explanation of connectivity and dynamics. Parametrically manipulated projection strengths are highlighted: (1) connectivity between color input and PFC (fully connected vs. one-to-one organized C-PFC mapping, which increases the likelihood that the network assigns distinct PFC states to distinct contexts); (2) STN to GPi strength (modulating the extent to which motor action selection is inhibited given conflict at the level of PFC task-set selection); (3) diagonal PFC to pStr connection strength (modulating task-set motor action preparation); (4) pStr learning rate. PFC: Prefrontal cortex; PC: Parietal cortex; PMC: Premotor cortex; STN: subthalamic nucleus; Str: Striatum; Thal: Thalamus; GPe: Globus Pallidus external segment; GPi: Globus Pallidus internal segment; SNc: Substancia nigra pars compacta. a and p indicate anterior and posterior loops. Bottom Example of the time course of PFC activations (for chosen and other TS), average STN activity and chosen motor output unit activity in correct stay and switch trials. In switch trials, co-activation of PFC stripes results in stronger STN activation, thus preventing action selection in the motor loop until conflict is resolved, leading to increased reaction times.
Figure 5
Figure 5. Neural Network Simulation 1
Top Right: Experimental design summary Left Learning curves for different conditions. Top: 75% of networks adequately learned to select a common PFC representation for the two contexts corresponding to the same rule, and thus learned faster (clustering networks). Bottom: the remaining 25% of the networks created two different rules for C0 and C1, and thus showed no improved learning. Bottom Middle Performance advantage for the clustering networks was significantly correlated with the proportion of trials in which the network gated the common PFC representation. Bottom Right Quantitative fits to network behavior with the C-TS model showed a significant increase in inferred number of hidden TS for clustering compared to non-clustering simulations.
Figure 6
Figure 6. Neural network results
Top a-c): test phase results, for Transfer (blue), New-overlap (green) and New-incongruent (red) conditions. Left: Proportion of correct trials as a function of input repetitions, inset: proportion of NC, NS and NA errors. Positive transfer is visible in the faster Transfer than New learning curves; Negative transfer is visible in the interaction between condition and error types and in the slower slope in New-overlap than New-Incongruent conditions. Right: Proportion of task-set TS1 (b), and blank TS (c) hidden state selections as a function of trials, for all conditions. Positive transfer is visible in the reuse of TS1 stripe in the transfer condition, and negative transfer in the reduced recruitment of the new TS stripe for new-overlap compared to new-incongruent conditions. Bottom: Asymptotic learning phase results. d): reaction-time switch-cost; e) error type and switch effects on error proportions. f): slower reaction-times for neglect L than neglect H errors.
Figure 7
Figure 7. Neural Network parameter robustness
Exploration of systematic modulations of key network parameters across a wide range. For each parameter, the significance values are plotted for each of five main behavioral effects (see descriptions in main text), from top to bottom: 1) Transfer versus new-overlap performance difference; 2) asymptotic learning phase error repartition effect; 3) asymptotic learning phase error reaction times NH < NL; 4) Test-phase old > new PFC stripe selection for the transfer condition; 5) Test phase new > old PFC stripe selection for the new condition. Simulations were conducted 100 times each, in each case with the other 4 parameters fixed to the corresponding white bar value, and 1 parameter varied along a wide range. 1st line: Cortico-striatal learning rate (here fixing learning rates to be the same for both loops); 2nd line: motor-cortex striatum learning rate; 3rd line: PFC-striatum learning rate; 4th line: Diagonal PFC-posterior striatum relative projection strength; 5th line: STN to 2nd loop GPi relative projection strength. Results across all five effects were largely robust to parameter changes.
Figure 8
Figure 8. Linking corticostriatal neural network to C-TS model
Mean C-TS fitted parameters are plotted against manipulated neural network parameters used for corresponding simulations. Diagonal PFC-STN projection strength was related to noise in TS selection; diagonal PFC-striatum was related to within-TS noise. C-PFC connectivity was related to clustering prior; Motor striatal learning rate was related to action learning rate parameter.
Figure 9
Figure 9. Effects of context-PFC prior connectivity
50 neural network simulations per C-PFC prior parameter value. The C-PFC parameter scales the weight of the organized (one-to-one) context input to task-set PFC layer projection relative to the fully connected uniform projection. a) Mean (standard error) performance and b) Proportion of new stripe selection on the transfer (blue) and new-overlap (green) test conditions as a function of C-PFC prior parameter. The stronger the prior for one-to-one connectivity, the more likely the network is to select a new stripe for new contexts 50 in the test phase, thereby suppressing any difference in performance between the three test conditions. Conversely, a greater ability to arbitrarily gate contexts into PFC stripes allows networks to re-use stripes when appropriate.
Figure 10
Figure 10. Experimental protocol
Left: experimental phases. The learning phase is comprised of pseudo-randomly intermixed colored shapes (in this example), comprising shapes S1 and S2, and colors C1 and C2. Each input combination is presented up to a fixed learning criterion, followed by a 10 trial (per input) asymptotic learning phase. Next, the test phase comprises 20 trials per four new inputs, comprising previous shapes in new colors. There is no break in between phases. Middle: Correct input-action associations. Right: Example of correct input-action associations with colors and shapes as context and stimuli. Note that correct actions for red shapes in the learning phase can be re-applied to the blue shapes in the test phase. Thus we refer to the blue condition as ‘transfer’. In contrast, in the ‘new’ green condition, there is no single previous task-set that can be re-applied (one shape-action taken from red and the other from yellow), thus a new task-set. Colors and shapes are used here for simplicity of presentation, but other visual dimensions could play the role of C or S, in a counter-balanced across subjects design. Associations between fingers and actions was also randomized.
Figure 11
Figure 11. Various Model Predictions
(a,c,e,f) Graphical representation of model information structures. Grey areas represent test-phase only associations. (b,d,g) Model test phase predictions for the transfer condition (blue) and the new condition (green): Proportion of correct responses as a function of input repetition, inset: proportion of errors of type; neglect color (NC), neglect shape (NS) or neglect all (NA). Model simulations were conducted using parameters chosen for best model performance within a qualitatively representative range, over 1000 repetitions. a) Flat model: all input-action associations are represented independently of each other (ie conjunctively). b) The flat model predicts no effect of test condition on learning or error type. c) Dimension-experts model. Appropriate actions for shapes and colors are represented separately. In the test phase the shape expert does not have any new links to learn (no new shapes, no new correct actions for the old shapes in new colors), while the color expert learns links for the new colors. d) No effect of test condition in this model, but a main effect of error type. e) S-TS(c) structure model: shape acts as a context for selecting task-sets that determine color stimulus-action associations, so that new test-phase colors are new stimuli to be learned within already created task-sets. Predictions for this model are qualitatively the same as for the dimension experts model (d). f) C-TS(s) structure model: color context determines a latent task-set, that contextualizes the learning of shape stimulus-action asociations. The C3 transfer context may be linked to TS1, whereas the C4 new context should be assigned to a new task-set. Curbed arrows indicate different kinds of errors: NS, NC or NA. g) C-TS(s) model predicts faster learning for test transfer condition than test new condition, and an interaction between condition and error type.
Figure 12
Figure 12. Test-phase behavioral results
(a,c-e): Proportion of correct responses as a function of input repetition. Insets: proportion of errors of type neglect color (NC), neglect shape (NS) or neglect all (NA). Blue: C3 transfer condition; green: C4 new condition. a) Whole group results (N=33). As predicted by C-TS(s) model, there was faster learning in the transfer condition, and a significant interaction between error type and condition. b) Color minus Shape switch-cost difference is predictive of performance differences between transfer and new conditions across the first 10 trials. Switch-costs are normalized sums of reaction-time and error switch-cost, in arbitrary measure. c) Group1 (N=11 highest C-switch-cost subjects). There was a significant positive transfer effect on learning curves, and negative transfer effect on error types, as predicted by the C-TS(s) model. d) Group2 (N=11) Again, significant positive transfer effect on learning curves, though non significant negative transfer effect. e) Group3 (N=11 highest S-switch-cost subjects) No positive transfer effect, and main effect of error type on error proportions, as predicted by the S-TS(c) model.
Figure 13
Figure 13. Asymptotic learning phase errors
Top-left One-to-one encoding of chosen actions as Correct, NH, NL or NA, as a function of trial input. Top-right Correct actions table for asymptotic learning phase, represented here with color as high dimension context, and shape as low dimension stimulus. Bottom-left, middle proportion of trials as a function of error types, for high and low dimension switch trials (swH and swL) or stay trials (stH and stL), for C-Structure (left) and S-Structure (middle) goups. Bottom-right High dimension switch error reaction-times were faster than those for low dimension switches.
Figure 14
Figure 14. Model-fitting
Top Left: Difference in pseudo-r2 fit value between C-TS(s) vs. S-TS(c) structure model, and overall structure vs. flat models. Group 1 and 3 are better fit by structure than flat, respectively by C and S- TS structure models. Differences in fit values are small because model prediction differences are limited to few trials mostly in the beginning of the test phase. Top Right: Predicted hybrid model probabilities using individual subject-fitted parameters against observed probabilities. Bottom: mean attentional for the 3 experts in competition within a single model confirm results from the separate fits.
Figure 15
Figure 15. Generalized structure model
In the above depiction we considered models for representing different sorts of structure, C-TS, S-TS, or flat. The generalized structure model represents all of these as potential descriptors of the data, and infers which one is more valid. We considered two ways to approach this issue: the first uses a mixture of experts architecture in which each expert learns assuming a different sort of structure, and then weights them according to their inferred validity for action selection. The second strategy considers all of the potential structures within the generative model itself. Both models produced similar behavior and predictions.
Figure 16
Figure 16. Generalized structure model results
Example simulation of general structure model. Left panel: model performance on transfer task. Qualitative results are similar to C-TS model predictions. Right panel: average attentional weights. During the training phase, no structure is a better predictor of outcomes. However, the model infers the C-TS structure over the test phase.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Acuña DE, Schrater P. Structure learning in human sequential decision-making. PLoS computational biology. 2010;6(12):e1001003. - PMC - PubMed
    1. Aisa B, Mingus B, O’Reilly R. The emergent neural modeling system. Neural networks: the official journal of the International Neural Network Society. 2008;21(8):1146–52. - PubMed
    1. Akaike H. A new look at the statistical model identification. IEEE Transactions on Automatic Control. 1974;19(6):716–723.
    1. Aldous D. Exchangeability and related topics. École dÉté de Probabilités de SaintFlour XIII 1983. 1985;1117(2):1–198.
    1. Alexander G, DeLong M. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annual review of neuroscience. 1986 - PubMed

Publication types