A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping - PubMed Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Dec 15;117(50):32124-32135.
doi: 10.1073/pnas.2005087117. Epub 2020 Nov 30.

A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping

Jonathan A Michaels  1   2   3 , Stefan Schaffelhofer  1 , Andres Agudelo-Toro  1 , Hansjörg Scherberger  4   5
Affiliations

A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping

Jonathan A Michaels et al. Proc Natl Acad Sci U S A. .

Abstract

One of the primary ways we interact with the world is using our hands. In macaques, the circuit spanning the anterior intraparietal area, the hand area of the ventral premotor cortex, and the primary motor cortex is necessary for transforming visual information into grasping movements. However, no comprehensive model exists that links all steps of processing from vision to action. We hypothesized that a recurrent neural network mimicking the modular structure of the anatomical circuit and trained to use visual features of objects to generate the required muscle dynamics used by primates to grasp objects would give insight into the computations of the grasping circuit. Internal activity of modular networks trained with these constraints strongly resembled neural activity recorded from the grasping circuit during grasping and paralleled the similarities between brain regions. Network activity during the different phases of the task could be explained by linear dynamics for maintaining a distributed movement plan across the network in the absence of visual stimulus and then generating the required muscle kinematics based on these initial conditions in a module-specific way. These modular models also outperformed alternative models at explaining neural data, despite the absence of neural data during training, suggesting that the inputs, outputs, and architectural constraints imposed were sufficient for recapitulating processing in the grasping circuit. Finally, targeted lesioning of modules produced deficits similar to those observed in lesion studies of the grasping circuit, providing a potential model for how brain regions may coordinate during the visually guided grasping of objects.

Keywords: electrophysiology; grasping; motor control; primates; recurrent neural networks.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interest.

Figures

Fig. 1.
Fig. 1.
Frontoparietal grasping circuit and experimental design. (A) Simplified brain schematic of the frontoparietal grasping circuit. Visual information is processed in two parallel streams carrying primarily object features or identity information, both converging on the anterior intraparietal sulcus (AIP). AIP has strong reciprocal connections with the F5 of the ventral premotor cortex, which has strong reciprocal connections to the hand area of the M1. M1 has the majority of subcortical and spinal cord output projections. (B) Location of implanted floating microelectrode arrays, covering the three desired regions. Black dots represent ground and reference electrodes. A, anterior; AS, arcuate sulcus; CS, central sulcus; IPS, intraparietal sulcus; L, lateral; M, medial; P, posterior. (C) Monkeys sat in front of a motorized turntable that presented one of six objects to be grasped on any given trial. Multiple turntables presented in random order across sessions allowed for a total of 42 objects. Gloves with magnetic sensors allowed full tracking of arm and hand kinematics on single trials. Modified from ref. , which is licensed under CC BY 4.0. h. and v. correspond to horizontal and vertical cylinders, respectively. (D) Trials began with visual fixation of a red dot for a variable period. Objects were illuminated temporarily, and monkeys were required to withhold movement until a go cue (blinking of fixation dot) instructed them to grasp and lift the object in darkness. Eye fixation was enforced throughout each trial.
Fig. 2.
Fig. 2.
Visual and kinematic features explain neural activity across the frontoparietal grasping circuit. (A) Simulated images of all objects were fed through a multilayer CNN pretrained to classify objects (VGG) (Methods). (B) Single-trial neural activity of each unit averaged during the cue period was regressed (leave one out cross-validated) against the representation of all objects in each layer of the CNN (first 20 PCs), and the median fit was taken over all units within one recording session. The solid lines and error surfaces represent the means and SEMs, respectively, over all recording sessions of each monkey. To ensure that results were not due to varying signal quality or firing rate between areas, regression results were normalized to the noise ceiling of each unit (Methods). conv refers to layers using convolution, relu to layers using rectified linear units, and prob to the probability layer. (C) Joint angles (27 degrees of freedom) recorded while monkeys performed the task were transformed into muscle length space (50 degrees of freedom) using a musculoskeletal model (Methods). For visualization purposes, not all muscles are shown. (D) Single-trial neural activity of each unit averaged during the movement initiation period (200 ms before to 200 ms after movement onset) was regressed (leave one out cross-validated) against the inferred muscle velocity of all grasping conditions averaged over the same time period. As in B, regression results were normalized to the noise ceiling of each unit. Each point represents one recording session of each monkey. (E) Example neural representation (first two PCs) of each object across all three areas during the cue period and during movement initiation (session M7). The size of each marker indicates the relative size of each grasping object. h. and v. correspond to horizontal and vertical cylinders, respectively. (F) Same as E but for session Z9.
Fig. 3.
Fig. 3.
The mRNN model of the frontoparietal grasping circuit. (A) Schematic of neural network model. Visual features of each object (first 20 PCs of relu5_4 layer) were fed into an input module, which is sparsely reciprocally connected to an intermediate module that is similarly connected to an output module. The output module must recapitulate the inferred muscle velocity for every object grasped by the monkey. Every module received a hold signal that was released 200 ms prior to movement onset. (B) Example input for an exemplary trial. (C) Average muscle velocity for four example muscles showing exemplar recorded kinematics and network output (session M2). EDCL, extensor digitorum communis digit 5; FDSL, flexor digitorum superficialis digit 5; FPL, flexor pollicis longus; TRIlong, triceps long head. h. and v. correspond to horizontal and vertical cylinders, respectively. (D) Two example units from each pair of modules and brain regions showing similar properties and highlighting common features of each area. Traces were aligned to two events, cue onset and movement onset, and concatenated together. The shaded gray areas represent the cue period, while the dashed lines represent movement onset. (E) Procrustes analysis (Overall Fit) comparing the dynamics of an exemplar mRNN model (ReTanh activation function, rate regularization: 1e-1; weight regularization: 1e-5; intermodule sparsity, 0.1) with neural data recorded from AIP, F5, and M1 (session M2). For visualization purposes, after model data were fit to neural data it was projected onto the first six PCs defined on the neural data, and percentages show variance explained (var expl) in the neural data per PC. (F) Pairwise Procrustes was performed between each brain region and a resampled version of its own activity (Upper) or between each module and brain region (Interarea Fit; Lower). Individual rows and columns specify from top to bottom and from left to right either the output, intermediate, and input module or M1, F5, and AIP, respectively. (G and H) Same as E and F but for session Z4. For C–E and G, the multiple traces for each type of object represent the different sizes within a turntable.
Fig. 4.
Fig. 4.
Modular mechanisms of memory and movement execution in the mRNN model. Fixed point analysis was performed to understand the computational mechanism used by the mRNN model to complete the task. (A, Left) The single fixed point of an exemplar mRNN model (same parameters as in Fig. 3) during the memory period (cue offset + 50 ms to cue offset + 500 ms) plotted in the first three model PCs alongside condition average activity. The eigenvectors of the two largest eigenvalues are plotted (gray), scaled by the magnitude of the eigenvalue. (A, Right) Condition average neural data from an example session (M6) over the same time period projected into the activity of the model using Procrustes. h. and v. correspond to horizontal and vertical cylinders, respectively. (B) Replacing the full nonlinear model with the linearized system around the fixed point yields very similar trajectories. (C) The complex eigenvalue spectrum for the fixed point in A and B (Inf corresponds to modes that do not decay). To determine which eigenvalues were essential for the linear dynamics, we removed individual eigenvalues and reran the dynamics (Methods), showing that removing certain eigenvalues decreased the variance explained (var explained) in the full model. (D) An example of removing one of the more important eigenvalues, in this case with a decay close to Inf and an oscillation frequency close to zero. In the equivalent movement period example, the large oscillatory mode was removed. (E) To determine which modules were contributing to the most essential eigenvalues, we damped synaptic weights within each module and recalculated the linear stability of the Jacobian (Methods), showing how the eigenvalues shift. For the memory period, the three most important eigenvalues, which all had slow decay, were distributed across the three modules. (F and G) Same analyses as in A–E for the movement period (150 ms before movement onset to 400 ms after movement onset). For the movement period, the oscillatory mode was localized in the output module, while a slow, nonoscillatory mode was localized mostly in the intermediate module.
Fig. 5.
Fig. 5.
mRNN outperforms tested alternative models in explaining neural data in the grasping circuit. (A) Average neural variance explained per recording session for the best set of regularization parameters for each architecture (averaged over five runs) for each of the three proposed metrics, Overall Fit (A), Area-Wise Fit (B), and Interarea Fit (C). Horizontal bars represent the mean, and each dot represents a single session. We tested five alternative models in addition to the Full model: 1) mRNN model with only feed-forward connections between modules; 2) mRNN model receiving a labeled line input (one hot), where each condition is represented by a separate input dimension; 3) mRNN model with output conditions shuffled (objects reassigned); 4) homogeneous, fully connected network; or 5) a single, sparsely connected network with the total number of synaptic connections matched to the Full model. *Significant difference as compared with the Full model (paired t test, P < 0.01). (D, F, and H) Procrustes analysis (Overall Fit) comparing the dynamics of two exemplar models with neural data across all brain regions (session M2). For visualization purposes, after model data were fit to neural data they were projected onto the first six PCs defined on the neural data, and percentages show variance explained (var expl) in the neural data per PC. h. and v. correspond to horizontal and vertical cylinders, respectively. Pairwise Procrustes was performed (E) between each brain region and a resampled version of its own activity or (G and I) between each module and brain region (Interarea Fit). Individual rows and columns specify from top to bottom and from left to right either the output, intermediate, and input module or M1, F5, and AIP, respectively. (F) Exemplar model with the parameters (homogeneous model, ReTanh activation function, L2 rate regularization, 1e-1; L2 weight regularization, 1e-5; intermodule sparsity, 0.1). (H) Exemplar model with the parameters (condition-shuffled output model, ReTanh activation function, rate regularization, 1e-1; weight regularization, 1e-5; intermodule sparsity, 0.1). For D, F, and H, the multiple traces for each type of object represent the different sizes within a turntable.
Fig. 6.
Fig. 6.
Targeted lesioning of rate-regularized modular networks produces unique behavioral deficits. (A) Activity within 240 networks was artificially silenced, varying the rate and weight regularizations, percent of units silenced (repeated 100 times with random units), and the module being silenced. (B) Average change in normalized kinematic error after silencing as compared with normal operation. (C) Changes in network behavior as a function of module and number of units silenced for an exemplar network with high robustness to silencing (i.e., high rate regularization). Premature movement refers to the change in variance of output behavior before movement was initiated. Kinematic error refers to the change in normalized kinematic error during the movement period. Movement amplitude refers to the change in absolute movement kinematics during the movement period. Generic kinematics refers to the change in normalized error in output behavior as compared with the mean kinematic behavior across all conditions. Shaded error bars represent SEM over 100 lesion repetitions. (D) Example network output before movement where 20% of the units in each module were silenced (example condition: midsized cylinder). (E) Example network output during the movement without silencing (Upper; midsized cylinder) and when 20% of the units in each module were silenced (Lower), showing that hand shape was not matched to the object (midsized cylinder) when the input or intermediate module was silenced (similar to the grip for the small cube or ball) while being degraded by silencing of the output module.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Luppino G., Murata A., Govoni P., Matelli M., Largely segregated parietofrontal connections linking rostral intraparietal cortex (areas AIP and VIP) and the ventral premotor cortex (areas F5 and F4). Exp. Brain Res. 128, 181–187 (1999). - PubMed
    1. Schaffelhofer S., Agudelo-Toro A., Scherberger H., Decoding a wide range of hand configurations from macaque motor, premotor, and parietal cortices. J. Neurosci. 35, 1068–1081 (2015). - PMC - PubMed
    1. Fluet M.-C., Baumann M. A., Scherberger H., Context-specific grasp movement representation in macaque ventral premotor cortex. J. Neurosci. 30, 15175–15184 (2010). - PMC - PubMed
    1. Baumann M. A., Fluet M.-C., Scherberger H., Context-specific grasp movement representation in the macaque anterior intraparietal area. J. Neurosci. 29, 6436–6448 (2009). - PMC - PubMed
    1. Murata A., et al. , Object representation in the ventral premotor cortex (area F5) of the monkey. J. Neurophysiol. 78, 2226–2230 (1997). - PubMed

Publication types

LinkOut - more resources