Robotically Embodied Biological Neural Networks to Investigate Haptic Restoration with Neuroprosthetic Hands

I. INTRODUCTION

Amputation of an upper limb is a devastating injury that impacts millions of people worldwide [1]. Severance of afferent neural pathways deprives amputees of the rich multimodal sensations of touch afforded by the broad distribution of mechanoreceptors in the human fingertips [2], adversely impacting motor control of prosthetic limbs [3]. The field of neuroprosthetics has tremendous potential to restore severed sensations of touch to amputees by use of electrodes implanted in peripheral nerves of the residual limb [4]. The robotic capability to sense haptic properties is well-established [2] and prosthetic fingertip sensations can be encoded into frequency-modulated spike trains to convey graded biomimetic sensations of touch to amputees outfitted with neuroprosthetic limbs [5]. Mechanoreceptors from the human glabrous skin fall into two broad categories: SA and RA. Each type of mechanoreceptor is responsible to process and encode a different aspect of tactile experience depending on specific spatiotemporal properties and response functions. Generally speaking, SA mechanoreceptors can detect static pressure, texture, and lateral skin stretch while the RA mechanoreceptors are used to sense sliding contact and high frequency vibration [2], among other sensations and modalities combining the two.

Several prior works fruitfully explored the control and robotic embodiment of biological neural networks (BNNs) in closed-loop architectures with MEAs in vitro. Potter and colleagues demonstrated control of neural network bursting by modulating the stimulation voltage based on the culture-wide firing rate [7]. Building upon this, the same group developed multiple closed loop MEA architectures for action control. One developed a stimulation technique for goal-directed motion guidance of an animated display using living cortical neurons [8]. Another demonstrated how the dynamics of BNNs impact control of a robotic arm for an artistic painting display [9] and yet another controlled the motion of a simulated mobile robot [10].

A breakthrough work integrated action and perception. It used a biological interface made from rats’ cortical neurons cultured in MEA chambers. This bidirectional mobile robot control architecture mapped robotic sensory input to stimulate and alter neuronal dynamics, and subsequently impact robotic behavior in an obstacle avoidance paradigm [11]. Compartmentalizing the MEA chamber into different sections produced different system level dynamics due to better separation between the input and output signals, producing simulated obstacle avoidance performances [12].

In a challenging paradigm in vivo, a study examined improvement in neuroprosthetic hand control in a subject who, following amputation, was implanted with a transverse intrafascicular multichannel electrode interface to provide a proportional sense of the grip force via her ulnar nerve, conveying signals that SA mechanoreceptors had provided prior to amputation [3]. The subject successfully demonstrated sensory-motor integration by improving grip force control of the artificial hand. Yet, the subject reported incongruent sensations of vibration (similar to what would naturally be transmitted by RA, not SA mechanoreceptors) [3], revealing the hurdles still existing towards restoring haptic information with neurophenomenological fidelity.

With this inherent complexity, there remains much to learn in the field of touch sensation restoration. Nevertheless, regulatory, ethical, and financial constraints remain considerable challenges for state-of-the-art experimentation in vivo. For these reasons, only a limited number of patients have used bidirectional neuroprosthetic hands thus far [6], limiting research progress.

To circumvent these bottlenecks, we propose a novel noninvasive neuroprosthetic research platform. It enables bidirectional electrical communication between a dexterous artificial hand and living BNN cultured in a MEA (Fig. 1). To demonstrate this platform, RA or SA tactile sensations from the robotic fingertip (Fig. 1(e)-(g)) are used to biomimetically stimulate the neurons in the MEA (Fig. 1(a), red) and the recorded neuronal activity (Fig. 1(a), black) is decoded to control the robotic hand (Fig. 1(b)-(d)). This form of BNN embodiment removes many of the regulatory and financial barricades required for invasive human studies. This platform could catalyze a deeper understanding of the interaction between biological and artificial components of neuroprosthetic limbs with high throughput implementation to aid in restoring severed sensations of touch to amputees.

Recently, ANNs have been used to pattern match inputs and outputs within bidirectional BNNs [16]. In this paper, we will show that ANNs can be used to classify differences in BNN activity due to different pulse train patterns, in this case bioinspired SA or RA encodings of fingertip tactile sensations. Furthermore, we will show that different neuronal activity patterns elicit different robotic behaviors in a closed loop fashion. We will describe the functional architecture of the closed loop neurorobotic system (II), detail its operational algorithms for efferent motor decoding (III), and afferent sensation encoding (IV). We next present our MEA culturing procedure, experimental setup (V), and provide proof of concept that different encoding methods (SA or RA) change both the robotic hand’s behavior and the closed-loop BNN dynamics that govern its behavior (VI). We then provide a short conclusion and outlook (VII).

II. Closed-Loop Neurorobotic System Overview

Two subsystems comprise this closed-loop platform. The first subsystem is the robotic unit (Fig. 2, right panel) which includes a Shadow Hand (Shadow Robot Company, London) fit with BioTac SP tactile sensor array (SynTouch, CA), a PC using Robot Operating System (ROS, Open Robotics, CA) managing efferent (Fig. 2(d), ROS Node 1) and afferent (Fig. 2(h), ROS Node 2) computations. Node 1 uses efferent signaling from the BNNs (Fig. 2(a)-(c)) to compute the motor control signals for the artificial hand. Node 2 converts the BioTac pressure signals into afferent pulse trains of action potentials in real-time (Fig. 2(g),(h)). The pulse trains were passed to a custom-fabricated action potential generator (APG) board (Fig. 2(i)) that triggered neurostimulations according to a neurocomputational model of haptic encoding, discussed in Section IV.

The second subsystem is the neurophysiological unit (Figure 2, left panel), which includes the head stage of the MEA (MultiChannel Systems, Reutlingen, Germany) that houses the BNN culture, signal collector unit, interface board and their interconnection. The MEA has 200μm inter-electrode distance and 30μm diameter electrodes made from titanium nitride. The data acquisition of the MEA relies on a dedicated high-performance workstation optimized for storage capacity, fast data transfer and temporal accuracy of the high-density, high-frequency signals (60 channels, 20kHz) that were sampled from the neuronal culture. Online data were filtered with lowpass and highpass Butterworth filters with cutoff frequencies set to 3.5 kHz and 100 Hz, respectively, to reliably eliminate unstable baselines in real time and avoid artifacts. An additional data stream was preserved for further offline classification with ANNs.

In the present work, a pair of electrodes was selected for their healthy spontaneous activity. One of the sites was assigned the function of recording electrode: it provided the efferent signals to operate the artificial hand. Its spike events (Fig. 2(a)) were routed through ROS node 1 (Fig. 2(d)) to elicit the robotic finger tapping behavior. The other site (Fig. 2(k)) was assigned the function of stimulation electrode, and it used the haptic feedback from the encoded robotic fingertip sensations. To ensure compatibility with spike signaling in the BNN, the tactile signals received from the BioTac sensor were transformed into afferent trains of action potentials (Fig. 1(e)-(g), Fig. 2(g)-(i)) via the Izhikevich model [14].

The SCB-68A DAQ (National Instruments, TX) was placed at the interface between both subsystems (Fig. 2(c)). Real-time feedback tests were performed to quantify the delay between instructed stimulation in Simulink (ROS Node 2), its detection by the MultiChannel System’s proprietary software of the MEA, and return of the signal back to Simulink. Closed-loop latency was consistently between 0.8-1.0 ms.

III. Efferent Decoding For Motor Control

An efferent neurorobotic control signal was implemented in ROS Node 1 (Fig. 2(d)), with spike trains from the recording site of the MEA as input to specify the desired joint angle ($\theta$) of the Shadow Hand’s index finger metacarpophalangeal (MCP) joint. Briefly, the algorithm performed three functions: {1} a thresholding of its biological neural input signal to separate spikes from background noise, {2} a temporal aggregation to recruit MEA neural activity that satisfies criteria over a neurophysiologically-meaningful time interval, and {3} a desired index finger MCP joint angle signal for triggering robotic fingertip tapping motions to generate fingertip forces.

For computational efficiency, we used the extracellular multiunit MEA activity, V_MEA (Fig. 2(a)), from the selected recording electrode as an input to the efferent decoding algorithm for motion control of the Shadow Hand. Spikes (S) were detected as:

where V_thresh is the action potential spike detection threshold. Subsequently, S is integrated over a window of time, BinSize (50ms, (Fig. 1(b),(c))), and compared to a spatiotemporal aggregation coefficient, S_thres (3 spikes to provide a tapping rate within the operational bandwidth of the finger). Then, the algorithm outputs a TTL pulse of 100 ms, MEA_out, determined by:

MEA_out was then passed through a low pass filter. MEA_out affects the desired joint angle (${\theta }_D$) of the Shadow Hand finger:

Desired joint angle ${\theta }_1$ corresponds to a fully open hand, where the fingertip does not contact anything. However, desired joint angle ${\theta }_2$ corresponds to index finger flexion to create fingertip contact with a surface, producing tactile forces. The measured joint angle, $\theta$, is realized by a PID joint angle controller of the tendon-driven Shadow Hand. The joint angle ($\theta$) is related to the fingertip force ($F_{DC}$) by:

where $B(\theta )$ is the inertia matrix, $C(\theta ,\stackrel{.}{\theta })$ is the matrix representing the Coriolis and centrifugal forces, $F_{fr}$ is the viscous friction coefficient, $g(\theta )$ is the vector representing the gravitational effect, ${\tau }_a$ is the actuating joint torque, ${\tau }_{fr}$ is the joint friction, $J$ is the Jacobian matrix of the finger kinematics, and $F_{DC}$ is the contact force at the fingertip [15].

Both $F_{DC}$ and the rate of change of the fingertip force ($F_{AC}$) are used within the Izhikevich neurocomputational model to generate SA and RA afferent action potential pulse trains that electrically stimulated the neuronal cultures (Fig. 1(e)-(g)).

IV. Afferent Encoding of Tactile Sensations

The afferent neurorobotic feedback signals were implemented in ROS Node 2 (Fig. 2(h)) for SA or RA encodings of tactile sensations as the feedback signals to the MEA (Fig. 2(g)-(k)).

V. Experimental Setup and Analysis Strategy

VI. RESULTS

Transfer of information through the noninvasive neuroprosthetic platform began when MEA efferent site activity (V_MEA, Fig. 4(c),(k)) rose above the voltage threshold, V_thresh, to trigger a spike, S (1). When S was triggered S_thresh=3 times within BinSize=50 ms (2) to trigger MEA_out, the desired joint angle of the finger (${\theta }_D$) increased from ${\theta }_1$ to ${\theta }_2$ (3). This caused the finger joint controller to increase the joint angle ($\theta$, Fig. 4(e)) so that the fingertip contacted the environment (Fig. 3(k)), increasing the fingertip force (F_DC, Fig. 4(f), (4)) and force rate of change (F_AC, Fig. 4(g)). Illustrative data from the SA and RA encoding of fingertip forces ((5), (6)) produced spike trains that were used to stimulate MEA_in (Fig. 4(h)). This process repeated cyclically (Fig. 3(a)), for both SA (Fig. 3(b)-(e)) and RA (Fig. 3(f)-(i)) experiments, producing repetitive finger tapping behavior with variable ITI dependent on the afferent tactile encoding.

VII. CONCLUSION

We have created a novel bidirectional noninvasive neuroprosthetic research platform that can be used to study the interaction between living BNNs and embodied robotic systems. Results showed that the encoding of tactile sensations from a robotic fingertip in an SA or RA action potential stimulation pattern impacted the behavior of the BNNs in the MEAs across multiple cultures and days. Correspondingly, the different patterns of coupled BNN activity impacted the behavior of the artificial hand in a closed loop fashion as evidenced by the statistically significant ITIs with both cultures. By demonstrating the ability to classify BNN patterns with ANNs, we show a pipeline toward an online approach to classify tactile interactions through different biomimetic stimulation patterns in the embodied BNN. This platform of robotically-embodied biological neurons could enable a model of invasive experiments to be performed in a noninvasive environment [17], [18], accelerating the rate of neuroprosthetic research for understanding the complex aspects of sensorimotor integration and restoration.

Supplementary Material