Shape-sensing by self-sensing of shape memory alloy instruments for minimal invasive surgery

1.1 Motivation

Minimally invasive surgery (MIS) is a type of surgical procedure that involves small incisions and specialized surgical instruments to perform medical procedures. Compared to traditional surgery, MIS has several benefits for patients, medical experts, and hospitals. One of the main advantages of MIS is that it is less traumatic for patients due to smaller incisions, with the consequence of reduced risk of complications such as infections and bleeding [1]. This leads to less pain and discomfort for patients, and thus to a shorter recovery time and less scarring [2].

Flexible instruments in particular can be useful in MIS because they provide greater maneuverability and access to hard-to-reach areas within the body in comparison with rigid instruments, while reducing the risk of injury to surrounding tissue and organs. While passive flexible instruments (e.g., steerable needles) deform due to interaction with tissue only, active flexible instruments provide in addition an actuation that allows to change their shape.

Shape memory alloys (SMAs) (e.g. nickel-titanium) are smart materials that autonomously find back into a preset original shape when undergoing a crystalline phase transition from martensite to austenite due to temperature changes. This effect is referred to as the shape memory effect and can be utilized in a flexible instrument to obtain multiple segments with individual actuation [3].

Despite the useful dexterity of a flexible instrument based on SMAs, their control within the human body remains challenging. The shape of the instrument within the body cavity depends on various influencing factors. Those factors are mainly the applied force at the external (proximal) end of the instrument, the mechanical property of the surrounding tissue, and the temperature of the SMA. Visual imaging devices (e.g., endoscopic cameras) that monitor the flexible instrument and the site of operation are thus crucial for clinicians to manipulate the instrument effectively. However, ideal imaging is not always provided during a medical procedure due to the lack of additional endoscopic devices, limited access into the confined space, or an impaired line of sight caused by body fluids and smoke.

For this reason, additional intraoperative information about the shape and the pose of the instrument’s distal tip are highly beneficial to the operator. Due to the confined space in the cavity and the compactness of the delicate instruments themselves, attaching and operating additional sensor technology (e.g., resistance based flex sensors [4], fiber grading sensors [5]) is often hardly feasible. Instead, flexible instruments with inherent actuation such as SMAs could use the structural material for both, actuating and sensing. They not only provide intrinsic actuation once heated, but also present a variability of their electrical resistance (ER) in accordance with the change of their shape (i.e., self-sensing) [6].

The change of the ER as an intrinsic property has been utilized in research in order to determine the strain of a nickel-titanium alloy wire, e.g., Ikuta et al. [7] and Prechtl et al. [8]. Kaiser et al. have developed measurement and control electronics to use a microcontroller to heat a SMA using electric current and measure the ER [9], while Ma et al. implemented the control of a lengthening SMA wire using a neural network [10]. Furthermore, grasping applications driven by SMA wire actuators were investigated by Wang et al. who controlled the deformation of a flexible gripper to grasp deformable objects [11]. Since the ER changes as a function of the wire strain, a control based on self-sensing could be demonstrated. In addition, Lan et al. were able to control the position of the tip of a flexible micro-gripper [12]. They used a spring to elongate a SMA wire and forced it to contract by electric current (i.e., Joule heating). The authors followed the approach of length control further and investigated a data-driven approach using a polynomial model [13].

1.2 Contribution

Previous research has predominantly focused on investigation of one-dimensional strain variations of SMAs and has even demonstrated controlled actuation for defined loading conditions based on electrical resistance. However, the potential of SMAs to perform complex three-dimensional shape changes has not been fully explored, even though such flexibility could be highly beneficial in the design of minimally invasive surgical instruments to reach further within confined spaces. This work presents a proof-of-concept modeling approach that correlates the electrical resistance of a simplified SMA-based actuator with its shape, enabling two-dimensional shape changes in novel smart minimally invasive instruments (see Figure 1).

Figure 1:

Vision of a minimally invasive surgery (MIS) with a flexible instrument made of a shape memory alloy (SMA). Augmenting the video endoscopy with information about deflection s and interaction forces F.

2.1 SMA actuator

For the actuator, we use a nickel-titanium-copper (NiTiCu) wire (Ø 0.7 mm, 230 mm long). Nickel and titanium form the basic components for an alloy with shape memory effect. By adding copper, characteristics can be adjusted, such as the transformation temperatures, i.e. austenite starting temperature A _s and austenite finish temperature A _f, as well as the hysteresis of the temperature-shape correlation [14]. The wire is then bent into a plane U-shaped loop with a bending radius at the tip of 5 mm (see Figure 2a). This shape is first cold-formed and then “imprinted” (i.e. shape-setting) by heating it to 425 °C and keeping it at that temperature for 30 min [15]. Afterwards it is quenched in water (<25 °C).

Figure 2:

Simplified actuator design with a flat U-shaped wire from NiTiCu (a). Schematic of the shape changes between martensite (blue) and austenite states at A _f = 65 °C (red). Known parameters include power supply voltage U _W, ambient temperature T _A, and load. Current I _W, force F, deflection s, and wire temperature T _w are measured during the experiments.

This basic straightened (austenite) shape can be cold-formed (i.e. bent) under an external load. If the bent wire is then heated above the austenite finish temperature (here: A _f = 65 °C), the original straight austenite form is restored (see Figure 2b). This effect of SMAs can be used to apply mechanical forces [16]. The aforementioned design allows to power the instrument at its proximal end, using Joule heating to increase the temperature of the wire, thus avoiding an interference of cable and connectors at the instrument’s body.

2.2 Experimental setup

The defined and measured values are schematically illustrated in Figure 2b. The inherent electric resistance of the wire R can be defined as

where U _W is the voltage applied at the proximal wire ends and I _W the electric current in the wire. Further experimental parameters of interest include the wire surface temperature T _W, the deflection s at the distal tip, as well as the applicable distal force F. The experimental setup is shown in Figure 3. The SMA actuator is mounted horizontally with its two proximal ends in a fixed clamp. In the unloaded state, the tip of the wire is aligned approximately at the horizontal level of the clamping. The actuator can be powered via the clamped wire ends and its temperature will increase due to Joule heating. This allows to trigger a phase transformation from various cold-formed shapes back into the set horizontal austenite shape.

Figure 3:

Experimental setup for data acquisition of the SMA actuator. Two load cells (red) record reaction forces F _x and F _z at the clamping. Tracking ArUco markers at the clamping and the distal tip (yellow) allow to measure the deflection s.

The electronics to power the wire and record the measurement data at the same time are controlled by a microcontroller (Arduino Uno Rev3 SMD). To measure the current I _W at a given voltage U _W, a measuring resistor (Isabellenhütte PBV R010, Germany) is used. Two operational amplifier circuits (Texas Instruments TLC 2272, USA) are implemented to amplify the voltage drops at the measuring resistor and the SMA wire. Then, R can be calculated with (1). The setup provides the restistance R of the SMA wire with a resolution of 1 mΩ and an accuracy of 3 mΩ.

To measure the force components of F = [F _x , F _z ] on the wire clamping, two load cells with measuring amplifier (Sparkfun Electronics TAL221, HX711, USA) are used. The signals are calculated and processed by the microcontroller and transmitted via Serial connection to a host computer. In the heating phase, F and R values are measured with a frequency of up to 385 Hz, limited by the electronic evaluation system of the load cells. Data acquisition in the cooling phase is conducted with 1.75 Hz due to fact that R could only be measured when current flows through the actuator. Consequently, the current rate must be kept at a minimum during cooling phase to avoid additional unintended Joule heating.

We developed a measurement script in Python (3.9) to track the position of the wire tip with an ArUco marker and a visual camera (Intel Corporation RealSense™ D435, USA), as well as to compute and store the position data. Preliminary tests revealed a marker tracking position accuracy of 0.2 mm. A thermal imaging camera (Teledyne FLIR LLC FLIR E60, USA) is used for contactless temperature monitoring of the wire surface T _W. The ambient temperature T _A is the constant room temperature (22.5 °C) of the laboratory and is monitored with an additional sensor. The temperature fluctuation remained within a range of 2 °C.

To bend the actuator at ambient temperature, weights are attached to the actuator’s distal tip. The load pulls the tip down in negative z direction. If the actuator is then powered, the shape memory effect “opposes” the shape change and lifts the weight. When the power is turned off and the temperature decreases, the wire again deforms by the gravitational force of the weight.

The generalized deflection s is utilized as a simplified measure of the bent shape of the wire (see Figures 2 and 4). In this work, s corresponds to z. It should be noted, that due to many past actuation cycles of the actuator (n > 300), a two-way shape memory effect was observed [17]. The bent shape in martensite phase had become part of the material’s memory. Thus, the bent shape was obtained even in cases without any load. This way, a common starting point between s = −138 mm (unloaded) and s = −143 mm (100 g) is provided for both cases, with and without load.

Figure 4:

Actuator bending shape at 0 s, 4.5 s and 12 s. A visible deflection of the distal tip over an actuation period of 0 s–12 s can be observed.

2.3 Training data acquisition

Load cases were defined for data acquisition in order to subsequently fit a polynomial model. In a load case, a specific weight and heating period are defined. Weights were applied in the range from 0 g to 100 g in 10 g steps. Preliminary tests revealed a maximum possible heating period of 12 s at 3.6 V, avoiding over-heating and damage to the setup. The heating period was set accordingly.

The experimental procedure of an actuation cycle started for each case with the loading of the actuator at ambient temperature T _A (i.e. in martensite state). This leads to a deflection of the distal tip from the initial horizontal position. Then, a voltage of 3.6 V was applied by a laboratory power supply to re-obtain the horizontal straight shape (see Figure 4). Data were recorded for 240 s from the start of heating. This procedure was repeated twice without resetting the actuator load-free to the horizontal shape. The overall training data thus included 22 experimental training cycles, namely 2 trials for each of the 11 weights. The typical hysteresis of deflection s and resistance R of an example actuation cycle is shown in Figure 5. Maximum temperatures on the actuator surface remained < 80 ◦ C during our investigations (see Figure 6).

Figure 5:

Measured resistance R and deflection s for no load of a full actuation cycle of the actuator including heating (red) and cooling phase (blue). Colored arrows indicate the chronicle order of Sections 1–4, subdividing the actuation cycle into four polynomial submodels, based on whether the resistance R increases or decreases. Semi-transparent circles denote the transitions between sections.

Figure 6:

Actuator surface temperature during an actuation cycle. Heating period was set to 12 s, followed by a (passive) cooling phase for the remaining 228 s of data recording. Every trial started at T _W < 27 °C.

2.4 Polynomial modeling

A polynomial approach was chosen for the shape estimation since it allows in a simple manner to represent a nonlinear relationship of the measurement in several dimensions. Our model was developed in MATLAB 2021b and should estimate the vertical deflection of the actuator tip s ̂ based on the measured input parameters of the resistance R, the vertical force at the clamping F _z , and the heating status, schematically shown in Figure 7. The force data is required in addition to the resistance to cover medical use cases of arbitrary unknown interaction forces between instrument and tissue. Only F _z was considered as preliminary tests revealed it as the dominant force component for the investigated load cases. The general representation of the two-dimensional-polynomials can be written as

where i and j indicate the power of R and F _z with the coefficient p _ij . M _R and N _F indicate the highest degree of the polynomial in the dimension R and F _z , respectively.

Figure 7:

Schematic of the polynomial model with the input variables of resistance R, force F _z and heat status. The output is a deflection estimate s ̂ .

The polynomials were fitted to the data with the rmse function minimizing the root-mean-square error Q defined as

where s ̂ denotes the estimated value, s the actual measured value, and N _data is the number of measurement points during data acquisition. Since the relationship between R and s is characterized by a hysteresis depending on heating and cooling phases, the actuation cycle is subdivided. In contrast to Lan et al. [13], the hysteresis was not reduced to one single polynomial. Instead, four separate polynomials were fitted for the four different sections, thus covering the hysteresis entirely (see Figure 5). Each section can be distinguished based on heating status (i.e. heating, cooling) and the resistance R (i.e. increasing, decreasing) and thus be modeled by a separate polynomial. For real time estimation, the model compares the actual measured resistance value to the last measured values. Based on the heating status and the behavior of the electrical resistance, the model estimates the deflection within the corresponding section. In Table 1 the computed coefficients p _ij are listed.

2.5 Model evaluation

For the purpose of model evaluation, further experiments were conducted to test the model for three simplified application scenarios, as shown in Figure 8. Those include the actuation against no resistance i.e. 0 g load (Figure 8a), small resistance i.e. 20 g load (Figure 8b), and a blocking resistance occurring at 50 % of maximum deflection (Figure 8c), which corresponds to s ≈ −116 mm.

Figure 8:

Load-heat scenarios for model evaluation with no load (a), 20 g (b) weight, and blocked at 50 % (c).

In addition, the heating period for each scenario was varied. Besides the maximum heating period of 12 s representing a full actuation cycle, also shorter heating periods must be considered. In medical use cases, flexible instruments might be actuated to transform into intermediate shapes between initial and maximum deflection. Therefore, the heating period of 4.5 s was also tested. In a third variation, we simulated a repetitive readjustment of the shape with a short heating period of 4.5 s, a 7.5 s cooling break, followed by another 6 s long heating period. Thus, for the evaluation, 9 different load-heat scenarios were conducted with 2 experimental trials, each.

Figure 9 presents the quantitative evaluation procedure schematically. The estimated deflection s ̂ of the wire was subsequently calculated with the polynomial model using only the measured values of resistance R, the vertical force F _z , and the heat status as inputs. These estimated values were then compared to the recorded s _eval data. In Figure 10 the fitted 2-dimensional polynomials are shown. The experimental training cycles for fitting the model include electrical resistance R values from 0.62 Ω to 0.7 Ω, loads from 0 g to 100 g and measured deflection values s from −143 mm to −92 mm. Deflection estimations out of these ranges are not valid due to extrapolation errors with high-degree polynomials. Additionally, a comparison of data points of one evaluation trial without load is plotted.

Figure 9:

Schematic of the evaluation of the polynomial model. Measured deflection data s _eval is collected for 9 scenario-heating cases and compared to the estimated deflection of the model s ̂ to find the estimation error e.

Figure 10:

Two-dimensional polynomial model for Sections 1–4 correlating resistance R and force F _z in order to find the deflection s. Comparing the model’s estimation s ̂ and measured deflection s _eval for 12 s heating time and without load.

The mean estimation error e ̄ k for all data points k in one scenario is defined as

where the maximum error is e _max = max(e _k ). The mean estimation error e ̄ about all evaluation trials is