Three-Dimensional Surgical Needle Localization and Tracking Using Stereo Endoscopic Image Streams

I. Introduction

Minimally Invasive Surgery (MIS) has demonstrated significant benefits including reduced patient cost, length of hospital stay, recovery time, and postoperative pain [1]. The main goal of MIS is to duplicate open surgical techniques without large incisions in order to minimize complications. Advanced robotic systems such as the da Vinci^® robotic surgical system (Intuitive Surgical, Inc., Sunnyvale, CA) allow surgeons to perform MIS using teleoperated robotic manipulators instead of manual surgical tools. Surgeons are provided with a console equipped with two master controllers that can manipulate robotic arms. The vision system provides a high-definition three-dimensional (3D) image for surgeons to see the surgical procedure in real-time, while the teleoperation system monitors the surgeon’s hand movements and mimics with the robotic grippers with high precision.

Using such a system, surgeons are able to perform intricate surgeries with high precision and dexterity. Robotic minimally invasive surgery (RMIS) has been widely used in a variety of procedures[2], [3], including, prostatectomy [3], cardiac surgery [4] and thyroidectomy [5].

However, due to the nature of master-slave teleoperation, RMIS usually results in longer operation times and the learning curve is steeper than other MIS techniques [6]. In addition, the more limited robotic workspace and narrow laparoscopic camera view can make low-level tasks more challenging and time consuming. To enhance surgeon performance and reduce the operation time, autonomous robotic surgical assistants [7] have been proposed to perform low-level surgical manipulation tasks such as suturing [8], [9], [10], debridement [11], dissection, and retraction [12].

Suturing is ideally suited to this form of automation due to its repetitive nature. To reduce the tissue trauma and operation time, an automation framework can keep the surgeon as the decision maker while relying on the robotic system to manage the execution of low-level motions.

The dynamic nature of surgical environments and the underlying substantial uncertainty necessitates the surgical manipulations to be performed under sensory guidance (as opposed to the use of primarily pre-planned manipulation strategies employed in traditional industrial robotics applications). As such, development of methods for perceiving the state of the surgical environment and the robotic system is a key requirement for autonomous and semi-autonomous execution of surgical manipulation tasks. Once such robust robotic perception algorithms are available, they can be used to perform precise visually guided manipulations by applying visual servo-control strategies, allow the robotic surgical system to operate under imperfect and varying robotic manipulator/camera calibration conditions, and to deal with the uncertainties resulting from unknown initial conditions and complex tissue deformation dynamics.

This study specifically focuses on the three-dimensional tracking of surgical needles from stereo endoscopic video image streams in RMIS. Localization and tracking of surgical needles is a key enabling technology for autonomous robotic execution of surgical suturing, since errors in needle trajectories resulting from incorrect needle grasps, needle slips, and robot/camera miscalibration may lead to task failures. For example, imperfect needle grasps may lead to increases in task completion time due to need for subsequent (possibly multiple) needle regrasps [13], whereas imperfect needle trajectories during a needle drive may cause increased tissue damage [14] or even result in complete failure of the suture. Knowledge of the pose of the surgical needle throughout the task would allow the system to deal with uncertainties in initial needle grasp as well as those resulting from unmodeled changes in the needle grasp configuration due to needle-tissue interaction forces.

We propose a vision-based probabilistic (Bayesian) state-estimation method for localizing and tracking the surgical needle. For state evolution, the proposed method employs the forward kinematics of the robotic surgical gripper manipulating the surgical needle whenever the needle is being directly grasped, and uses a Brownian motion model whenever the needle is not being directly grasped. Video images from stereo endoscopes provide the sensory feedback used in the measurement updates of the state estimation. A particle filter is used as the Bayesian estimator, as the underlying system is neither linear nor Gaussian. The proposed needle tracking algorithm is quantitatively validated by evaluating the needle tracking accuracy under different noise and occlusion conditions, using a ROS/Gazebo-based simulation of the da Vinci^® surgical robotic system. The performance of the proposed algorithm is also validated in hardware experiments on an actual da Vinci^® surgical robotic system for tracking a surgical needle and grasping it autonomously.

The rest of the paper is organized as follows. Section II discusses the related studies on surgical needle tracking. In Section III, the problem formulation and the proposed methods are introduced. The specific details of simulation and hardware based validation tests and the results of the needle tracking algorithm are presented in Section IV. The conclusions are presented in Section V.

II. Related Studies

The success of the autonomous suturing task is highly dependent on tracking task-critical elements such as the surgical needle, suture thread, and tissue. In order to complete the suturing task autonomously, the surgical needle needs to be localized and tracked during the execution of the task. Therefore, researchers studying the autonomous suturing task tried to solve this problem using various methods.

Nageotte et al. [15] proposed a circular needle and needle-holder localization algorithm for computer-assisted suturing in laparoscopic surgery. They modeled the needle-holder as a cylinder and attached passive visual markers to track its location. The surgical needle was colored to simplify its segmentation, and an ellipse-fitting algorithm was applied on the segmented image to calculate the 3D pose of the needle.

Sen et al. [16] developed an automated multi-throw surgical suturing algorithm using the da Vinci^® surgical system. Instead of the da Vinci^® endoscope, a custom stereo camera pair was employed to provide a larger workspace, and a larger-than-normal, yellow-painted surgical needle was used. In order to segment the image, the needle’s distinctive shape and color were leveraged to assist HSV (Hue, Saturation, Value) segmentation to identify it. Once a set of image points are extracted from the segmented image, the 3D pose of the needle was calculated using an ellipse fitting algorithm.

Iyer et al. [17] attempted to solve the suturing problem by using a single-arm robotic manipulator with a standard laparoscopic needle holder, curved surgical needle and a clinical endoscope (single camera). In order to track the surgical needle, the obtained image was segmented using OpenCV thinning and smoothing functions. A monocular pose measurement method was then used along with the least-square ellipse fitting OpenCV functions to estimate the position and orientation of the semi-circular needle.

Kurose et al. [18] studied needle tracking for a micro-surgery robotic system. They generated an off-line data set of 216,000 needle contour models using a 3D CAD model. During tracking, the image of the operation area was obtained from the microscope and segmented using the Canny edge detection algorithm and needle color information. Then the segmented needle image was searched against the database of the needle contour models to find the best match.

The earlier studies in the literature on vision-based localization and tracking of surgical needles all rely on simplifying assumptions such as artificially colored needles, unchanged lighting conditions, a pregrasped needle, no tissue deformations, or custom built camera systems, none of which are applicable to practical RMIS scenarios. In contrast, the present study aims to perform needle tracking using images from the endoscopic stereo cameras of a realistic RMIS system without any modification to the robotic tools, endoscopes, or the surgical needles. Additionally, the goal is to achieve tracking which is robust to occlusions of the needle. Specifically, the availability of the state evolution model as part of the proposed Bayesian state-estimation scheme would allow the system to maintain tracking and to gracefully recover from periods of needle occlusion. To the best of authors’ knowledge, there are no earlier published studies where detailed experiments were performed to test the repeatably and robustness of the tracking algorithm under occlusion, which is an inevitable scenario during RMIS.

III. Particle Filter Algorithm

Bayesian filtering algorithms enable estimation of a system’s belief state under system and environmental uncertainties. In these algorithms, the belief (bel(x_t)) at time t is recursively calculated from the belief (bel(x_t−1)) at time t-1 using measurement and control data as well as probabilistic models of the system dynamics and the measurement error. Bayesian state estimation methods have been used extensively in the robotics literature, and shown to be very suitable to solve real-time tracking problems [19], [20], [21].

We employ a particle filter as our underlying Bayesian filtering algorithm. The particle filter algorithm, also known as the sequential Monte Carlo estimation algorithm [22], [23], is one of the most popular Bayesian filtering techniques for nonlinear and non-Gaussian problems. The core idea of the particle filter algorithm is to approximate the posterior probability distribution of the state, such as the surgical needle pose, by using a finite number of randomly generated samples, called particles. Particle filters have been widely used in robotics and computer vision applications [22], including pose estimation on the SE(3) group [24]. The particle filter algorithm can be divided into two main steps: state prediction and state update. In the prediction step, the particles are propagated using the system model. In the update step, the weight of each particle is measured based on the observation model. The overall flow diagram can be seen on Fig. 1, and the summary of the particle filter algorithm is given in Algorithm 1.

There are different approaches for generating the initial set of particles, such as randomly sampling the entire state space if there is no a priori information or generating samples in the region where the target is expected to be if there is a priori information. In this study, there are two cases: the needle is either initially held by the robotic hand or it is freely placed in the surgical scene. If the needle is grasped by the robot, the forward kinematics of the robot arm are used to generate the initial set of samples around the expected needle pose. If the needle is free, then it is assumed that the rough pose of the needle is known, and the initial set particles are randomly generated in this region.¹

The first step in the algorithm (Algorithm 1, Line 5) is to update the particle states using the motion model. In this study, there are two different cases for the update step. If the needle is free, i.e., not being held by the robot, then the motion of the needle is modeled as Brownian motion, where the state of each particle is perturbed at each time step with Gaussian noise. This model would allow the particle filter to follow a (slowly) moving needle, as well as to accommodate uncertainties in the initialized needle pose. If the needle is grasped by the robot, then each particle state is updated using the incremental motion of the robotic gripper holding the needle. The incremental motion of the gripper can be estimated using the forward kinematics or the Jacobian of the manipulator, with added Gaussian noise to account for uncertainties in robot motion as well as errors in robot calibration.

The next step in the algorithm is the measurement update (Algorithm 1, Line 6). In the particle filter, each particle represents a hypothesis for the position and orientation of the needle. As part of the measurement update, the observation likelihoods for each of these hypotheses need to be evaluated. The observation likelihoods quantify how well each of the needle pose hypotheses match the observed images of the surgical scene captured by the endoscopic stereo vision system. In our proposed approach, the images acquired by the endoscope are segmented using a thin feature extraction algorithm in order to emphasize the needle outline (and other thin features in the scene). The observation likelihoods are then estimated from the image space similarity between the virtual images of the needle geometry generated from the needle pose hypotheses and the observed segmented images of the scene, as calculated using the normalized cross-correlation between them.

The final step of the algorithm is the resampling step (Algorithm 1, Line 9). In our proposed approach, the low variance resampling method [25] is employed.

D. Motion Model

When tracking a surgical needle, the motion of the real object needs to be modeled. In this study, the motion model is separated into two cases: one where the needle is free and one where the needle is held by the manipulator. If the needle is free, then there is no motion to be observed from the manipulator sensors. In this case, the motion of the needle is modeled as Brownian motion, perturbing the particles with a small amount of Gaussian noise at every iteration. This perturbation motion spreads the particles to avoid false convergence and enables a search mechanism. Once the free needle is localized, it can be grabbed by the manipulator at its now known configuration.

In the second case, the needle needs to be tracked while being held and moved by a manipulator. The coordinate transformations that will be used for the derivation of the motion model for this case are shown in Fig. 5.²

(6)

where $V_{CN}^b$ is the body velocity of the needle in terms of the camera frame, $V_{SN}^b$ is the body velocity of the needle in terms of the manipulator frame and $V_{CS}^b$ is the body velocity of the manipulator in terms of the camera frame. Since the transformation between the camera and the manipulator is fixed, the body velocity of the camera in terms of the manipulator is $V_{CS}^b=0$. Then, the body velocity of the needle, in terms of the camera frame, is the same as the body velocity of the needle, in terms of the manipulator frame:

$$V_{CN}^b=V_{SN}^b.$$

(7)

Fig. 5.

Open in a new tab

Using the robot kinematics, the body velocity of the needle, in terms of the manipulator frame, can be calculated as

(8)

where $V_{SN}^b$ and $V_{ST}^b$ are respectively the body velocities of the needle and manipulator tip, in terms of the manipulator frame, and $V_{TN}^b$ is the body velocity of the needle, in terms of the manipulator tip frame. Here, the needle is grasped by the manipulator with the configuration of g_TN. This configuration is assumed to be known and to remain constant during the motion. Therefore, the body velocity of the needle, in terms of the manipulator tip frame, is $V_{TN}^b=0$.

The body velocity of the tool tip in terms of the manipulator frame is computed using the manipulator Jacobian as

$$V_{ST}^b=J_{ST}^b{\dot{\theta }}_{\text{robot }},$$

(9)

where $J_{ST}^b$ is the robot’s body Jacobian and ${\dot{\theta }}_{\text{robot }}$ is a vector containing the sensor readings of the robot joints. Using (7), the body velocity of the needle in terms of the camera frame can be written as:

$$V_{CN}^b=A{d}_{g_{TN}^{-1}}{J}_{ST}^b{\dot{\theta }}_{\text{robot }}.$$

(10)

In order to find the displacement of the real needle in space, the spatial velocity of the needle, in terms of the camera frame, $V_{CN}^s$, is computed from the body velocity of the needle in terms of the camera frame, $V_{CN}^b$:

$$V_{CN}^S=A{d}_{g_{CN}}{V}_{CN}^b,$$

(11)

where g_CN is the transformation between the needle and the camera. Finally, each particle is updated as:

$$g_{CN}(t+\Delta t)=e^{{\widehat{\tilde{V}}}_{CN}^s\Delta t}{g}_{CN}(t),$$

(12)

where $e^{{\widehat{\tilde{V}}}_{CN}^s\Delta t}$ is the needle displacement predicted by the motion model, and

$${\tilde{V}}_{CN}^s=V_{CN}^s+\mathcal{N}(\Sigma )$$

(13)

is the spatial velocity of the needle, in terms of the camera frame, with additive Gaussian noise $\mathcal{N}(\Sigma )$ to account for robot motion uncertainties.

IV. Validation Results

The particle filter algorithm for the surgical needle localization and tracking was evaluated both in a simulation environment and on the real robot system. In the simulation and hardware tests, a 26 mm diameter semi-circular surgical needle is being localized and tracked by the proposed algorithm using images form the stereo endoscopic cameras of the da Vinci^® surgical robotic system.

V. Conclusions

This study presents a particle filter-based framework for tracking surgical needles using stereo endoscopic images. A virtual needle rendering procedure is implemented to produce candidate needle models in the image space. Using the virtually rendered tool model, an image matching method is applied to form a measurement model for the particle filter algorithm. The performance of the tracking is quantitatively validated in a simulation environment and experimentally evaluated using the physical da Vinci^® surgical robotic system. The validation results indicate that the proposed algorithm can successfully localize the needle starting from moderate levels of initialization uncertainty, and successfully track the needle while it is being manipulated by a robotic gripper. The algorithm was also able to maintain tracking in complex scenes, and under intermittent occlusions of the needle.

Future work will proceed in several directions. The proposed method uses the segmentation method implemented on a GPU-based parallel computing scheme. However, other components of the algorithm are implemented as a serial algorithm on a CPU. Although the tracking algorithm runs in real-time, the frame rate (3 frames per second for 250 particles) is currently insufficient for closed-loop visual servo control. The most time consuming component of the tracking algorithm is the virtual tool rendering part since the needle frame points projected into the image space several times for each particle. As a part of the future work, we are working on a GPU-based parallel implementation of the particle filter algorithm to increase the speed of the algorithm up to a sufficient frame rate for a closed-loop visual servo control.