Machine Learning for Surgical Phase Recognition

The newest form of computer-based automation, where many different types of data are self-captured and self-analyzed, such as self-driving cars, is advancing rapidly in all fields of daily life.¹ However, this huge amount of data can be difficult to interpret. To make this data usable, artificial intelligence (AI) and its subdomains such as machine learning (ML) can be applied. AI is generally defined as the ability of machines or computers to show intelligent behavior, for example, learning and solving tasks for which they were not explicitly programmed.² ML can roughly be described as the automated extraction of knowledge from data³ and is already commonly used in fields such as economics, advertisement, military, and engineering.⁴ Its implementation into healthcare has proven more difficult and has thus lagged behind other disciplines,⁵ though it is rapidly gaining popularity as well.^2,6,7 For example, ML models have recently demonstrated success in identifying cancerous lesions in radiographs⁸ and in identifying diabetic retinopathy,⁹ with the first medical device using ML receiving Food and Drug Administration (FDA) approval in 2018.^10,11 Furthermore, Shademan et al built an autonomous robot capable of suturing an intestinal anastomosis on an animal model that was of equal quality to anastomoses performed by surgeons.¹² Likewise, the number of hits on Medline for the search term “artificial intelligence” or “machine learning” show a steep increase since the introduction of ML in the last decade (Fig. 1).

The technological advancements in surgery, especially minimally invasive and robotic surgery, have led to a growing number of devices used in the operating room. These devices can provide increasingly more information from surgical procedures (eg, surgical video, instrument use, pneumoperitoneal pressure, staff participation, table position, and instrument trajectories).^13,14 As the amount of information has increased, the field of surgical data science subsuming the collection, processing, and analysis of surgical data to help improve patient care, has become increasingly important. Within this field, the automation of data analysis is essential to reduce complexity while maximizing the utility of the data to enable new opportunities. For example, automated intraoperative assistance and cognition guidance for surgeons in real time is a potential future outcome, and automated and improved feedback for trainees.^15–18 Besides the potential to increase surgical quality itself, autonomously analyzing this data also offers the potential to optimize the entire workflow in surgical departments.^19–21 Having organized, annotated data during surgery could enable standardization and objectivity of surgical care including assessment, early detection of errors, and deviation from the normal operative course, potentially resulting in improved patient care.

Automated surgical phase recognition is a cornerstone for the realization of the aforementioned applications of ML in surgery.²² Surgical phases are defined as higher-level tasks that constitute an entire surgical procedure, e.g. dissecting Calots’ triangle to achieve critical view of safety in laparoscopic cholecystectomy (LC). This is illustrated in Figure 2. As surgical literature involving ML expands, surgeons should have an understanding of the types of studies being conducted. Thus, the aim of this systematic review was to summarize the ML models used for automated phase recognition in general surgery.

METHODS

This systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement²³ and Measurement Tool to Assess Systematic Reviews (AMSTAR 2) criteria.²⁴ Additionally, the concept of the review was registered with PROSPERO (CRD42018108907).

Literature Search

A comprehensive literature search was conducted using Medline (via PubMed) and Web of Science, and the IEEEXplore database to account for technical papers. The full PubMed, Web of Science, and IEEExplore search strategies can be found in Supplementary File 3, https://links.lww.com/SLA/C591. To optimize sensitivity of the search, the search was designed according to the Problem Intervention Control Outcome (PICO) criteria.²² Free-text words were combined using Boolean operators, in addition to medical subject headings terms. The search was performed by a professional librarian at Heidelberg University in May 2018 (see acknowledgments, Supplementary File 3, https://links.lww.com/SLA/C591). No language restrictions were applied.

To account for grey literature and further articles, specifically technical papers not published in a medical or IEEE journal, a manual search of google.scholar.com, CiteSeerX, and the references of relevant articles was performed. Finally, experts at the interface of ML and medicine were contacted.

Inclusion and Exclusion Criteria

Inclusion criteria included: a general surgery procedure; the capture of intraoperative signals; and phase recognition based on ML models. Exclusion criteria included: a study not performed on human patients (eg, models or animal experiments) and a surgical procedure from a specialty other than general surgery.

Study Selection

Two reviewers independently screened titles and abstracts for relevance. Disagreement was solved by coming to a consensus or with the help of a third reviewer. In the next step, the same process was performed for eligible full-texts. Data were extracted into a dedicated spreadsheet which was pre-tested on 5 studies to test its suitability.

Outcomes

The primary aim of this systematic review was to summarize the most important ML and computer models used to realize phase recognition in general surgery.

Secondary aims were to present commonly used vocabulary and concepts of AI and ML to the readership. Additionally, types of data that can be used as input for phase recognition are presented. Finally, accuracy rates and limitations of currently implemented methods were summarized.

Bias

Due to the nature of the review, bias analysis according to the Newcastle-Ottawa-scale or the Cochrane tool for assessing risk of bias was not applicable. However, all studies were discussed in the author panel and, if doubt about methodological quality was raised, results were interpreted carefully, and limitations discussed in the review.

RESULTS

The literature search retrieved a total of 2254 studies. After title and abstract screening, the full-texts of 68 studies were analyzed and 35 studies were found eligible for inclusion (Fig. 3).

Figure 4 provides a comprehensive overview of the entire phase recognition process, incorporating data types, ML algorithms, surgical phases, and applications. Although the ML concepts are introduced in general in the manuscript, more detailed descriptions are provided in Supplementary File 1, https://links.lww.com/SLA/C571. Table 1 provides general information about included studies with references, and Supplementary File 2, https://links.lww.com/SLA/C572 includes a detailed summary table of each study included in this review for further reading.

ML Methods and Other Algorithms

All of the ML models presented in this review were based on supervised learning, in which humans manually annotate a dataset with the appropriate labels – in this case, labeling surgical video with the appropriate phases (Fig. 4). ML models are then trained on the annotated surgical data, allowing the models to create general rules that can later be used to make informed inferences about new, unseen data during testing (Fig. 4). Learning is guided by optimization techniques to minimize error. In the case of phase recognition, ML models are performing a classification task: assigning segments of surgical video to a given operative phase, based on its characteristics. Table 2 summarizes the most frequently used algorithms identified in this review, whereas Supplementary File 1, https://links.lww.com/SLA/C571 provides in-depth explanations for each type of model.

Dynamic time warping was used to construct a reference model in 2 studies. These were the first studies to attempt phase recognition for a complete surgery.^25,26 Two other studies used dynamic time warping in combination with another ML model for the purposes of developing an average phase model for a surgery,³⁵ and accounting for temporal information.⁴³ Hidden Markov model (HMM) was one of the most popularly used ML models for phase recognition. One research group from the Technical University of Munich has performed extensive research in using HMM for phase recognition of LC,^28,30,32,35 with other groups performing similar research.^{31,33,34,40,41} An additional 4 studies investigated the use of HMM in combination with another ML model, typically to take temporal information into account and improve accuracy.^38,39,53,54 Several of the papers were able to achieve an error rate of less than 10%. Another 4 studies tested support vector machines (SVM) as the main model for phase prediction. The first study to utilize SVM for phase recognition was published in 2008 by Klank et al.²⁹ Other literature describing SVM was not published until 2015 and later.^38,45,54 One study fed results from their convolutional neural networks model into a SVM for final phase prediction, achieving 92% accuracy.⁵³ Random forests (RF) for surgical phase recognition were first proposed in 2015 and have been less frequently reported for surgical phase recognition in the present review, with 3 studies published in the literature.^36,37,52 Like SVMs, RF can be used in conjunction with neural network and other approaches. Sahu et al fed results from their convolutional neural networks model into 2 RFs for final phase recognition.⁴⁶

Overall, artificial neural networks (ANN) have been the most popularly tested ML type, with 14 published studies included in the present systematic review.^{39,42,43,46–50,53,55–59} ANNs are capable of learning the most important discriminating features simply by receiving information, such as a raw image with phase annotations, as input (feature learning from video). Although ANNs require more annotated data and higher computational power than other approaches, they also have the potential to improve accuracy with increased data inputs and to simplify data acquisition. Interestingly, the first phase recognition paper utilizing an ANN was not published until 2016, with 6 of these studies published in response to the International Conference on Medical Image Computing & Computer Assisted Intervention (MICCAI) 2016 Workflow Challenge.^{39,42,46,48,56,57}

Data Types

To identify the phase of a surgery using ML, at least 1 type of data has to be used as input to an algorithm (Fig. 4A). In this case, an input can be defined as any data that is used to provide information about different time points during a surgery. Table 3 provides explanations of each data type used as input, whereas Supplementary File 1, https://links.lww.com/SLA/C571 can be referenced for further information on data types. The most frequently used data types were instrument use from manual annotation (n = 13) and feature learning from surgical videos (n = 12). Other data types included: instrument use from radio-frequency identification tags (n = 3); intraoperative characteristics (n = 4); instrument use detected from surgical videos (n = 5); and feature extraction from surgical videos (n = 7). Studies were likely to combine multiple data types as input to improve accuracy.

Accuracy and Types of Surgery

LC was the most common surgery that was used to create and test the ML models. There was heterogeneity among the definitions of the different phases of LC. The number of defined phases for LC ranged from 3 to 20. However, there was a consistency for certain key steps such as Calot triangle dissection, clipping, and cutting of cystic duct and artery, and gall bladder dissection and removal (see Table 1 and Fig. 2).

Publicly available datasets that provided annotated phase information of LC for training and testing were utilized by multiple studies (see Table 1). The EndoVis workflow challenge set was used in 4 studies,^{40,41,43,50,53} Cholec80 dataset in 6 studies,^{48,49,53,55,56,58} and the MICCAI 2016 workflow challenge dataset in 6 studies.^{39,42,46,48,56,57}

Other studies focused on bariatric surgery and colorectal surgery. Laparoscopic sleeve gastrectomy was investigated in 2 studies,^51,54 with both achieving a phase recognition accuracy over 90%. The same ML model was compared between LC and colorectal surgery in 1 study,⁵⁰ achieving 74% and 67% accuracy, respectively. However, because colorectal procedures are less standardized and more complex than LC, the authors stated that the lower accuracy for colorectal procedures was an expected outcome.

The reported accuracy for automated phase recognition ranged widely based on a number of factors. The type and design of the ML model greatly affected accuracy. Although many of the studies compared multiple model designs to discover the best-performing model, this can also be demonstrated by looking at studies that used the same dataset but different model designs, such as Cadene et al (93.5% accuracy)³⁹ and Loukas et al (86% accuracy).⁵⁷ The type of data used as input and how it was annotated also showed a large effect on accuracy. For example, Twinanda et al compared the performance of the same model design trained on the MICCAI 2016 and Cholec80 data sets, finding an accuracy of 79.5% and 71.1%, respectively.⁴⁸ In addition, studies reported their accuracy using a variety of metrics, making direct comparison between all of the studies difficult. Although most studies simply reported percent accuracy, others reported percent error, or used more complex measures such as the area under the ROC curve or the Jaccard Score. Explanations of these measuring systems can be found in the Table 1 description. However, it does not seem that accuracy increased over time or with more complex ML methods overall.

DISCUSSION

This review provides an overview of ML for automated surgical phase recognition in general surgery procedures with 35 included studies reporting on various types of ML models and data input. The most commonly used ML models were HMM and ANN, whereas the most frequently used data types were instrument use from manual annotation and feature learning from surgical videos. From the first study on this topic published in 2006 to the most recent studies, the complexity of the ML models and the data types has increased, and improved computer power has contributed in part to the increase in the use of computation-heavy ANNs. Automated surgical phase recognition can currently be performed with high accuracy depending on the granularity of the phases, the ML model, the type of data input, and the complexity of the surgical procedure.

The majority of the studies found in this review used LC as their operation of choice. The fact that publicly available data sets, such as the Cholec80,⁵³ focus on LC is likely a contributing factor. Additionally, as discussed in Klank et al, “cholecystectomy is a common surgery that is performed minimally invasively in 95% of the cases, with a low conversion rate and well-defined phases. It is for this reason often used for phase recognition.”²⁹ Though LC does follow a fairly standardized workflow, it is interesting to mention the disparity in the number of formally identified phases found in this review. Table 1 details that the number of phases described in the studies ranged from 5 to 20. Despite this heterogeneity, a certain consistency for including key steps for LC was apparent. As mentioned by Padoy et al,²⁶ a smaller number of phases should theoretically be easier for a ML model, because each phase has more data to train the algorithm. However, a larger number of phases can be beneficial because increased granularity may be more clinically relevant.⁶⁰

The ubiquity of LC in general surgery practice makes it a useful model to illustrate concepts in phase recognition. For example, the use of specific instruments can improve the identification of surgical phases, such as clipping the cystic duct or artery, as the only action that can be performed with a clip applier is clipping. However, instrumentation can also provide clues on deviation from an expected operative trajectory, such as placing additional clips to stop unexpected bleeding. Such examples are important for establishing the distribution of expected deviations from the procedural “standard” that would still be within normal limits, and this has been explored in the surgical education literature.⁶¹ By establishing a distribution of “normal” in operations, deviations from normal could be utilized to detect abnormalities due to unique operative circumstances, operative errors, or unsafe technique.^2,60 For example, video analysis showing blood in the field and the re-insertion of the clipping device could indicate clinically significant bleeding and an increased risk of intra- or postoperative complications. Data about additional fluid or medications given by anesthesia could further signal or rule out hemodynamically significant bleeding. Therefore, a single data input may not suffice to recognize a deviation from the normal course, but the comprehensive, automated and simultaneous interpretation of multiple inputs in the data set could.

Automated surgical phase recognition is a foundational step for many different applications (see Fig. 4D). In the future, surgical data will likely be annotated and indexed automatically during surgery, allowing surgeons to quickly review certain steps of previous operations while creating informative and focused education material for students and residents. Automated phase recognition is a prerequisite for clinical decision support systems that will provide information to surgeons relative to the actual phase and course of the operation. Such support could be the depiction of target or at-risk structures such as tumor margins, vessels or nerves through augmented reality overlays.^62,63 Additionally, automated phase recognition can improve efficiency in the OR by estimating the remaining surgery duration in real-time and initiating the preparation of the next patient at the most optimal time point. Thus, during the perioperative process, automated phase recognition could provide current information to robotic surgical assistants, provide context-specific information for the surgical phase, improve team work, predict the remaining length of the procedure to optimize operating room management, identify the risk for complications or assess performance.^22,64

Furthermore, as the use of robotics in surgery has increased, more accurate tracking based on instrument kinematics has become possible. Groups led by Hung, Jarc, and Gao have introduced the use of automated performance metrics with ML to assess surgeon's performance, recognize surgical activity, and even anticipate surgical outcome.^64–66 Automated performance metrics also provide a mechanism for robotic systems to potentially learn how to perform surgical tasks and procedures. However, the plausibility of robotic systems operating autonomously, even for limited steps of an operation, remains to be demonstrated.⁶⁷

Despite the increasing number of studies in the field of ML for automated surgical phase recognition, limitations in the field still exist; data annotation, in particular, is a bottleneck in the process. As supervised learning is the dominant methodology, raw data is of little utility without some annotation. In an effort to reduce the amount of manual annotation needed, Funke et al investigated model accuracy when partial annotations were provided to an algorithm, finding that annotating one fourth of the data led to only about a 4% decrease in accuracy when compared with three fourths.⁵⁵ Other methods for improving the annotation process have been suggested as well, such as active learning with Deep Bayesian Networks that can preselect parts of the image data to be manually annotated and then learn iteratively with decreasing amount of manual work needed over time.^68,69 Furthermore, standardization of annotation is necessary to enable comparison of results across studies.

The collection of sufficient amounts of data for research and implementation of surgical phase recognition is still difficult in everyday clinical practice. Integrated ORs, which allow for easier collection and synchronization of various input data and other perioperative information, have still not been widely implemented. Accuracy of ML for surgical phase recognition can still vary widely, and many models require a prohibitive amount of manual annotation and computer processing power. In addition, legal issues exist in accessing and using video and other data from a patient's surgery, with laws varying by country. These challenges have prevented surgical phase recognition from being utilized outside of academic settings to date. The more widespread implementation of integrated ORs, solutions to decrease manual annotation, improved computer performance, and the standardization of data types are prerequisites for the research on ML for surgical phase recognition to translate into clinical practice in the future.

A challenge in the acceptance of ML in the surgical realm is that ML works differently than human intelligence. It is often difficult to interpret the processes happening within a ML model, if this information is available at all. Additionally, the discriminative features as learned by an ANN often differ from those that would be identified by an expert surgeon, though the ML technique may be highly accurate in surgical phase recognition. This may lead to distrust of ML in its ability to successfully perform its assigned tasks. However, as demonstrated in this review, ML is slowly beginning to be welcomed into the field of surgery and recognized as a new technology that can vastly improve healthcare. The acceptance of ML in surgery will depend not only on accuracy but also on transparency, which is termed “Explainable Artificial Intelligence.”⁷⁰ ML for automated surgical phase recognition is a crucial step for many different applications and not an end in itself. Rather, it is the basis for a comprehensive workflow optimization to further improve the quality of healthcare in the future.

CONCLUSIONS

ML for automated surgical phase recognition is a growing research topic with the potential for important innovation in the future. Automated surgical phase recognition identifies different phases and steps of an operation through data inputs such as videos or instrument use. This information then serves to optimize the workflow including intraoperative assistance, automation, surgical training, and patient safety. For operations with well-defined, simple workflows, such as LC, automated phase recognition can be performed with high accuracy, though analysis of more complex surgical procedures remains more challenging. There is a need for the collection of surgical data ready for ML analysis and for standardization of surgical phases in workflow annotation. Given the multiple potential applications of the technology, the use of ML in the field of surgery will likely benefit patients and complement the knowledge and expertise of surgeons to enable higher quality care and improved efficiency.

Supplemental Digital Content

• SLA_2020_08_20_NICKEL_ANNSURG-D-19-02620_SDC3.docx; [Word] (15 KB)
• SLA_2020_08_09_NICKEL_ANNSURG-D-19-02620_SDC1.docx; [Word] (217 KB)
• SLA_2020_08_09_NICKEL_ANNSURG-D-19-02620_SDC2.docx; [Word] (32 KB)