Abstract
Augmented Reality (AR) in monocular liver laparoscopy requires one to register a preoperative 3D liver model to a laparoscopy image. This is a difficult problem because the preoperative shape may significantly differ from the unknown intraoperative shape and the liver is only partially visible in the laparoscopy image. Previous approaches are either manual, using a rigid model, or automatic, using visual cues and a biomechanical model. We propose a new approach called the hybrid approach combining the best of both worlds. The visual cues allow us to capture the machine perception while user interaction allows us to take advantage of the surgeon’s prior knowledge and spatial understanding of the patient anatomy. The registration accuracy and repeatability were evaluated on phantom, animal ex vivo and patient data respectively. The proposed registration outperforms the state of the art methods both in terms of accuracy and repeatability. An average registration error below the 1 cm oncologic margin advised in the literature for tumour resection in laparoscopy hepatectomy was obtained.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adagolodjo, Y., R. Trivisonne, N. Haouchine, S. Cotin, and H. Courtecuisse. Silhouette-based pose estimation for deformable organs application to surgical augmented reality. In: IROS, 2017
Adams, F., T. Qiu, A. Mark, B. Fritz, L. Kramer, D. Schlager, U. Wetterauer, A. Miernik, and P. Fischer. Soft 3D-printed phantom of the human kidney with collecting system. Ann. Biomed. Eng. 45:963–972, 2017.
AgiSoft PhotoScan Professional (Version 1.2.6) (Software), 2016
Barr, A. H. Global and local deformations of solid primitives. Comput. Graph. 18(3):21–30, 1984.
Bechmann, D., and D. Gerber. Arbitrary shaped deformations with dogme. Vis. Comput. 19:175–186, 2003.
Bender, J., D. Koschier, P. Charrier, and D. Weber. Position-based simulation of continuous materials. Comput. Graph. 44:1–10, 2014.
Bernhardt, S., S. Nicolau, A. Bartoli, V. Agnus, L. Soler, and C. Doignon. Using shading to register an intraoperative CT scan to a laparoscopic image. Computer-Assisted Robotic Endoscopy. CARE, 2015
Clements, L., J. Collins, J. Weis, A. Simpson, T. Kingham, W. Jarnagin, and M. Miga. Deformation correction for image guided liver surgery: an intraoperative fidelity assessment. Surgery 162(3):537–547, 2017.
Feng, J., J. Shao, X. Jin, Q. Peng, and R. Forrest. Multiresolution free-form deformations with subdivision surfaces of arbitrary topology. Vis. Comput. 22(1):28–42, 2005.
Haouchine, N., F. Roy, L. Untereiner, and S. Cotin. Using contours as boundary conditions for elastic registration during minimally invasive hepatic surgery. In: IROS, 2016.
Koo, B., E. Ozgur, B. Le Roy, E. Buc, and A. Bartoli Deformable registration of a preoperative 3D liver volume to a laparoscopy image using contour and shading cues. In: MICCAI, 2017
Lamata, P., F. Lamata, V. Sojar, P. Makowski, L. Massoptier, S. Casciaro, W. Ali, T. Studeli, J. Declerck, O. Elle, and B. Edwin. Use of the Resection Map system as guidance during hepatectomy. Surg. Endosc. 24:2327–2337, 2010.
Margonis, G., T. Sergentanis, I. Ntanasis-Stathopoulos, N. Andreatos, I. Tzanninis, K. Sasaki, T. Psaltopoulou, J. Wang, S. Buettner, A. Papalois, J. He, C. Wolfgang, T. Pawlik, and M. Weiss. Impact of surgical margin width on recurrence and overall survival following R0 hepatic resection of colorectal metastases: a systematic review and meta-analysis. Ann. Surg. 267(6):1047–1055, 2018.
Moghari, M., B. Ma, and P. Abolmaesumi. A theoretical comparison of different target registration error estimators. In: MICCAI, 2008.
Nicolau, S., L. Soler, D. Mutter, and J. Marescaux. Augmented reality in laparoscopic surgical oncology. Surg. Oncol. 20(3):189–201, 2011.
Oktay, O., L. Zhang, T. Mansi, P. Mountney, P. Mewes, S. Nicolau, L. Soler, and C. Chefd'hotel. Biomechanically driven registration of pre- to intra-operative 3d images for laparoscopic surgery. In: MICCAI, 2013
Ozgur, E., B. Koo, B. Le Roy, E. Buc, and A. Bartoli. Preoperative liver registration for augmented monocular laparoscopy using backward-forward biomechanical simulation. Int. J. Comput. Assist. Radiol. Surg. 13:1629–1640, 2018.
Robu, M. R., J. Ramalhinho, S. Thompson, K. Gurusamy, B. Davidson, D. Hawkes, D. Stoyanov, and M. Clarkson. Global rigid registration of CT to video in laparoscopic liver surgery. Int. J. Comput. Assist. Radiol. Surg. 13:947–956, 2018.
Sacht, L., E. Vouga, and A. Jacobson. Nested cages. ACM Trans. Graph. 34(6) Article No. 170, 2015.
Sederberg, T. W., and S. R. Parry. Free-form deformation of solid geometric models. Comput. Graph. 20(4):151–160, 1986.
Thompson, S., J. Totz, Y. Song, S. Johnsen, D. Stoyanov, S. Ourselin, K. Gurusamy, C. Schneider, B. Davidson, D. Hawkes, and M. Clarkson. Accuracy validation of an image guided laparoscopy system for liver resection. Proc. SPIE 9415(09):1–12, 2015.
Zhong, F. P., Y. J. Zhang, Y. Liu, and S. B. Zou. Prognostic impact of surgical margin in patients with hepatocellular carcinoma: a meta-analysis. Medicine 96(37):e8043, 2017.
Conflict of interest
The authors of this article declare no potential conflicts of interest.
Ethical Approval
All procedures involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This study is also supported by an ethical approval with ID IRB00008526-2019-CE58 issued by CPP Sud-Est VI in Clermont-Ferrand, France.
Informed Consent
Informed consent was obtained from the patients included in the study.
Author information
Authors and Affiliations
Corresponding author
Additional information
Associate Editor Elena S. Di Martino oversaw the review of this article.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Espinel, Y., Özgür, E., Calvet, L. et al. Combining Visual Cues with Interactions for 3D–2D Registration in Liver Laparoscopy. Ann Biomed Eng 48, 1712–1727 (2020). https://doi.org/10.1007/s10439-020-02479-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10439-020-02479-z