Stereoscopic calibration for augmented reality visualization in microscopic surgery | International Journal of Computer Assisted Radiology and Surgery Skip to main content
Springer Nature Link
Log in

Stereoscopic calibration for augmented reality visualization in microscopic surgery

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Purpose

Middle and inner ear procedures target hearing loss, infections, and tumors of the temporal bone and lateral skull base. Despite the advances in surgical techniques, these procedures remain challenging due to limited haptic and visual feedback. Augmented reality (AR) may improve operative safety by allowing the 3D visualization of anatomical structures from preoperative computed tomography (CT) scans on real intraoperative microscope video feed. The purpose of this work was to develop a real-time CT-augmented stereo microscope system using camera calibration and electromagnetic (EM) tracking.

Methods

A 3D printed and electromagnetically tracked calibration board was used to compute the intrinsic and extrinsic parameters of the surgical stereo microscope. These parameters were used to establish a transformation between the EM tracker coordinate system and the stereo microscope image space such that any tracked 3D point can be projected onto the left and right images of the microscope video stream. This allowed the augmentation of the microscope feed of a 3D printed temporal bone with its corresponding CT-derived virtual model. Finally, the calibration board was also used for evaluating the accuracy of the calibration.

Results

We evaluated the accuracy of the system by calculating the registration error (RE) in 2D and 3D in a microsurgical laboratory setting. Our calibration workflow achieved a RE of 0.11 ± 0.06 mm in 2D and 0.98 ± 0.13 mm in 3D. In addition, we overlaid a 3D CT model on the microscope feed of a 3D resin printed model of a segmented temporal bone. The system exhibited small latency and good registration accuracy.

Conclusion

We present the calibration of an electromagnetically tracked surgical stereo microscope for augmented reality visualization. The calibration method achieved accuracy within a range suitable for otologic procedures. The AR process introduces enhanced visualization of the surgical field while allowing depth perception.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Availability of data and materials

Not applicable.

References

  1. Li L, Yang J, Chu Y, Wu W, Xue J, Liang P, Chen L (2016) A novel augmented reality navigation system for endoscopic sinus and skull base surgery: a feasibility study. PLoS One 11(1):e0146,996

  2. Ridge SE, Shetty KR, Lee DJ (2021) Heads-up surgery: Endoscopes and exoscopes for otology and neurotology in the era of the covid-19 pandemic. Otolaryngol Clin North Am 54(1):11–23

    Article  PubMed  Google Scholar 

  3. Storz P, Buess GF, Kunert W, Kirschniak A (2012) 3d hd versus 2d hd: surgical task efficiency in standardised phantom tasks. Surg Endosc 26(5):1454–1460

    Article  PubMed  Google Scholar 

  4. Smith R, Day A, Rockall T, Ballard K, Bailey M, Jourdan I (2012) Advanced stereoscopic projection technology significantly improves novice performance of minimally invasive surgical skills. Surg Endosc 26(6):1522–1527

    Article  CAS  PubMed  Google Scholar 

  5. Roberts DW, Strohbehn JW, Hatch JF, Murray W, Kettenberger H (1986) A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. J Neurosurg 65(4):545–549

    Article  CAS  PubMed  Google Scholar 

  6. Edwards P, Hawkes D, Hill D, Jewell D, Spink R, Strong A, Gleeson M (1995) Augmentation of reality using an operating microscope for otolaryngology and neurosurgical guidance. J Image Guid Surg 1(3):172–178

    Article  CAS  PubMed  Google Scholar 

  7. Hussain R, Lalande A, Berihu Girum K, Guigou C, Grayeli AB (2020) Augmented reality for inner ear procedures: visualization of the cochlear central axis in microscopic videos. Int J Comput Assist Radiol Surg 15(10):1703–1711

    Article  PubMed  Google Scholar 

  8. Marroquin R, Lalande A, Hussain R, Guigou C, Grayeli AB (2018) Augmented reality of the middle ear combining otoendoscopy and temporal bone computed tomography. Otology & Neurotology 39(8):931–939

    Article  Google Scholar 

  9. Citardi MJ, Agbetoba A, Bigcas JL, Luong A (2016) Augmented reality for endoscopic sinus surgery with surgical navigation: a cadaver study. In: International forum of allergy & rhinology, Wiley Online Library, pp 523–528

  10. Chu Y, Yang J, Ma S, Ai D, Li W, Song H, Li L, Chen D, Chen L, Wang Y (2017) Registration and fusion quantification of augmented reality based nasal endoscopic surgery. Med Image Anal 42:241–256

    Article  PubMed  Google Scholar 

  11. Bong JH, Song Hj OhY, Park N, Kim H, Park S (2018) Endoscopic navigation system with extended field of view using augmented reality technology. The International Journal of Medical Robotics and Computer Assisted Surgery 14(2):e1886

    Article  Google Scholar 

  12. Winne C, Khan M, Stopp F, Jank E, Keeve E (2011) Overlay visualization in endoscopic ent surgery. Int J Comput Assist Radiol Surg 6(3):401–406

    Article  PubMed  Google Scholar 

  13. Thoranaghatte R, Garcia J, Caversaccio M, Widmer D, Gonzalez Ballester MA, Nolte LP, Zheng G (2009) Landmark-based augmented reality system for paranasal and transnasal endoscopic surgeries. The International Journal of Medical Robotics and Computer Assisted Surgery 5(4):415–422

    Article  PubMed  Google Scholar 

  14. Caversaccio M, Thoranaghatte R, Zheng G, Eggli P, Nolte L, Ballester G et al (2008) Augmented reality endoscopic system (ares): preliminary results. Rhinology 46(2):156–158

    PubMed  Google Scholar 

  15. Caversaccio M, Garcia-Giraldez J, Gonzalez-Ballester M, Marti G (2007) Image-guided surgical microscope with mounted minitracker. The Journal of Laryngology & Otology 121(2):160–162

    Article  CAS  Google Scholar 

  16. Liu WP, Richmon JD, Sorger JM, Azizian M, Taylor RH (2015) Augmented reality and cone beam ct guidance for transoral robotic surgery. J Robot Surg 9(3):223–233

    Article  PubMed  PubMed Central  Google Scholar 

  17. Liu WP, Azizian M, Sorger J, Taylor RH, Reilly BK, Cleary K, Preciado D (2014) Cadaveric feasibility study of da vinci si-assisted cochlear implant with augmented visual navigation for otologic surgery. JAMA Otolaryngology-Head & Neck Surgery 140(3):208–214

    Article  Google Scholar 

  18. Shiu YC, Ahmad S (1987) Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form ax= xb. IEEE Transactions on Robotics and Automation

  19. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  20. Heikkila J, Silvén O (1997) A four-step camera calibration procedure with implicit image correction. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, IEEE, pp 1106–1112

  21. Marchand E, Uchiyama H, Spindler F (2015) Pose estimation for augmented reality: a hands-on survey. IEEE Trans Visual Comput Graphics 22(12):2633–2651

    Article  Google Scholar 

  22. Geiger A, Moosmann F, Car Ö, Schuster B (2012) Automatic camera and range sensor calibration using a single shot. In: 2012 IEEE international conference on robotics and automation, IEEE, pp 3936–3943

  23. Liu X, Plishker W, Zaki G, Kang S, Kane TD, Shekhar R (2016) On-demand calibration and evaluation for electromagnetically tracked laparoscope in augmented reality visualization. Int J Comput Assist Radiol Surg 11(6):1163–1171

    Article  PubMed  PubMed Central  Google Scholar 

  24. Neves C, Tran E, Kessler I, Blevins N (2021) Fully automated preoperative segmentation of temporal bone structures from clinical ct scans. Sci Rep 11(1):1–11

    Article  Google Scholar 

  25. Rose AS, Kimbell JS, Webster CE, Harrysson OL, Formeister EJ, Buchman CA (2015) Multi-material 3d models for temporal bone surgical simulation. Annals of Otology, Rhinology & Laryngology 124(7):528–536

    Article  Google Scholar 

  26. Mowry SE, Jammal H, Myer C IV, Solares CA, Weinberger P (2015) A novel temporal bone simulation model using 3d printing techniques. Otology & Neurotology 36(9):1562–1565

    Article  Google Scholar 

  27. Hochman JB, Kraut J, Kazmerik K, Unger BJ (2014) Generation of a 3d printed temporal bone model with internal fidelity and validation of the mechanical construct. Otolaryngology-Head and Neck Surgery 150(3):448–454

    Article  PubMed  Google Scholar 

  28. Rose AS, Webster CE, Harrysson OL, Formeister EJ, Rawal RB, Iseli CE (2015) Pre-operative simulation of pediatric mastoid surgery with 3d-printed temporal bone models. Int J Pediatr Otorhinolaryngol 79(5):740–744

    Article  PubMed  Google Scholar 

  29. Mowry SE, Jabbour N, Rose AS, Wiet GJ, Svrakic M, Zopf DA, Vankoevering K, Powell A, Freiser ME, Hochman J et al (2021) Multi-institutional comparison of temporal bone models: a collaboration of the aao-hnsf 3d-printed temporal bone working group. Otolaryngology-Head and Neck Surgery 164(5):1077–1084

    Article  PubMed  Google Scholar 

  30. Citardi MJ, Yao W, Luong A (2017) Next-generation surgical navigation systems in sinus and skull base surgery. Otolaryngol Clin North Am 50(3):617–632

    Article  PubMed  Google Scholar 

  31. Liu X, Plishker W, Shekhar R (2021) Hybrid electromagnetic-aruco tracking of laparoscopic ultrasound transducer in laparoscopic video. Journal of Medical Imaging 8(1):015,001

  32. Hu X, Wang G, Wang J, Sun P, Fan J, Chen F (2018) Xie Y (2018) A robust and accurate calibration method for out-of-focus camera. Electronic Imaging 2:263–1

    Google Scholar 

  33. Baba M, Mukunoki M, Asada N (2006) A unified camera calibration using geometry and blur of feature points. In: 18th International Conference on Pattern Recognition (ICPR’06), IEEE, pp 816–819

  34. Ha H, Bok Y, Joo K, Jung J, Kweon IS (2015) Accurate camera calibration robust to defocus using a smartphone. In: Proceedings of the IEEE International conference on computer vision, pp 828–836

  35. Bell T, Xu J, Zhang S (2016) Method for out-of-focus camera calibration. Appl Opt 55(9):2346–2352

    Article  PubMed  Google Scholar 

  36. Wang Y, Chen X, Tao J, Wang K, Ma M (2016) Accurate feature detection for out-of-focus camera calibration. Appl Opt 55(28):7964–7971

    Article  PubMed  Google Scholar 

  37. Liu X, Rice CE, Shekhar R (2017) Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes. Int J Comput Assist Radiol Surg 12:1685–1695

    Article  PubMed  PubMed Central  Google Scholar 

  38. Hermann J, Mueller F, Weber S, Caversaccio M, O’Toole Bom Braga G (2021) In silico assessment of safety and efficacy of screw placement for pediatric image-guided otologic surgery. Frontiers in Surgery 8(736):217

    Google Scholar 

  39. Labadie RF, Shah RJ, Harris SS, Cetinkaya E, Haynes DS, Fenlon MR, Juszczyk AS, Galloway RL, Fitzpatrick JM (2005) In vitro assessment of image-guided otologic surgery: submillimeter accuracy within the region of the temporal bone. Otolaryngology-Head and Neck Surgery 132(3):435–442

Download references

Funding

We gratefully acknowledge the Kaufer Family Fund for its philanthropic support of this project.

Author information

Authors and Affiliations

  1. Department of Bioengineering, Stanford University, Stanford, CA, USA

    Trishia El Chemaly & Brian Hargreaves

  2. Department of Otolaryngology, Stanford School of Medicine, Stanford, CA, USA

    Trishia El Chemaly, Caio Athayde Neves & Nikolas H. Blevins

  3. Department of Radiology, Stanford School of Medicine, Stanford, CA, USA

    Trishia El Chemaly, Christoph Leuze & Brian Hargreaves

  4. Faculty of Medicine, University of Brasília, Brasília, Brazil

    Caio Athayde Neves

  5. Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA

    Christoph Leuze

  6. Department of Electrical Engineering, Stanford University, Stanford, CA, USA

    Brian Hargreaves

Authors
  1. Trishia El Chemaly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caio Athayde Neves
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christoph Leuze
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brian Hargreaves
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nikolas H. Blevins
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Trishia El Chemaly.

Ethics declarations

Conflict of interest

Not applicable.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Code availability

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 53623 KB)

Supplementary file 2 (mp4 61631 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

El Chemaly, T., Athayde Neves, C., Leuze, C. et al. Stereoscopic calibration for augmented reality visualization in microscopic surgery. Int J CARS 18, 2033–2041 (2023). https://doi.org/10.1007/s11548-023-02980-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-023-02980-5

Keywords

Search

Navigation