A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation | International Journal of Computer Assisted Radiology and Surgery Skip to main content
Springer Nature Link
Log in

A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Due to the complex anatomical structure of bronchi and the resembling inner surfaces of airway lumina, bronchoscopic examinations require additional 3D navigational information to assist the physicians. A bronchoscopic navigation system provides the position of the endoscope in CT images with augmented anatomical information. To overcome the shortcomings of previous navigation systems, we propose using a technique known as visual simultaneous localization and mapping (SLAM) to improve bronchoscope tracking in navigation systems.

Methods

We propose an improved version of the visual SLAM algorithm and use it to estimate nt-specific bronchoscopic video as input. We improve the tracking procedure by adding more narrow criteria in feature matching to avoid mismatches. For validation, we collected several trials of bronchoscopic videos with a bronchoscope camera by exploring synthetic rubber bronchus phantoms. We simulated breath by adding periodic force to deform the phantom. We compared the camera positions from visual SLAM with the manually created ground truth of the camera pose. The number of successfully tracked frames was also compared between the original SLAM and the proposed method.

Results

We successfully tracked 29,559 frames at a speed of 80 ms per frame. This corresponds to 78.1% of all acquired frames. The average root mean square error for our technique was 3.02 mm, while that for the original was 3.61 mm.

Conclusion

We present a novel methodology using visual SLAM for bronchoscope tracking. Our experimental results showed that it is feasible to use visual SLAM for the estimation of the bronchoscope camera pose during bronchoscopic navigation. Our proposed method tracked more frames and showed higher accuracy than the original technique did. Future work will include combining the tracking results with virtual bronchoscopy and validation with in vivo cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Kathleen AC, Andrew JL, Susan S, Recinda LS, Anne-Michelle N, Nadia H, Jane SH, Robert NA, Albert UF, Jiemin M, Betsy AK, Ahmedin J (2018) Annual report to the nation on the status of cancer, part I: National cancer statistics. Am Cancer Soc 124(13):2785–2800

    Google Scholar 

  2. Bricault I, Ferretti G, Cinquin P (1998) Registration of real and CT-derived virtual bronchoscopic images to assist transbronchial biopsy. IEEE Trans Med Imaging 17(5):703–714

    Article  CAS  Google Scholar 

  3. Deguchi D, Mori K, Feuerstein M, Kitasaka T, Maurer CR, Suenaga Y, Takabatake H, Mori M, Natori H (2009) Selective image similarity measure for bronchoscope tracking based on image registration. Med Image Anal 13(4):621–633

    Article  Google Scholar 

  4. Bowling MR, Folch EE, Khandhar SJ, Kazakov J, Krimsky WS, LeMense GP, Linden PA, Murillo BA, Nead MA, Pritchett MA, Teba CV, Towe CW, Williams T, Anciano CJ (2019) Fiducial marker placement with electromagnetic navigation bronchoscopy: a subgroup analysis of the prospective, multicenter NAVIGATE study. Ther Adv Respir Dis 13:1–11

    Article  Google Scholar 

  5. Merritt SA, Khare R, Bascom R, Higgins WE (2013) Interactive CT-video registration for the continuous guidance of bronchoscopy. IEEE Trans Med Imaging 32(8):1376–1396

    Article  Google Scholar 

  6. Gilboa P, Blecher D (2004) Intrabody navigation system for medical applications, Google Patents. Patent number: US Patent 6,833,814. Dec. 21, 2004

  7. Appelbaum L, Sosna J, Nissenbaum Y, Benshtein A, Goldberg SN (2011) Electromagnetic navigation system for CT-guided biopsy of small lesions. Am J Roentgenol 196(5):1194–1200

    Article  Google Scholar 

  8. Luo X, Feuerstein M, Kitasaka T, Mori K (2012) Robust bronchoscope motion tracking using sequential Monte Carlo methods in navigated bronchoscopy: dynamic phantom and patient validation. Int J Comput Assist Radiol Surg 7(3):371–387

    Article  Google Scholar 

  9. Deguchi D, Mori K, Hasegawa J, Toriwaki J, Takabatake H, Natori H (2002) Camera motion tracking of real bronchoscope using epipolar geometry analysis and CT-derived bronchoscopic images. In: Medical imaging 2002: physiology and function from multidimensional images, vol 4683, pp 30–42

  10. Luo X, Mori K (2014) Robust endoscope motion estimation via an animated particle filter for electromagnetically navigated endoscopy. IEEE Trans Biomed Eng 61(1):85–95

    Article  Google Scholar 

  11. Mori K, Deguchi D, Ishitani K, Kitasaka T, Suenaga Y, Hasegawa Y, Imaizumi K, Takabatake H (2007) Bronchoscope tracking without fiducial markers using ultra-tiny electromagnetic tracking system and its evaluation in different environments. Int Conf Med Image Comput Comput Assis Interv 4792:644–651

    Google Scholar 

  12. Mori K, Deguchi D, Akiyama K (2005) Hybrid bronchoscope tracking using a magnetic tracking sensor and image registration. Int Conf Med Image Comput Comput Assis Interv 3750:543–550

    Google Scholar 

  13. Deguchi D, Feuerstein M, Kitasaka T, Suenaga Y, Ide I, Murase H, Imaizumi K, Hasegawa Y, Mori K (2012) Real-time marker-free patient registration for electromagnetic navigated bronchoscopy: a phantom study. Int J Comput Assist Radiol Surg 7(3):359–369

    Article  Google Scholar 

  14. Luo X, Mori K (2014) A discriminative structural similarity measure and its application to video-volume registration for endoscope three-dimensional motion tracking. IEEE Trans Med Imaging 33(6):1248–1261

    Article  Google Scholar 

  15. Kuo SW, TsengYF Dai KY, Chang YC, Chen KC, Lee JM (2019) Electromagnetic navigation bronchoscopy localization versus percutaneous CT-guided localization for lung resection via video-assisted thoracoscopic surgery: a propensity-matched study. J Clin Med 8(3):379

    Article  Google Scholar 

  16. Mori K, Deguchi D, Sugiyama J, Suenaga Y, Toriwaki J, Maurer JCR, Takabatake H, Natori H (2002) Tracking of a bronchoscope using epipolar geometry analysis and intensity-based image registration of real and virtual endoscopic images. Med Image Anal 6(3):321–336

    Article  CAS  Google Scholar 

  17. Wegner I, Biederer J, Tetzlaff R, Wolf I, Meinzer HP (2007) Evaluation and extension of a navigation system for bronchoscopy inside human lungs. In: Medical imaging 2007: international society for optics and photonics, visualization and image-guided procedures, p 65091H

  18. Wang C, Oda M, Hayashi Y, Kitasaka T, Honma H, Takabatake H, Mori M, Natori H, Mori K (2018) Improvement of robustness of SLAM-based bronchoscope tracking by posture guided feature matching. Int J Comput Assist Radiol Surg (CARS) 2018 13:11–12

    CAS  Google Scholar 

  19. Davison AJ (2003) Real-time simultaneous localisation and mapping with a single camera. In: International conference on computer vision, pp 1403–1410

  20. Engel, J, Schops T, Cremers D (2014) LSD-SLAM: large-scale direct monocular SLAM. In: European conference on computer vision, pp 834–849

  21. Newcombe RA, Lovegrove SJ, Davison AJ (2011) DTAM: dense tracking and mapping in real-time. In: 2011 International conference on computer vision, pp 2320–2327

  22. Mur-Artal R, Montiel JMM, Tardos JD (2015) ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans Robot 31(5):1147–1163

    Article  Google Scholar 

  23. Rublee E, Rabaud V, Konolige K, Bradski G (2011) ORB: an efficient alternative to SIFT or SURF. In: IEEE International conference on computer vision (ICCV), 2011, pp 2564–2571

  24. Nimura Y (2008) PLUTO: a common platform for computer-aided diagnosis. Med Imag Tech 26(3):187–191

    Google Scholar 

  25. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  26. Mori K, Suenaga Y, Toriwaki J (2002) Fast volume rendering based on software optimisation using multimedia instructions on PC platforms. In: Lemke HU, Vannier MW, Inamura K, Farman K, Doi K, Reiber JHC (eds) CARS 2002. Springer, Berlin, pp 467–472

  27. Mori K, Suenaga Y and Toriwaki J (2003) Fast software-based volume rendering using multimedia instructions on PC platforms and its application to virtual endoscopy. In Medical imaging 2003: physiology and function: methods, systems, and applications, vol 5031, pp 111–122

  28. Levoy M (1988) Display of surfaces from volume data. IEEE Comput Graph Appl 8(3):29–37

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank our laboratory members for their kind advice on experiments and the writing of the manuscript. Parts of this study were supported by JSPS KAKENHI (Grant Numbers 17H00867, 26108006 and 17K20099), and the JSPS Bilateral International Collaboration Grants.

Author information

Authors and Affiliations

  1. Graduate School of Informatics, Nagoya university, Furo-cho, Chikusa-ku, Nagoya, Aichi, 464-8601, Japan

    Cheng Wang, Masahiro Oda, Yuichiro Hayashi, Benjamin Villard & Kensaku Mori

  2. School of Information Science, Aichi Institute of Technology, Yachigusa 1247, Yakusa-cho, Toyota, Aichi, 470-0356, Japan

    Takayuki Kitasaka

  3. Department of Respiratory Medicine, Sapporo Minami-Sankyo Hospital, Nishi-6-chome, Minami-3-jo, Chuo-ku, Sapporo, Hokkaido, 060-0063, Japan

    Hirotsugu Takabatake

  4. Department of Respiratory Medicine, Sapporo-Kosei General Hospital, Higashi-8-chome, Kita-3-jo, Chuo-ku, Sapporo, Hokkaido, 060-0033, Japan

    Masaki Mori & Hirotoshi Honma

  5. Department of Respiratory Medicine, Keiwakai Nishioka Hospital, 1-52, 4-jo 4-chome, Nishioka, Toyohira-ku, Sapporo, Hokkaido, 062-0034, Japan

    Hiroshi Natori

Authors
  1. Cheng Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Masahiro Oda
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yuichiro Hayashi
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Benjamin Villard
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Takayuki Kitasaka
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Hirotsugu Takabatake
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Masaki Mori
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Hirotoshi Honma
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Hiroshi Natori
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Kensaku Mori
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Cheng Wang.

Ethics declarations

Conflict of interest

K. Mori is receiving research funding from Olympus (Grant No. 30,000USD).

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

This article does not contain any patient data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, C., Oda, M., Hayashi, Y. et al. A visual SLAM-based bronchoscope tracking scheme for bronchoscopic navigation. Int J CARS 15, 1619–1630 (2020). https://doi.org/10.1007/s11548-020-02241-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02241-9

Keywords