Multimodal registration network with multi-scale feature-crossing | International Journal of Computer Assisted Radiology and Surgery Skip to main content
Springer Nature Link
Log in

Multimodal registration network with multi-scale feature-crossing

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

A critical piece of information for prostate intervention and cancer treatment is provided by the complementary medical imaging modalities of ultrasound (US) and magnetic resonance imaging (MRI). Therefore, MRI–US image fusion is often required during prostate examination to provide contrast-enhanced TRUS, in which image registration is a key step in multimodal image fusion.

Methods

We propose a novel multi-scale feature-crossing network for the prostate MRI–US image registration task. We designed a feature-crossing module to enhance information flow in the hidden layer by integrating intermediate features between adjacent scales. Additionally, an attention block utilizing three-dimensional convolution interacts information between channels, improving the correlation between different modal features. We used 100 cases randomly selected from The Cancer Imaging Archive (TCIA) for our experiments. A fivefold cross-validation method was applied, dividing the dataset into five subsets. Four subsets were used for training, and one for testing, repeating this process five times to ensure each subset served as the test set once.

Results

We test and evaluate our technique using fivefold cross-validation. The cross-validation trials result in a median target registration error of 2.20 mm on landmark centroids and a median Dice of 0.87 on prostate glands, both of which were better than the baseline model. In addition, the standard deviation of the dice similarity coefficient is 0.06, which suggests that the model is stable.

Conclusion

We propose a novel multi-scale feature-crossing network for the prostate MRI–US image registration task. A random selection of 100 cases from The Cancer Imaging Archive (TCIA) was used to test and evaluate our approach using fivefold cross-validation. The experimental results showed that our method improves the registration accuracy. After registration, MRI and TURS images were more similar in structure and morphology, and the location and morphology of cancer were more accurately reflected in the images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Abbreviations

US:

Ultrasound

MRI:

Magnetic resonance imaging

TCIA:

The Cancer Imaging Archive

CNN:

Convolutional neural network

T2WI:

T2-weighted MRI images

DDF:

Dense displacement field

CC:

Correlation coefficient

MSE:

Mean square error

NCC:

Normalized correlation coefficient

MI:

Mutual information

TRE:

Target registration error

DSC:

Dice similarity coefficient

DSCmean:

Mean value of the dice similarity coefficient

DSCmedian:

Median of the dice similarity coefficient

DSCstd:

Standard deviation of the dice similarity coefficient

TREmean:

Mean value of the target registration error

TREmedian:

Median of the target registration error

ROIs:

Regions of interest

Bx Tip:

Location of the tip of the biopsy needle

Bx Base:

Location of the base of the biopsy needle

References

  1. Siegel R, Ma J, Zou Z, Jemal A (2014) Cancer statistics, 2014. CA Cancer J Clin 64(1):9–29

    Article  PubMed  Google Scholar 

  2. Cleary K, Peters TM (2010) Image-guided interventions: technology review and clinical applications. In: Yarmush M, Duncan J, Gray M (eds) Annual review of biomedical engineering, vol 12, pp 119–142. https://doi.org/10.1146/annurev-bioeng-070909-105249

  3. Yacoub JH, Verma S, Moulton JS, Eggener S, Oto A (2012) Imaging-guided prostate biopsy: conventional and emerging techniques. Radiographics 32(3):819–837. https://doi.org/10.1148/rg.323115053

    Article  PubMed  Google Scholar 

  4. Yang X, Rossi P, Mao H, Jani AB, Ogunleye T, Curran WJ, Liu T (2015) A MR-TRUS registration method for ultrasound-guided prostate interventions. In: Yaniv Z, Webster R (eds) Medical Imaging 2015: image-guided procedures, robotic interventions, and modeling. Proceedings of SPIE, vol. 9415 . https://doi.org/10.1117/12.2077825. SPIE; ALIO Ind; Alpin Med Syst; Modus Med Devices Inc; Bruker; Siemens; Natl Diagnost Imaging. Conference on Medical Imaging—Image-Guided Procedures, Robotic Interventions, and Modeling, Orlando, FL, FEB 22-24, 2015

  5. Lei Y, He X, Yao J, Wang T, Wang L, Li W, Curran WJ, Liu T, Xu D, Yang X (2021) Breast tumor segmentation in 3D automatic breast ultrasound using mask scoring R-CNN. Med Phys 48(1):204–214. https://doi.org/10.1002/mp.14569

    Article  CAS  PubMed  Google Scholar 

  6. Zhang Y, He X, Tian Z, Jeong JJ, Lei Y, Wang T, Zeng Q, Jani AB, Curran WJ, Patel P, Liu T, Yang X (2020) Multi-needle detection n 3D ultrasound images using unsupervised order-graph regularized sparse dictionary learning. IEEE Trans Med Imaging 39(7):2302–2315. https://doi.org/10.1109/TMI.2020.2968770

    Article  PubMed  PubMed Central  Google Scholar 

  7. Zhang Y, Tian Z, Lei Y, Wang T, Patel P, Jani A.B, Curran W.J, Liu T, Yang X (2020) Automatic multi-needle localization in ultrasound images using large margin mask RCNN for ultrasound-guided prostate brachytherapy. Phys Med Biol 6(5):20. https://doi.org/10.1088/1361-6560/aba410

    Article  Google Scholar 

  8. Hegde JV, Mulkern RV, Panych LP, Fennessy FM, Fedorov A, Maier SE, Tempany CMC (2013) Multiparametric MRI of prostate cancer: an update on state-of-the-art techniques and their performance in detecting and localizing prostate cancer. J Magn Reson Imaging 37(5):1035–1054. https://doi.org/10.1002/jmri.23860

    Article  PubMed  PubMed Central  Google Scholar 

  9. Lei Y, Jeong JJ, Wang T, Shu H-K, Patel P, Tian S, Liu T, Shim H, Mao H, Jani AB, Curran WJ, Yang X (2018) MRI-based pseudo CT synthesis using anatomical signature and alternating random forest with iterative refinement model. J Med Imaging 5(4):043504–043504. https://doi.org/10.1117/1.JMI.5.4.043504

    Article  Google Scholar 

  10. Lei Y, Wang T, Harms J, Fu Y, Dong X, Curran WJ, Liu T, Yang X ( 2019) CBCT-based synthetic MRI generation for CBCT-guided adaptive radiotherapy. In: Nguyen D, Xing L, Jiang S (eds) Artificial Intelligence in Radiation Therapy. First International Workshop, AIRT 2019. Held in Conjunction with MICCAI 2019. Proceedings. Lecture Notes in Computer Science (LNCS 11850), pp 154–61

  11. Liu Y, Lei Y, Wang Y, Shafai-Erfani G, Wang T, Tian S, Patel P, Jani AB, McDonald M, Curran WJ, Liu T, Zhou J, Yang X (2019) Evaluation of a deep learning-based pelvic synthetic CT generation technique for MRI-based prostate proton treatment planning. Phys Med Biol. https://doi.org/10.1088/1361-6560/ab41af

    Article  PubMed  PubMed Central  Google Scholar 

  12. Liu Y, Lei Y, Wang Y, Wang T, Ren L, Lin L, McDonald M, Curran WJ, Liu T, Zhou J, Yang X (2019) MRI-based treatment planning for proton radiotherapy: dosimetric validation of a deep learning-based liver synthetic CT generation method. Phys Med Biol. https://doi.org/10.1088/1361-6560/ab25bc

    Article  PubMed  PubMed Central  Google Scholar 

  13. Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. IEEE Trans Med Imaging 32(7):1153–1190. https://doi.org/10.1109/TMI.2013.2265603

    Article  PubMed  PubMed Central  Google Scholar 

  14. Boveiri HR, Khayami R, Javidan R, Mehdizadeh A (2020) Medical image registration using deep neural networks: a comprehensive review. Comput Electr Eng 87:106767. https://doi.org/10.1016/j.compeleceng.2020.106767

    Article  Google Scholar 

  15. Fu Y, Lei Y, Wang T, Curran WJ, Liu T, Yang X (2020) Deep learning in medical image registration: a review. Phys Med Biol. https://doi.org/10.1088/1361-6560/ab843e

    Article  PubMed  PubMed Central  Google Scholar 

  16. Eppenhof KAJ, Lafarge MW, Veta M, Pluim JPW (2020) Progressively trained convolutional neural networks for deformable image registration. IEEE Trans Med Imaging 39(5):1594–1604. https://doi.org/10.1109/TMI.2019.2953788

    Article  PubMed  Google Scholar 

  17. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV (2019) VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging 38(8):1788–1800. https://doi.org/10.1109/TMI.2019.2897538

    Article  Google Scholar 

  18. Hu Y, Modat M, Gibson E, Ghavami N, Bonmati E, Moore CM, Emberton M, Noble JA, Barratt DC, Vercauteren T ( 2018) Label-driven weakly-supervised learning for multimodal deformable image registration. In: 2018 IEEE 15TH international symposium on biomedical imaging (ISBI 2018). IEEE International Symposium on Biomedical Imaging, pp 1070–1074

  19. Hu Y, Modat M, Gibson E, Li W, Ghavamia N, Bonmati E, Wang G, Bandula S, Moore CM, Emberton M, Ourselin S, Noble JA, Barratt DC, Vercauteren T (2018) Weakly-supervised convolutional neural networks for multimodal image registration. Med Image Anal 49:1–13. https://doi.org/10.1016/j.media.2018.07.002

    Article  PubMed  PubMed Central  Google Scholar 

  20. Hu Y, Gibson E, Ghavami N, Bonmati E, Moore CM, Emberton M, Vercauteren T, Noble JA, Barratt DC (2018) Adversarial deformation regularization for training image registration neural networks. In: Frangi A, Schnabel J, Davatzikos C, AlberolaLopez C, Fichtinger G (eds) Medical image computing and computer assisted intervention—MICCAI 2018. PT I. Lecture Notes in Computer Science, vol 11070, pp 774–782. https://doi.org/10.1007/978-3-030-00928-1_87

  21. Zeng Q, Fu Y, Tian Z, Lei Y, Zhang Y, Wang T, Mao H, Liu T, Curran WJ, Jani AB, Patel P, Yang X (2020) Label-driven magnetic resonance imaging (MRI)-transrectal ultrasound (TRUS) registration using weakly supervised learning for MRI-guided prostate radiotherapy. Phys Med Biol. https://doi.org/10.1088/1361-6560/ab8cd6

    Article  PubMed  PubMed Central  Google Scholar 

  22. Song X, Chao H, Xu X, Guo H, Xu S, Turkbey B, Wood BJ, Sanford T, Wang G, Yan P (2022) Cross-modal attention for multi-modal image registration. Med Image Anal 82:102612. https://doi.org/10.1016/j.media.2022.102612

    Article  PubMed  PubMed Central  Google Scholar 

  23. Hu J, Shen L, Sun G ( 2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132– 7141

  24. Sonn GA, Natarajan S, Margolis DJA, MacAiran M, Lieu P, Huang J, Dorey FJ, Marks LS (2013) Targeted biopsy in the detection of prostate cancer using an office based magnetic resonance ultrasound fusion device. J Urol 189(1):86–91. https://doi.org/10.1016/j.juro.2012.08.095

    Article  PubMed  Google Scholar 

  25. Girija SS (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. Software available from tensorflow.org, vol 39(9)

  26. Gibson E, Li W, Sudre C, Fidon L, Shakir DI, Wang G, Eaton-Rosen Z, Gray R, Doel T, Hu Y, Whyntie T, Nachev P, Modat M, Barratt D.C, Ourselin S, Cardoso M.J, Vercauteren T (2018) NiftyNet: a deep-learning platform for medical imaging. Comput Methods Programs Biomed 158:113–122. https://doi.org/10.1016/j.cmpb.2018.01.025

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62273239.

Author information

Author notes

  1. Yi Fan, Lei Chen and Zhaodong Zhang have contributed equally to this work.

Authors and Affiliations

  1. Business School, University of Shanghai for Science and Technology, Jungong Road, Shanghai, 200093, China

    Shuting Liu & Guoliang Wei

  2. Puncture Intelligent Medical Technology Co Ltd, Xinzhuan Road, Shanghai, 201600, China

    Yi Fan & Zhaodong Zhang

  3. Shanghai Sixth People’s Hospital, Yishan Road, Shanghai, 200233, China

    Lei Chen

Authors
  1. Shuting Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Guoliang Wei
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yi Fan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Lei Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Zhaodong Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Guoliang Wei.

Ethics declarations

Conflict of interest

There is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Wei, G., Fan, Y. et al. Multimodal registration network with multi-scale feature-crossing. Int J CARS 19, 2269–2278 (2024). https://doi.org/10.1007/s11548-024-03258-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-024-03258-0

Keywords