Abstract
This paper proposes a novel transformer-based model architecture for medical imaging problems involving analysis of vertebrae. It considers two applications of such models in MR images: (a) detection of spinal metastases and the related conditions of vertebral fractures and metastatic cord compression, (b) radiological grading of common degenerative changes in intervertebral discs. Our contributions are as follows: (i) We propose a Spinal Context Transformer (SCT), a deep-learning architecture suited for the analysis of repeated anatomical structures in medical imaging such as vertebral bodies (VBs). Unlike previous related methods, SCT considers all VBs as viewed in all available image modalities together, making predictions for each based on context from the rest of the spinal column and all available imaging modalities. (ii) We apply the architecture to a novel and important task – detecting spinal metastases and the related conditions of cord compression and vertebral fractures/collapse from multi-series spinal MR scans. This is done using annotations extracted from free-text radiological reports as opposed to bespoke annotation. However, the resulting model shows strong agreement with vertebral-level bespoke radiologist annotations on the test set. (iii) We also apply SCT to an existing problem – radiological grading of inter-vertebral discs (IVDs) in lumbar MR scans for common degenerative changes. We show that by considering the context of vertebral bodies in the image, SCT improves the accuracy for several gradings compared to previously published models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: a joint video and image encoder for end-to-end retrieval. In: IEEE International Conference on Computer Vision (2021)
Burns, J.E., Yao, J., Wiese, T.S., Muñoz, H.E., Jones, E.C., Summers, R.M.: Automated detection of sclerotic metastases in the thoracolumbar spine at CT. Radiology 268(1), 69–78 (2013)
Cai, Y., Osman, S., Sharma, M., Landis, M., Li, S.: Multi-modality vertebra recognition in arbitrary views using 3D deformable hierarchical model. IEEE Trans. Med. Imaging 34(8), 1676–1693 (2015)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Casey, A., et al.: A systematic review of natural language processing applied to radiology reports. BMC Med. Inf. Decis. Making 21(1), 1–18 (2021)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of CVPR (2009)
Fisher, C.G., et al.: A novel classification system for spinal instability in neoplastic disease: an evidence-based approach and expert consensus from the spine oncology study group. Spine 35(22), E1221–E1229 (2010)
Forsberg, D., Sjöblom, E., Sunshine, J.L.: Detection and labeling of vertebrae in MR images using deep learning with clinical annotations as training data. J. Digit. Imaging 30(4), 406–412 (2017)
Gabeur, V., Sun, C., Alahari, K., Schmid, C.: Multi-modal transformer for video retrieval. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 214–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_13
Glocker, B., Feulner, J., Criminisi, A., Haynor, D.R., Konukoglu, E.: Automatic localization and identification of vertebrae in arbitrary field-of-view CT scans. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7512, pp. 590–598. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33454-2_73
Glocker, B., Zikic, D., Konukoglu, E., Haynor, D.R., Criminisi, A.: Vertebrae localization in pathological spine CT via dense classification from sparse annotations. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8150, pp. 262–270. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40763-5_33
Hammon, M., et al.: Automatic detection of lytic and blastic thoracolumbar spine metastases on computed tomography. Eur. Radiol. 23(7), 1862–1870 (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of CVPR (2016)
Jamaludin, A., Kadir, T., Zisserman, A.: Self-supervised learning for spinal MRIs. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 294–302. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_34
Jamaludin, A., Kadir, T., Zisserman, A.: SpineNet: automated classification and evidence visualization in spinal MRIs. Med. Image Anal. 41, 63–73 (2017)
Jamaludin, A., et al.: Automation of reading of radiological features from magnetic resonance images (MRIs) of the lumbar spine without human intervention is comparable with an expert radiologist. Eur. Spine J. 26, 1374–1383 (2017)
Lewandrowskl, K.U., et al.: Feasibility of deep learning algorithms for reporting in routine spine magnetic resonance imaging. Int. J. Spine Surg. 14, S86–S97 (2022)
Lootus, M., Kadir, T., Zisserman, A.: Vertebrae detection and labelling in lumbar MR images. In: Yao, J., Klinder, T., Li, S. (eds.) Computational Methods and Clinical Applications for Spine Imaging. LNCVB, vol. 17, pp. 219–230. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07269-2_19
Lu, J.T., et al.: Deep spine: automated lumbar vertebral segmentation, disc-level designation, and spinal stenosis grading using deep learning. In: Machine Learning for Healthcare (2018)
Maccauro, G., Spinelli, M.S., Mauro, S., Perisano, C., Graci, C., Rosa, M.A.: Physiopathology of spine metastasis. Int. J. Surg. Oncol. 2011 (2011)
Meinhardt, T., Kirillov, A., Leal-Taixe, L., Feichtenhofer, C.: TrackFormer: multi-object tracking with transformers. In: Proceedings of CVPR (2022)
Merali, Z., Wang, J.Z., Badhiwala, J.H., Witiw, C.D., Wilson, J.R., Fehlings, M.G.: A deep learning model for detection of cervical spinal cord compression in MRI scans. Sci. Rep. 11(1), 1–11 (2021)
Ortiz Gomez, J.: The incidence of vertebral body metastases. Int. Orthop. 19(5), 309–311 (1995)
Pfirrmann, C.W.A., Metzdorf, A., Zanetti, M., Hodler, J., Boos, N.: Magnetic resonance classification of lumbar intervertebral disc degeneration. Spine 26(17), 1873–1878 (2001)
Shaw, B., Mansfield, F.L., Borges, L.: One-stage posterolateral decompression and stabilization for primary and metastatic vertebral tumors in the thoracic and lumbar spine. J. Neurosurg. 70(3), 405–410 (1989)
Tao, R., Zheng, G.: Spine-transformers: vertebra detection and localization in arbitrary field-of-view spine CT with transformers. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 93–103. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87199-4_9
Tokuhashi, Y., Uei, H., Oshima, M., Ajiro, Y.: Scoring system for prediction of metastatic spine tumor prognosis. World J. Orthop. 5(3), 262–271 (2014)
van Tol, F.R., Massier, J.R.A., Frederix, G.W.J., Öner, F.C., Verkooijen, H.M., Verlaan, J.J.: Costs associated with timely and delayed surgical treatment of spinal metastases. Glob. Spine J. (2021)
van Tol, F.R., Versteeg, A.L., Verkooijen, H.M., Öner, F.C., Verlaan, J.J.: Time to surgical treatment for metastatic spinal disease: identification of delay intervals. Glob. Spine J. (2021)
Wang, J., Fang, Z., Lang, N., Yuan, H., Su, M.Y., Baldi, P.: A multi-resolution approach for spinal metastasis detection using deep Siamese neural networks. Comput. Biol. Med. 84, 137–146 (2017)
Windsor, R., Jamaludin, A.: The ladder algorithm: finding repetitive structures in medical images by induction. In: IEEE ISBI (2020)
Windsor, R., Jamaludin, A., Kadir, T., Zisserman, A.: A convolutional approach to vertebrae detection and labelling in whole spine MRI. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12266, pp. 712–722. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59725-2_69
Windsor, R., Jamaludin, A., Kadir, T., Zisserman, A.: SpineNetV2: automated detection, labelling and radiological grading of clinical MR scans. In: Technical report arXiv:2205.01683 (2022)
Zhao, S., Chen, B., Chang, H., Wu, X., Li, S.: Discriminative dictionary-embedded network for comprehensive vertebrae tumor diagnosis. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12266, pp. 691–701. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59725-2_67
Acknowledgements and Ethics:
Ethics for the spinal cancer dataset extraction are provided by OSCLMRIC (IRAS project ID: 207857). We are grateful to Dr. Sarim Ather, Dr. Jill Urban and Prof. Jeremy Fairbank for insightful conversations on the clinical aspect of this work as well as Prof. Ian McCall for annotating the data. Finally, we thank to our funders: Cancer Research UK via the EPSRC AIMS CDT and EPSRC Programme Grant Visual AI (EP/T025872/1).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Windsor, R., Jamaludin, A., Kadir, T., Zisserman, A. (2022). Context-Aware Transformers for Spinal Cancer Detection and Radiological Grading. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13433. Springer, Cham. https://doi.org/10.1007/978-3-031-16437-8_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-16437-8_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16436-1
Online ISBN: 978-3-031-16437-8
eBook Packages: Computer ScienceComputer Science (R0)
