Arges: Spatio-Temporal Transformer for Ulcerative Colitis Severity Assessment in Endoscopy Videos | SpringerLink
Skip to main content

Arges: Spatio-Temporal Transformer for Ulcerative Colitis Severity Assessment in Endoscopy Videos

  • Conference paper
  • First Online:
Machine Learning in Medical Imaging (MLMI 2024)

Abstract

Accurate assessment of disease severity from endoscopy videos in ulcerative colitis (UC) is crucial for evaluating drug efficacy in clinical trials. Severity is often measured by the Mayo Endoscopic Subscore (MES) and Ulcerative Colitis Endoscopic Index of Severity (UCEIS) score. However, expert MES/UCEIS annotation is time-consuming and susceptible to inter-rater variability, factors addressable by automation. Automation attempts with frame-level labels face challenges in fully-supervised solutions due to the prevalence of video-level labels in clinical trials. CNN-based weakly-supervised models (WSL) with end-to-end (e2e) training lack generalization to new disease scores and ignore spatio-temporal information crucial for accurate scoring. To address these limitations, we propose “Arges”, a deep learning framework that utilizes a transformer with positional encoding to incorporate spatio-temporal information from frame features to estimate disease severity scores in endoscopy video. Extracted features are derived from a foundation model (ArgesFM), pre-trained on a large diverse dataset from multiple clinical trials (61M frames, 3927 videos). We evaluate four UC disease severity scores, including MES and three UCEIS component scores. Test set evaluation indicates significant improvements, with F1 scores increasing by 4.1% for MES and 18.8%, 6.6%, 3.8% for the three UCEIS component scores compared to state-of-the-art methods. Prospective validation on previously unseen clinical trial data further demonstrates the model’s successful generalization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 6634
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 8293
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Allez, M., et al.: A phase 2b, randomised, double-blind, placebo-controlled, parallel-arm, multicenter study evaluating the safety and efficacy of tesnatilimab in patients with moderately to severely active Crohn’s disease. J. Crohn’s Colitis, jjad047 (2023)

    Google Scholar 

  2. Byrne, M.F., et al.: Application of deep learning models to improve ulcerative colitis endoscopic disease activity scoring under multiple scoring systems. J. Crohns Colitis 17(4), 463–471 (2023)

    Article  Google Scholar 

  3. Caron, M., et al.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650–9660 (2021)

    Google Scholar 

  4. Daperno, M., et al.: Inter-observer agreement in endoscopic scoring systems: preliminary report of an ongoing study from the Italian group for inflammatory bowel disease (IG-IBD). Dig. Liver Dis. 46(11), 969–973 (2014)

    Article  Google Scholar 

  5. Dosovitskiy, A., et al.: An image is worth 16 \(\times \) 16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  6. Hirsch, R., et al.: Self-supervised learning for endoscopic video analysis. In: Greenspan, H., et al. (eds.) International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 569–578. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43904-9_55

  7. Iacucci, M., et al.: A virtual chromoendoscopy artificial intelligence system to detect endoscopic and histologic activity/remission and predict clinical outcomes in ulcerative colitis. Endoscopy 55(04), 332–341 (2023)

    Article  Google Scholar 

  8. Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning, pp. 2127–2136. PMLR (2018)

    Google Scholar 

  9. Oquab, M., et al.: DINOv2: learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

  10. Paul, S., Roy, S., Roy-Chowdhury, A.K.: W-TALC: weakly-supervised temporal activity localization and classification. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 588–607. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_35

    Chapter  Google Scholar 

  11. Peyrin-Biroulet, L., et al.: Guselkumab in patients with moderately to severely active ulcerative colitis: QUASAR Phase 2b induction study. Gastroenterology 165(6), 1443–1457 (2023)

    Article  Google Scholar 

  12. Polat, G., Ergenc, I., Kani, H.T., Alahdab, Y.O., Atug, O., Temizel, A.: Class distance weighted cross-entropy loss for ulcerative colitis severity estimation. In: Yang, G., Aviles-Rivero, A., Roberts, M., Schonlieb, C.B. (eds.) Annual Conference on Medical Image Understanding and Analysis, pp. 157–171. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-12053-4_12

  13. Principi, M., et al.: Inter-observer agreement of a new endoscopic score for ulcerative colitis activity: preliminary experience. Diagnostics 10(4), 213 (2020)

    Article  Google Scholar 

  14. Rubin, D.T., et al.: Development of a novel ulcerative colitis endoscopic Mayo score prediction model using machine learning. Gastro Hep Advances (2023)

    Google Scholar 

  15. Sands, B.E., et al.: Ustekinumab versus adalimumab for induction and maintenance therapy in biologic-naive patients with moderately to severely active Crohn’s disease: a multicentre, randomised, double-blind, parallel-group, phase 3b trial. Lancet 399(10342), 2200–2211 (2022)

    Article  Google Scholar 

  16. Sands, B.E., et al.: Peficitinib, an oral janus kinase inhibitor, in moderate-to-severe ulcerative colitis: results from a randomised, phase 2 study. J. Crohns Colitis 12(10), 1158–1169 (2018)

    Article  Google Scholar 

  17. Sands, B.E., et al.: Ustekinumab as induction and maintenance therapy for ulcerative colitis. N. Engl. J. Med. 381(13), 1201–1214 (2019)

    Article  Google Scholar 

  18. Schroeder, K.W., Tremaine, W.J., Ilstrup, D.M.: Coated oral 5-aminosalicylic acid therapy for mildly to moderately active ulcerative colitis. N. Engl. J. Med. 317(26), 1625–1629 (1987)

    Article  Google Scholar 

  19. Schwab, E., et al.: Automatic estimation of ulcerative colitis severity from endoscopy videos using ordinal multi-instance learning. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 10(4), 425–433 (2022)

    Article  Google Scholar 

  20. Stidham, R.W., et al.: Using computer vision to improve endoscopic disease quantification in therapeutic clinical trials of ulcerative colitis. Gastroenterology 166(1), 155–167 (2024)

    Article  Google Scholar 

  21. Stidham, R.W., et al.: Performance of a deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis. JAMA Netw. Open 2(5), e193963–e193963 (2019)

    Article  MathSciNet  Google Scholar 

  22. Tian, Y., et al.: Contrastive transformer-based multiple instance learning for weakly supervised polyp frame detection. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 88–98. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16437-8_9

  23. Travis, S.P., et al.: Developing an instrument to assess the endoscopic severity of ulcerative colitis: the Ulcerative Colitis Endoscopic Index of Severity (UCEIS). Gut 61(4), 535–542 (2012)

    Article  Google Scholar 

  24. Vasilakakis, M.D., Diamantis, D., Spyrou, E., Koulaouzidis, A., Iakovidis, D.K.: Weakly supervised multilabel classification for semantic interpretation of endoscopy video frames. Evol. Syst. 11, 409–421 (2020)

    Article  Google Scholar 

  25. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  26. Wang, L., Xiong, Y., Lin, D., Van Gool, L.: UntrimmedNets for weakly supervised action recognition and detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4325–4334 (2017)

    Google Scholar 

  27. Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In: Greenspan, H., et al. (eds.) International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 101–111. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43996-4_10

  28. Xu, Z., Ali, S., Gupta, S., Leedham, S., East, J.E., Rittscher, J.: Patch-level instance-group discrimination with pretext-invariant learning for colitis scoring. In: Lian, C., Cao, X., Rekik, I., Xu, X., Cui, Z. (eds.) International Workshop on Machine Learning in Medical Imaging, pp. 101–110. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-21014-3_11

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Krishna Chaitanya .

Editor information

Editors and Affiliations

Ethics declarations

Disclosure of Interests

All authors were employees of Janssen R&D, LLC, and may own company stock/stock options.

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 131 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chaitanya, K. et al. (2025). Arges: Spatio-Temporal Transformer for Ulcerative Colitis Severity Assessment in Endoscopy Videos. In: Xu, X., Cui, Z., Rekik, I., Ouyang, X., Sun, K. (eds) Machine Learning in Medical Imaging. MLMI 2024. Lecture Notes in Computer Science, vol 15242. Springer, Cham. https://doi.org/10.1007/978-3-031-73290-4_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73290-4_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73292-8

  • Online ISBN: 978-3-031-73290-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics