{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,13]],"date-time":"2025-04-13T12:06:31Z","timestamp":1744545991543,"version":"3.37.3"},"reference-count":47,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,11,23]],"date-time":"2022-11-23T00:00:00Z","timestamp":1669161600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Institute for Artificial Intelligence Research and Development of Serbia"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"Recent methods for automatic blood vessel segmentation from fundus images have been commonly implemented as convolutional neural networks. While these networks report high values for objective metrics, the clinical viability of recovered segmentation masks remains unexplored. In this paper, we perform a pilot study to assess the clinical viability of automatically generated segmentation masks in the diagnosis of diseases affecting retinal vascularization. Five ophthalmologists with clinical experience were asked to participate in the study. The results demonstrate low classification accuracy, inferring that generated segmentation masks cannot be used as a standalone resource in general clinical practice. The results also hint at possible clinical infeasibility in experimental design. In the follow-up experiment, we evaluate the clinical quality of masks by having ophthalmologists rank generation methods. The ranking is established with high intra-observer consistency, indicating better subjective performance for a subset of tested networks. The study also demonstrates that objective metrics are not correlated with subjective metrics in retinal segmentation tasks for the methods involved, suggesting that objective metrics commonly used in scientific papers to measure the method\u2019s performance are not plausible criteria for choosing clinically robust solutions.<\/jats:p>","DOI":"10.3390\/s22239101","type":"journal-article","created":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T08:58:16Z","timestamp":1669280296000},"page":"9101","source":"Crossref","is-referenced-by-count":1,"title":["Comparing the Clinical Viability of Automated Fundus Image Segmentation Methods"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6100-5871","authenticated-orcid":false,"given":"Gorana","family":"Goji\u0107","sequence":"first","affiliation":[{"name":"The Institute for Artificial Intelligence Research and Development of Serbia, 21102 Novi Sad, Serbia"},{"name":"Faculty of Technical Sciences, University of Novi Sad, 21102 Novi Sad, Serbia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8662-1366","authenticated-orcid":false,"given":"Veljko B.","family":"Petrovi\u0107","sequence":"additional","affiliation":[{"name":"Faculty of Technical Sciences, University of Novi Sad, 21102 Novi Sad, Serbia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8623-1923","authenticated-orcid":false,"given":"Dinu","family":"Dragan","sequence":"additional","affiliation":[{"name":"Faculty of Technical Sciences, University of Novi Sad, 21102 Novi Sad, Serbia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0495-8788","authenticated-orcid":false,"given":"Du\u0161an B.","family":"Gaji\u0107","sequence":"additional","affiliation":[{"name":"Faculty of Technical Sciences, University of Novi Sad, 21102 Novi Sad, Serbia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0455-9552","authenticated-orcid":false,"given":"Dragi\u0161a","family":"Mi\u0161kovi\u0107","sequence":"additional","affiliation":[{"name":"The Institute for Artificial Intelligence Research and Development of Serbia, 21102 Novi Sad, Serbia"}]},{"given":"Vladislav","family":"D\u017eini\u0107","sequence":"additional","affiliation":[{"name":"Eye Clinic D\u017eini\u0107, 21107 Novi Sad, Serbia"}]},{"given":"Zorka","family":"Grgi\u0107","sequence":"additional","affiliation":[{"name":"Eye Clinic D\u017eini\u0107, 21107 Novi Sad, Serbia"}]},{"given":"Jelica","family":"Panteli\u0107","sequence":"additional","affiliation":[{"name":"Institute of Eye Diseases, University Clinical Center of Serbia, 11000 Belgrade, Serbia"}]},{"given":"Ana","family":"Oros","sequence":"additional","affiliation":[{"name":"Eye Clinic D\u017eini\u0107, 21107 Novi Sad, Serbia"},{"name":"Institute of Neonatology, 11000 Belgrade, Serbia"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1136\/bjo.85.3.261","article-title":"World blindness: A 21st century perspective","volume":"85","author":"Taylor","year":"2001","journal-title":"Br. J. Ophthalmol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1136\/svn-2017-000101","article-title":"Artificial intelligence in healthcare: Past, present and future","volume":"2","author":"Jiang","year":"2017","journal-title":"Stroke Vasc. Neurol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","article-title":"Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation","volume":"36","author":"Kamnitsas","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neuroimage.2019.03.068","article-title":"Multi-branch convolutional neural network for multiple sclerosis lesion segmentation","volume":"196","author":"Aslani","year":"2019","journal-title":"NeuroImage"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"105241","DOI":"10.1016\/j.cmpb.2019.105241","article-title":"Skin lesion segmentation using high-resolution convolutional neural network","volume":"186","author":"Xie","year":"2020","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.mri.2019.05.020","article-title":"Automatic brain tissue segmentation in fetal MRI using convolutional neural networks","volume":"64","author":"Khalili","year":"2019","journal-title":"Magn. Reson. Imaging"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6769","DOI":"10.1007\/s00521-019-04700-0","article-title":"A two-stage approach for automatic liver segmentation with Faster R-CNN and DeepLab","volume":"32","author":"Tang","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_8","unstructured":"Abraham, A., Siarry, P., Ma, K., and Kaklauskas, A. (2021). Automatic Lung Segmentation in CT Images Using Mask R-CNN for Mapping the Feature Extraction in Supervised Methods of Machine Learning. Intelligent Systems Design and Applications, Springer International Publishing."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Shaziya, H., and Shyamala, K. (2020, January 16\u201318). Pulmonary CT Images Segmentation using CNN and UNet Models of Deep Learning. Proceedings of the 2020 IEEE Pune Section International Conference (IEEE PuneCon 2020), Maharashtra, India.","DOI":"10.1109\/PuneCon50868.2020.9362463"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-020-19557-4","article-title":"A convolutional neural network segments yeast microscopy images with high accuracy","volume":"11","author":"Dietler","year":"2020","journal-title":"Nat. Comm."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Scherr, T., L\u00f6ffler, K., B\u00f6hland, M., and Mikut, R. (2020). Cell segmentation and tracking using CNN-based distance predictions and a graph-based matching strategy. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0243219"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"56","DOI":"10.3389\/fncom.2019.00056","article-title":"Automatic Brain Tumor Segmentation Based on Cascaded Convolutional Neural Networks With Uncertainty Estimation","volume":"13","author":"Wang","year":"2019","journal-title":"Front. Comp. Neurosci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-021-90428-8","article-title":"Brain tumor segmentation based on deep learning and an attention mechanism using MRI multi-modalities brain images","volume":"11","author":"Ranjbarzadeh","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Nadeem, M.W., Goh, H.G., Hussain, M., Liew, S.Y., Andonovic, I., and Khan, M.A. (2022). Deep Learning for Diabetic Retinopathy Analysis: A Review, Research Challenges, and Future Directions. Sensors, 22.","DOI":"10.3390\/s22186780"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kelly, C.J., Karthikesalingam, A., Suleyman, M., Corrado, G., and King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC Med., 17.","DOI":"10.1186\/s12916-019-1426-2"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"429","DOI":"10.3233\/IDA-2002-6504","article-title":"The class imbalance problem: A systematic study","volume":"6","author":"Japkowicz","year":"2002","journal-title":"Intell. Data Anal."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1791","DOI":"10.1038\/s41433-019-0510-3","article-title":"Evaluation of an AI system for the automated detection of glaucoma from stereoscopic optic disc photographs: The European Optic Disc Assessment Study","volume":"33","author":"Rogers","year":"2019","journal-title":"Eye"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.ajo.2019.11.006","article-title":"Human Versus Machine: Comparing a Deep Learning Algorithm to Human Gradings for Detecting Glaucoma on Fundus Photographs","volume":"211","author":"Jammal","year":"2020","journal-title":"Am. J. Ophthalmol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"419","DOI":"10.2147\/OPTH.S235751","article-title":"Artificial Intelligence to Identify Retinal Fundus Images, Quality Validation, Laterality Evaluation, Macular Degeneration, and Suspected Glaucoma","volume":"14","author":"Zapata","year":"2020","journal-title":"Clin. Ophthalmol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.ogla.2018.04.002","article-title":"A Deep Learning-Based Algorithm Identifies Glaucomatous Discs Using Monoscopic Fundus Photographs","volume":"1","author":"Liu","year":"2018","journal-title":"Ophthalmol. Glaucoma"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.eclinm.2019.03.001","article-title":"Diagnostic Efficacy and Therapeutic Decision-making Capacity of an Artificial Intelligence Platform for Childhood Cataracts in Eye Clinics: A Multicentre Randomized Controlled Trial","volume":"9","author":"Lin","year":"2019","journal-title":"eClinicalMedicine"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"101633","DOI":"10.1016\/j.eclinm.2022.101633","article-title":"A cascade eye diseases screening system with interpretability and expandability in ultra-wide field fundus images: A multicentre diagnostic accuracy study","volume":"53","author":"Cao","year":"2022","journal-title":"eClinicalMedicine"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1264","DOI":"10.1016\/j.ophtha.2018.01.034","article-title":"Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy","volume":"125","author":"Krause","year":"2018","journal-title":"Ophthalmology"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1533","DOI":"10.1016\/j.ophtha.2019.06.005","article-title":"A Deep Learning Approach for Automated Detection of Geographic Atrophy from Color Fundus Photographs","volume":"126","author":"Keenan","year":"2019","journal-title":"Ophthalmology"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1111\/aos.14306","article-title":"Evaluation of a deep learning system for the joint automated detection of diabetic retinopathy and age-related macular degeneration","volume":"98","author":"Contreras","year":"2020","journal-title":"Acta Ophthalmol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1016\/j.ophtha.2018.11.015","article-title":"DeepSeeNet: A Deep Learning Model for Automated Classification of Patient-based Age-related Macular Degeneration Severity from Color Fundus Photographs","volume":"126","author":"Peng","year":"2019","journal-title":"Ophthalmology"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1016\/j.ophtha.2021.12.017","article-title":"DeepLensNet: Deep Learning Automated Diagnosis and Quantitative Classification of Cataract Type and Severity","volume":"129","author":"Keenan","year":"2022","journal-title":"Ophthalmology"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1016\/j.jaapos.2007.09.005","article-title":"Plus disease in retinopathy of prematurity: Pilot study of computer-based and expert diagnosis","volume":"11","author":"Gelman","year":"2007","journal-title":"J. Am. Assoc. Pediatric Ophthalmol. Strabismus"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1038\/s41551-020-00626-4","article-title":"A deep-learning system for the assessment of cardiovascular disease risk via the measurement of retinal-vessel calibre","volume":"5","author":"Cheung","year":"2021","journal-title":"Nat. Biomed. Eng."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1007\/s13244-018-0639-9","article-title":"Convolutional neural networks: An overview and application in radiology","volume":"9","author":"Yamashita","year":"2018","journal-title":"Insights Imaging"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Guo, C., Szemenyei, M., Yi, Y., Wang, W., Chen, B., and Fan, C. (2021, January 10\u201315). SA-UNet: Spatial Attention U-Net for Retinal Vessel Segmentation. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9413346"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Li, L., Verma, M., Nakashima, Y., Nagahara, H., and Kawasaki, R. (2020, January 1\u20135). IterNet: Retinal Image Segmentation Utilizing Structural Redundancy in Vessel Networks. Proceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093621"},{"key":"ref_33","unstructured":"Zhuang, J. (2019). LadderNet: Multi-path networks based on U-Net for medical image segmentation. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Navab, N., Hornegger, J., Wells, W.M., and Frangi, A.F. (2015, January 5\u20139). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer\u2014Assisted Intervention\u2014MICCAI 18th International Conference, Munich, Germany. Lecture Notes in Computer, Science.","DOI":"10.1007\/978-3-319-24571-3"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1016\/j.eswa.2018.06.034","article-title":"Retinal vessel segmentation based on Fully Convolutional Neural Networks","volume":"112","author":"Oliveira","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_36","unstructured":"Son, J., Park, S.J., and Jung, K.H. (2017). Retinal Vessel Segmentation in Fundoscopic Images with Generative Adversarial Networks. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yao, Z., He, K., Zhou, H., Zhang, Z., Zhu, G., Xing, C., Zhang, J., Zhang, Z., Shao, B., and Tao, Y. (2020, January 14\u201317). Eye3DVas: Three-dimensional reconstruction of retinal vascular structures by integrating fundus image features. Proceedings of the Frontiers in Optics, Washington, DC, USA.","DOI":"10.1364\/FIO.2020.JTu1B.22"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"501","DOI":"10.1109\/TMI.2004.825627","article-title":"Ridge-based vessel segmentation in color images of the retina","volume":"23","author":"Staal","year":"2004","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1109\/42.845178","article-title":"Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response","volume":"19","author":"Hoover","year":"2000","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2538","DOI":"10.1109\/TBME.2012.2205687","article-title":"An Ensemble Classification-Based Approach Applied to Retinal Blood Vessel Segmentation","volume":"59","author":"Fraz","year":"2012","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_41","unstructured":"(2022, November 16). SurveyJS\u2014JavaScript Survey and Form Library. Available online: https:\/\/github.com\/surveyjs\/survey-library."},{"key":"ref_42","first-page":"51","article-title":"The Copeland method","volume":"8","author":"Saari","year":"1996","journal-title":"Econ. Theory"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1037\/1082-989X.10.3.317","article-title":"An alternative to Cohen\u2019s standardized mean difference effect size: A robust parameter and confidence interval in the two independent groups case","volume":"10","author":"Algina","year":"2005","journal-title":"Psychol. Methods"},{"key":"ref_44","unstructured":"R Core Team (2022). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_45","unstructured":"Revelle, W. (2022). Psych: Procedures for Psychological, Psychometric, and Personality Research, Northwestern University. R package version 2.2.5."},{"key":"ref_46","unstructured":"Gamer, M. (2019). irr: Various Coefficients of Interrater Reliability and Agreement, R package version 0.84.1."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"464","DOI":"10.3758\/s13428-019-01246-w","article-title":"Robust Statistical Methods in R Using the WRS2 Package","volume":"52","author":"Mair","year":"2020","journal-title":"Behav. Res. Methods"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9101\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,6]],"date-time":"2025-01-06T04:07:31Z","timestamp":1736136451000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9101"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,23]]},"references-count":47,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["s22239101"],"URL":"https:\/\/doi.org\/10.3390\/s22239101","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,11,23]]}}}