Abstract
In this paper we perform an empirical evaluation of variants of deep learning methods to automatically localize anatomical landmarks in bioimages of fishes acquired using different imaging modalities (microscopy and radiography). We compare two methodologies namely heatmap based regression and multivariate direct regression, and evaluate them in combination with several Convolutional Neural Network (CNN) architectures. Heatmap based regression approaches employ Gaussian or Exponential heatmap generation functions combined with CNNs to output the heatmaps corresponding to landmark locations whereas direct regression approaches output directly the (x, y) coordinates corresponding to landmark locations. In our experiments, we use two microscopy datasets of Zebrafish and Medaka fish and one radiography dataset of gilthead Seabream. On our three datasets, the heatmap approach with Exponential function and U-Net architecture performs better. Datasets and open-source code for training and prediction are made available to ease future landmark detection research and bioimaging applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
Code: https://github.com/cytomine-uliege. Demo server with datasets: http://research.uliege.cytomine.org/ username: eccv2022bic password: deep-fish.
References
Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/, software available from tensorflow.org
Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: new benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3686–3693 (2014)
Aubert, B., Vazquez, C., Cresson, T., Parent, S., De Guise, J.: Automatic spine and pelvis detection in frontal x-rays using deep neural networks for patch displacement learning. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 1426–1429. IEEE (2016)
Bookstein, F.L.: Combining the tools of geometric morphometrics. In: Marcus, L.F., Corti, M., Loy, A., Naylor, G.J.P., Slice, D.E. (eds) Advances in Morphometrics. NATO ASI Series, vol 284, pp. 131–151. Springer, Boston (1996). https://doi.org/10.1007/978-1-4757-9083-2_12(1996)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. Ieee (2009)
Edwards, C.A., Goyal, A., Rusheen, A.E., Kouzani, A.Z., Lee, K.H.: Deepnavnet: automated landmark localization for neuronavigation. Front. Neurosci. 15, 730 (2021)
Fragkoulis, S., Printzi, A., Geladakis, G., Katribouzas, N., Koumoundouros, G.: Recovery of haemal lordosis in gilthead seabream (sparus aurata l.). Sci. Rep. 9(1), 1–11 (2019)
Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_6
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, W., Yang, C., Hou, T.: Spine landmark localization with combining of heatmap regression and direct coordinate regression. arXiv preprint arXiv:2007.05355 (2020)
Ibragimov, B., Vrtovec, T.: Landmark-based statistical shape representations. In: Statistical Shape and Deformation Analysis, pp. 89–113. Elsevier (2017)
Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 34–50. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_3
Jarque, S., Rubio-Brotons, M., Ibarra, J., Ordoñez, V., Dyballa, S., Miñana, R., Terriente, J.: Morphometric analysis of developing zebrafish embryos allows predicting teratogenicity modes of action in higher vertebrates. Reprod. Toxicol. 96, 337–348 (2020)
Khabarlak, K., Koriashkina, L.: Fast facial landmark detection and applications: A survey. arXiv preprint arXiv:2101.10808 (2021)
Lai, H., Xiao, S., Pan, Y., Cui, Z., Feng, J., Xu, C., Yin, J., Yan, S.: Deep recurrent regression for facial landmark detection. IEEE Trans. Circuits Syst. Video Technol. 28(5), 1144–1157 (2016)
Lee, H., Park, M., Kim, J.: Cephalometric landmark detection in dental x-ray images using convolutional neural networks. In: Medical Imaging 2017: Computer-Aided Diagnosis, vol. 10134, p. 101341W. International Society for Optics and Photonics (2017)
Lindner, C., Cootes, T.F.: Fully automatic cephalometric evaluation using random forest regression-voting. In: IEEE International Symposium on Biomedical Imaging (ISBI) 2015-Grand Challenges in Dental X-ray Image Analysis-Automated Detection and Analysis for Diagnosis in Cephalometric X-ray Image (2015)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Loy, B.A., Boglione, C., Cataudella, S.: Geometric morphometrics and morpho-anatomy: a combined tool in the study of sea bream (sparus aurata, sparidae) shape. J. Appl. Ichthyol. 15(3), 104–110 (1999)
Mahpod, S., Das, R., Maiorana, E., Keller, Y., Campisi, P.: Facial landmarks localization using cascaded neural networks. Comput. Vis. Image Underst. 205, 103171 (2021)
Marée, R., Rollus, L., Stévens, B., Hoyoux, R., Louppe, G., Vandaele, R., Begon, J.M., Kainz, P., Geurts, P., Wehenkel, L.: Collaborative analysis of multi-gigapixel imaging data using cytomine. Bioinformatics 32(9), 1395–1401 (2016)
Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., Bethge, M.: Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21(9), 1281–1289 (2018)
Mohseni, H., Kasaei, S.: Automatic localization of cephalometric landmarks. In: 2007 IEEE International Symposium on Signal Processing and Information Technology, pp. 396–401. IEEE (2007)
Park, J.H., Hwang, H.W., Moon, J.H., Yu, Y., Kim, H., Her, S.B., Srinivasan, G., Aljanabi, M.N.A., Donatelli, R.E., Lee, S.J.: Automated identification of cephalometric landmarks: Part 1-comparisons between the latest deep-learning methods yolov3 and ssd. Angle Orthod. 89(6), 903–909 (2019)
Payer, C., Štern, D., Bischof, H., Urschler, M.: Regressing heatmaps for multiple landmark localization using CNNs. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 230–238. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_27
Payer, C., Štern, D., Bischof, H., Urschler, M.: Integrating spatial configuration into heatmap regression based CNNs for landmark localization. Med. Image Anal. 54, 207–219 (2019)
Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015)
Riegler, G., Urschler, M., Ruther, M., Bischof, H., Stern, D.: Anatomical landmark detection in medical applications driven by synthetic data. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 12–16 (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, U-net: Convolutional networks for biomedical image segmentation (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Samet, N., Akbas, E.: Hprnet: hierarchical point regression for whole-body human pose estimation. Image Vis. Comput. 115, 104285 (2021)
Song, Y., Qiao, X., Iwamoto, Y., Chen, Y.W.: Automatic cephalometric landmark detection on x-ray images using a deep-learning method. Appl. Sci. 10(7), 2547 (2020)
Stern, O., Marée, R., Aceto, J., Jeanray, N., Muller, M., Wehenkel, L., Geurts, P.: Automatic localization of interest points in zebrafish images with tree-based methods. In: Loog, M., Wessels, L., Reinders, M.J.T., de Ridder, D. (eds.) PRIB 2011. LNCS, vol. 7036, pp. 179–190. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24855-9_16
Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)
Torosdagli, N., Liberton, D.K., Verma, P., Sincan, M., Lee, J.S., Bagci, U.: Deep geodesic learning for segmentation and anatomical landmarking. IEEE Trans. Med. Imaging 38(4), 919–931 (2018)
Vandaele, R., Aceto, J., Muller, M., Peronnet, F., Debat, V., Wang, C.W., Huang, C.T., Jodogne, S., Martinive, P., Geurts, P., et al.: Landmark detection in 2d bioimages for geometric morphometrics: a multi-resolution tree-based approach. Sci. Rep. 8(1), 1–13 (2018)
Verhaegen, Y., Adriaens, D., De Wolf, T., Dhert, P., Sorgeloos, P.: Deformities in larval gilthead sea bream (sparus aurata): a qualitative and quantitative analysis using geometric morphometrics. Aquaculture 268(1–4), 156–168 (2007)
Weinhardt, V., Shkarin, R., Wernet, T., Wittbrodt, J., Baumbach, T., Loosli, F.: Quantitative morphometric analysis of adult teleost fish by x-ray computed tomography. Sci. Rep. 8(1), 1–12 (2018)
Xu, Z., Li, B., Yuan, Y., Geng, M.: Anchorface: an anchor-based facial landmark detector across large poses. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 3092–3100 (2021)
Yeh, Y.C., Weng, C.H., Huang, Y.J., Fu, C.J., Tsai, T.T., Yeh, C.Y.: Deep learning approach for automatic landmark detection and alignment analysis in whole-spine lateral radiographs. Sci. Rep. 11(1), 1–15 (2021)
Acknowledgements
N.K., R.R., C.D.B. and Z.D. are fellows of the BioMedAqu project, M.M. is a “Maître de Recherche" at FNRS. Raphaël Marée is supported by the BigPicture EU Research and Innovation Action (Grant agreement number 945358). This work was partially supported by Service Public de Wallonie Recherche under Grant No. 2010235 - ARIAC by DIGITALWALLONIA4.AI. Computational infrastructure is partially supported by ULiège, Wallonia and Belspo funds. We would like to thank the GIGA zebrafish platform (H. Pendeville-Samain) for taking care of and delivering the zebrafish larvae.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, N. et al. (2023). Empirical Evaluation of Deep Learning Approaches for Landmark Detection in Fish Bioimages. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13804. Springer, Cham. https://doi.org/10.1007/978-3-031-25069-9_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-25069-9_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25068-2
Online ISBN: 978-3-031-25069-9
eBook Packages: Computer ScienceComputer Science (R0)