Abstract
Filamentous objects are ubiquitous in biomedical images, and segmenting individual filaments is fundamental for biomedical research. Unlike common objects with well-defined boundaries and centers, filaments are thin, non-rigid, varying in shape, and often densely overlapping. These properties make it extremely challenging to extract individual filaments. This paper proposes a novel approach to extract filamentous objects by transforming an instance segmentation problem into a sequence modeling problem. Our approach first identifies filaments’ tip points, and we segment each instance by tracing them from each tip with a sequential encoder-decoder framework. The proposed method simulates the process of humans extracting filaments: pick a tip and trace the filament. As few datasets contain instance labels of filaments, we first generate synthetic filament datasets for training and evaluation. Then, we collected a dataset of 15 microscopic images of microtubules with instance labels for evaluation. Our proposed method can alleviate the data shortage problem since our proposed model can be trained with synthetic data and achieve state-of-art results when directly evaluated on the microtubule dataset and P. rubescens dataset. We also demonstrate our approaches’ capabilities in extracting short and thick elongated objects by evaluating on the C. elegans dataset. Our method achieves a comparable result compared to the state-of-art method with faster processing time. Our code is available at https://github.com/VimsLab/DRIFT.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, H., Qi, X., Yu, L., Heng, P.A.: DCAN: deep contour-aware networks for accurate gland segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2487–2496 (2016)
Chen, L., Strauch, M., Merhof, D.: Instance segmentation of biomedical images with an object-aware embedding learned with local constraints. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 451–459. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_50
Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., Sun, J.: Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7103–7112 (2018)
Dai, J., He, K., Li, Y., Ren, S., Sun, J.: Instance-sensitive fully convolutional networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 534–549. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_32
De Brabandere, B., Neven, D., Van Gool, L.: Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551 (2017)
Gao, N., et al.: SSAP: single-shot instance segmentation with affinity pyramid. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 642–651 (2019)
Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Simultaneous detection and segmentation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 297–312. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_20
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988. IEEE (2017)
Hirsch, P., Mais, L., Kainmueller, D.: PatchPerPix for instance segmentation. arXiv preprint arXiv:2001.07626 (2020)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Januszewski, M., et al.: High-precision automated reconstruction of neurons with flood-filling networks. Nat. Methods 15(8), 605–610 (2018)
Januszewski, M., Maitin-Shepard, J., Li, P., Kornfeld, J., Denk, W., Jain, V.: Flood-filling networks. arXiv preprint arXiv:1611.00421 (2016)
Ke, L., Tai, Y.W., Tang, C.K.: Deep occlusion-aware instance segmentation with overlapping bilayers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4019–4028 (2021)
Kulikov, V., Lempitsky, V.: Instance segmentation of biological images using harmonic embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3843–3851 (2020)
Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection (2017)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Liu, Y., Kolagunda, A., Treible, W., Nedo, A., Caplan, J., Kambhamettu, C.: Intersection to overpass: instance segmentation on filamentous structures with an orientation-aware neural network and terminus pairing algorithm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 125–133 (2019)
Liu, Y., et al.: Densely connected stacked U-network for filament segmentation in microscopy images. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11134, pp. 403–411. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_30
Liu, Y., et al.: Affinity derivation and graph merge for instance segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 708–724. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_42
Ljosa, V., Sokolnicki, K.L., Carpenter, A.E.: Annotated high-throughput microscopy image sets for validation. Nat. Methods 9(7), 637–637 (2012)
Novotny, D., Albanie, S., Larlus, D., Vedaldi, A.: Semi-convolutional operators for instance segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 89–105. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_6
Paszke, A., et al.: Automatic differentiation in PyTorch (2017)
Ren, M., Zemel, R.S.: End-to-end instance segmentation with recurrent attention. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 21–26 (2017)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Salvador, A., et al.: Recurrent neural networks for semantic instance segmentation. arXiv preprint arXiv:1712.00617 (2017)
Xu, T., et al.: SOAX: a software for quantification of 3D biopolymer networks. Sci. Rep. 5, 9081 (2015)
Yi, J., et al.: Multi-scale cell instance segmentation with keypoint graph based bounding boxes. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 369–377. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_41
Zeder, M., Van den Wyngaert, S., Köster, O., Felder, K.M., Pernthaler, J.: Automated quantification and sizing of unbranched filamentous cyanobacteria by model-based object-oriented image analysis. Appl. Environ. Microbiol. 76(5), 1615–1622 (2010)
Zhang, Z., Nishimura, Y., Kanchanawong, P.: Extracting microtubule networks from superresolution single-molecule localization microscopy data. Mol. Biol. Cell 28(2), 333–345 (2017)
Acknowledgement
This work was supported by the National Institute of General Medical Sciences grant (R01 GM097587) from the National Institutes of Health. Microscopy equipment was acquired with an NIH-shared instrumentation grant (S10 OD016361) and, access was supported by the NIH-NIGMS (P20 GM103446 and P20 GM139760) and the State of Delaware.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, Y., Peng, S., Caplan, J., Kambhamettu, C. (2023). Pick and Trace: Instance Segmentation for Filamentous Objects with a Recurrent Neural Network. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14227. Springer, Cham. https://doi.org/10.1007/978-3-031-43993-3_61
Download citation
DOI: https://doi.org/10.1007/978-3-031-43993-3_61
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43992-6
Online ISBN: 978-3-031-43993-3
eBook Packages: Computer ScienceComputer Science (R0)
