Abstract
We present a motion analysis framework for anomaly detection in the context of health monitoring in carp and koi ponds. We study recent advances in computer vision and deep learning for an automated motion assessment based on video streams and propose a specifically designed image acquisition system.
It turned out that the accurate detection and recognition of individual fish objects remains a difficult topic for scenarios with dense homogeneous groups and frequently occurring occlusions. We thus tackled this challenging field of aquatic scene understanding by applying deep state-of-the-art architectures from the areas of object detection, semantic segmentation and instance segmentation as a first step for further extraction of motion information. We used dense optical flow as an estimation of collective fish movements and restricted the motion extraction according to the resulting masks from the previous image segmentation step.
We introduce a heatmap visualization as an intermediate representation of the spatio-temporal distribution of fish locations. We derived several metrics to quantify changes in motion patterns and apparent location hotspots as indicators of anomalous behavior. Based on this representation, we were able to identify different classes of behavior like feeding times, shoaling or night’s rest as well as anomalous group behavior like mobbing or hiding in an experimental setup.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The recording of the fish video dataset, as well as the manual fish observations, were done by co-author Arndt Christian Hofmann as part of the data acquisition process for his unpublished PhD thesis at Friedrich-Loeffler-Institut.
References
Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online, realtime tracking. In: IEEE International Conference on Image Processing, pp. 3464–3468 (2016)
Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection, semantic segmentation. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 580–587 (2014)
Haralick, R.M., Shanmugam, K., Dinstein, I.: Textural features for image classification. IEEE Trans. Syst. Man Cybern. SMC 3(6), 610–621 (1973)
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
He, X., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 770–778 (2016)
Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2015)
Jiang, Y., Fang, J., Li, Z., Yue, J., Wang, Z., Li, D.: Automatic tracking of swimming koi using a particle filter with a center-surrounding cue. Mathe. Comput. Model. 58(3), 859–867 (2013)
Kuznetsova, A., et al.: The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00982 (2018)
Li, X., Wei, Z., Huang, L., Nie, J., Zhang, W., Wang, L.: Real-time underwater fish tracking based on adaptive multi-appearance model. In: 25th IEEE International Conference on Image Processing, pp. 2710–2714 (2018)
Li, X., Shang, M., Qin, H., Chen, L.: Fast accurate fish detection, recognition of underwater images with Fast R-CNN. In: OCEANS 2015 MTS, pp. 1–5. IEEE, Washington (2015)
Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 936–944 (2017)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Niu, B., Li, G., Peng, F., Wu, J., Zhang, L., Li, Z.: Survey of fish behavior analysis by computer vision. J. Aquac. Res. Dev. 9(5) (2018)
Peng, C., Zhang, X., Yu, G., Luo, G., Sun, J.: Large kernel matters - improve semantic segmentation by global convolutional network. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 1743–1751 (2017)
Raman, R., Prakash, C., Marappan, M., Pawar, N.A.: Environmental stress mediated diseases of fish: an overview. Adv. Fish Res. 5, 141–158 (2013)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 779–788 (2016)
Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 6517–6525 (2017)
Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv:1804.02767 (2018)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)
Saberioon, M., Gholizadeh, A., Cisar, P., Pautsina, A., Urban, J.: Application of machine vision systems in aquaculture with emphasis on fish: state-of-the-art, key issues. Rev. Aquac. 9(4), 369–387 (2017)
Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision, Pattern Recognition, pp. 3431–3440 (2015)
Shevchenko, V., Eerola, T., Kaarna, A.: Fish detection from low visibility underwater videos. In: 24th International Conference on Pattern Recognition, pp. 1971–1976 (2018)
Snieszko, S.F.: The effects of environmental stress on outbreaks of infectious diseases of fishes. J. Fish Biol. 6(2), 197–208 (1974)
Sung, M., Yu, S., Girdhar, Y.: Vision based real-time fish detection using convolutional neural network. In: OCEANS 2017, Aberdeen, pp. 1–6 (2017)
Terayama, K., Habe, H., Sakagami, M.: Multiple fish tracking with an NACA airfoil model for collective behavior analysis. IPSJ Trans. Comput. Vis. Appl. 8(1), 1–7 (2016)
Terayama, K., Hioki, H., Sakagami, M.: A measurement method for speed distribution of collective motion with optical flow, its application to estimation of rotation curve. In: IEEE International Symposium on Multimedia, pp. 32–39 (2014)
Terayama, K., Hongo, K., Habe, H., Sakagami, M.: Appearance-based multiple fish tracking for collective motion analysis. In: 3rd IAPR Asian Conference on Pattern Recognition, pp. 361–365 (2015)
Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: IEEE International Conference on Computer Vision, pp. 1385–1392 (2013)
Yu, X., Hou, X., Lu, H., Yu, X., Fan, L., Liu, Y.: Anomaly detection of fish school behavior based on features statistical, optical flow methods. Trans. Chin. Soc. Agric. Eng. 30(2), 162–168 (2014)
Zhao, X., Yan, S., Gao, Q.: An algorithm for tracking multiple fish based on biological water quality monitoring. IEEE Access 7, 15018–15026 (2019)
Zion, B.: The use of computer vision technologies in aquaculture - a review. Comput. Electron. Agric. 88, 125–132 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Hümmer, C. et al. (2019). Analysis of Motion Patterns in Video Streams for Automatic Health Monitoring in Koi Ponds. In: Lee, C., Su, Z., Sugimoto, A. (eds) Image and Video Technology. PSIVT 2019. Lecture Notes in Computer Science(), vol 11854. Springer, Cham. https://doi.org/10.1007/978-3-030-34879-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-34879-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34878-6
Online ISBN: 978-3-030-34879-3
eBook Packages: Computer ScienceComputer Science (R0)