A benchmark dataset and baseline model for co-salient object detection within RGB-D images | Multimedia Tools and Applications Skip to main content

Advertisement

Springer Nature Link
Log in

A benchmark dataset and baseline model for co-salient object detection within RGB-D images

  • 1190: Depth-Related Processing and Applications in Visual Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Within-image co-salient object detection (wCoSOD) identifies the common and salient objects within an image, which can benefit for many applications, such as reducing information redundancy, animation synthesis, and so on. Besides, the introduction of depth information that conforms to the stereo perception of human is also more conducive to accurately detecting salient objects. Thus, in this paper, we focus on a new task from the perspective of the benchmark dataset and baseline model, i.e., within-image co-salient object detection in RGB-D images. To bridge the gap the new task and algorithm verification, we first collect a new dataset containing 240 RGB-D images and the corresponding pixel-wise ground truth. Then, we propose an unsupervised method for within-image co-salient object detection in RGB-D images. Under the constraint of depth information, our model decomposes the within-image co-salient object detection task into two parts: determining the salient object proposals; combining the similarity constraint and cluster-based constraint between different proposals to locate the co-salient object and generate the final result. The experimental results on the collected dataset demonstrate that our method achieves competitive performance both qualitatively and quantitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Achanta R, Hemami S, Estrada F, Susstrunk, S (2009) Frequency-tuned salient region detection. In: Proceedings of the IEEE computer vision and pattern recognition, pp. 1597–1604

  2. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11):2274–2282

    Article  Google Scholar 

  3. Arthur D, Vassilvitskii S (2007) K-means++: The advantages of careful seeding. In: Proceedings of the eighteenth annual ACM-SIAM symposium on discrete algorithms, pp. 1027–1035

  4. Borji A, Cheng MM, Jiang H, Li J (2015) Salient object detection: A benchmark. IEEE Transactions on Image Processing 24(12):5706–5722

    Article  MathSciNet  Google Scholar 

  5. Cao X, Tao Z, Zhang B, Fu H, Feng W (2014) Self-adaptively weighted co-saliency detection via rank constraint. IEEE Transactions on Image Processing 23(9):4175–4186

    MathSciNet  MATH  Google Scholar 

  6. Cheng M, Mitra NJ, Huang X, Torr PHS, Hu S (2015) Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(3):569–582

    Article  Google Scholar 

  7. Cheng Y, Fu H, Wei X, Xiao J, Cao X (2014) Depth enhanced saliency detection method. In: Proceedings of international conference on internet multimedia computing and service, pp. 23–27

  8. Cong R, Lei J, Fu H, Cheng MM, Lin W, Huang Q (2019) Review of visual saliency detection with comprehensive information. IEEE Transactions on Circuits and Systems for Video Technology 29(10):2941–2959

    Article  Google Scholar 

  9. Cong R, Lei J, Fu H, Huang Q, Cao X, Hou C (2017) Co-saliency detection for RGB-D images based on multi-constraint feature matching and cross label propagation. IEEE Transactions on Image Processing 27(2):568–579

    Article  Google Scholar 

  10. Cong R, Lei J, Zhang C, Huang Q, Cao X, Hou C (2016) Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion. IEEE Signal Processing Letters 23(6):819–823

    Article  Google Scholar 

  11. Fan DP, Cheng MM, Liu Y, Li T, Borji A (2017) Structure-measure: A new way to evaluate foreground maps. In: Proceedings of the IEEE computer Vvsion and pattern recognition, pp. 4548–4557

  12. Fan DP, Lin Z, Zhang Z, Zhu M, Cheng MM (2020) Rethinking RGB-D salient object detection: models, data sets, and large-scale benchmarks. IEEE Transactions on Neural Networks and Learning Systems 32(5):2075–2089

    Article  Google Scholar 

  13. Fang Y, Zeng K, Wang Z, Lin W, Fang Z, Lin CW (2014) Objective quality assessment for image retargeting based on structural similarity. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 4(1):95–105

    Article  Google Scholar 

  14. Fu H, Cao X, Tu Z (2013) Cluster-based co-saliency detection. IEEE Transactions on Image Processing 22(10):3766–3778

    Article  MathSciNet  Google Scholar 

  15. Gao Y, Shi M, Tao D, Xu C (2015) Database saliency for fast image retrieval. IEEE Transactions on Multimedia 17(3):359–369

    Article  Google Scholar 

  16. Godard C, Mac Aodha O, Firman M, Brostow GJ (2019) Digging into self-supervised monocular depth estimation. In: IEEE international conference on computer vision, pp. 3828–3838

  17. Guo H, Zheng K, Fan X, Yu H, Wang S (2019) Visual attention consistency under image transforms for multi-label image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 729–739

  18. Han S, Vasconcelos N (2006) Image compression using object-based regions of interest. In: IEEE international conference on image processing, pp. 3097–3100

  19. He X, Gould S (2014) An exemplar-based CRF for multi-instance object segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 296–303

  20. Jeong DJ, Hwang I, Cho NI (2018) Co-salient object detection based on deep saliency networks and seed propagation over an integrated graph. IEEE Transactions on Image Processing 27(12):5866–5879

    Article  MathSciNet  Google Scholar 

  21. Ju R, Ge L, Geng W, Ren T, Wu G (2014) Depth saliency based on anisotropic center-surround difference. In: IEEE International conference on image processing (ICIP), pp. 1115–1119

  22. Kim H, Kim Y, Sim JY, Kim CS (2015) Spatiotemporal saliency detection for video sequences based on random walk with restart. IEEE Transactions on Image Processing 24(8):2552–64

    Article  MathSciNet  Google Scholar 

  23. Li C, Cong R, Piao Y, Xu Q, Loy CC (2020) RGB-D salient object detection with cross-modality modulation and selection. In: Proceedings of the european conference on computer vision, pp. 225–241

  24. Lin TY, Maire M, Belongie S, Hays J, Zitnick CL (2014) Microsoft coco: Common objects in context. In: Proceedings of the european conference on computer vision, pp. 740–755

  25. Liu N, Zhang N, Shao L, Han J (2020) Learning selective mutual attention and contrast for RGB-D saliency detection. arXiv:2010.05537

  26. Niu Y, Geng Y, Li X, Liu F (2012) Leveraging stereopsis for saliency analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 454–461

  27. Peng H, Li B, Xiong W, Hu W, Ji R (2014) RGB-D salient object detection: a benchmark and algorithms. In: Proceedings of the IEEE european conference on computer vision, pp. 92–109

  28. Piao Y, Ji W, Li J, Zhang M, Lu H (2019) Depth-induced multi-scale recurrent attention network for saliency detection. In: Proceedings of the IEEE international conference on computer vision, pp. 7254–7263

  29. Piao Y, Rong Z, Zhang M, Ren W, Lu H (2020) A2dele: Adaptive and attentive depth distiller for efficient RGB-D salient object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9060–9069

  30. Qu L, He S, Zhang J, Tian J, Tang Y, Yang Q (2017) RGBD salient object detection via deep fusion. IEEE Transactions on Image Processing 26(5):2274–2285

    Article  MathSciNet  Google Scholar 

  31. Song H, Liu Z, Xie Y, Wu L, Huang M (2016) RGBD co-saliency detection via bagging-based clustering. IEEE Signal Processing Letters 23(12):1722–1726

    Article  Google Scholar 

  32. Song S, Yu H, Miao Z, Guo D, Ke W, Ma C, Wang S (2019) An easy-to-hard learning strategy for within-image co-saliency detection. Neurocomputing 358(17):166–176

    Article  Google Scholar 

  33. Uijlings JR, Van De Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. International Journal of Computer Vision 104(2):154–171

    Article  Google Scholar 

  34. Xi T, Zhao W, Wang H, Lin W (2017) Salient object detection with spatiotemporal background priors for video. IEEE Transactions on Image Processing 26(7):3425–3436

    Article  MathSciNet  Google Scholar 

  35. Xu X, Wan L, Liu X, Wong TT, Wang L, Leung CS (2008) Animating animal motion from still. In: Proceedings of the ACM transactions on graphics, pp. 1–8

  36. Ye L, Liu Z, Li L, Shen L, Bai C, Wang Y (2017) Salient object segmentation via effective integration of saliency and objectness. IEEE Transactions on Multimedia 19(8):1742–1756

    Article  Google Scholar 

  37. Yu H, Zheng K, Fang J, Guo H, Wang S (2018) Co-saliency detection within a single image. In: Proceedings of the AAAI conference on artificial intelligence, pp. 7509–7516

  38. Yu H, Zheng K, Fang J, Guo H, Wang S (2020) A new method and benchmark for detecting co-saliency within a single image. IEEE Transactions on Multimedia 22(12):3051–3063

    Article  Google Scholar 

  39. Zhu C, Li G (2017) A three-pathway psychobiological framework of salient object detection using stereoscopic technology. In: Proceedings of the IEEE international conference on computer vision workshops, pp. 3008–3014

  40. Zhu C, Li G, Wang W, Wang R (2017) An innovative salient object detection using center-dark channel prior. In: Proceedings of the IEEE international conference on computer vision workshops, pp. 1509–1515

Download references

Acknowledgements

This work was supported by the Beijing Nova Program under Grant Z201100006820016.

Author information

Authors and Affiliations

  1. Institute of Information Science, Beijing Jiaotong University, 100044, Beijing, China

    Ning Yang, Chen Zhang, Yumo Zhang & Haowei Yang

  2. The Beijing Key Laboratory of Advanced Information Science and Network Technology, 100044, Beijing, China

    Ning Yang, Chen Zhang, Yumo Zhang & Haowei Yang

  3. School of Computer Science and Technology, Tiangong University, 300387, Tianjin, China

    Ling Du

Authors
  1. Ning Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Chen Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yumo Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Haowei Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Ling Du
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ling Du.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, N., Zhang, C., Zhang, Y. et al. A benchmark dataset and baseline model for co-salient object detection within RGB-D images. Multimed Tools Appl 81, 35831–35842 (2022). https://doi.org/10.1007/s11042-021-11555-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11555-y

Keywords