A comprehensive survey on object detection in Visual Art: taxonomy and challenge | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

A comprehensive survey on object detection in Visual Art: taxonomy and challenge

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Cultural heritage data plays a key role in the understanding of past human history and culture, enriches the present and prepares the future. A wealth of information is buried in artwork images that can be extracted via digitization and analysis. While a huge number of methods exists, a deep review of the literature concerning object detection in visual art is still lacking. In this study, after reviewing several related papers, a comprehensive review is presented, including (i) an overview of major computer vision applications for visual art, (ii) a presentation of previous related surveys, (iii) a comprehensive overview of relevant object detection methods for artistic images. Considering the studied object detection methods, we propose a new taxonomy based on the supervision learning degree, the adopted framework, the adopted methodology (classical or deep-learning based method), the type of object to detect and the depictive style of the painting images. Then the several challenges for object detection in artistic images are described and the proposed ways of solving some encountered problems are discussed. In addition, available artwork datasets and metrics used for object detection performance evaluation are presented. Finally, we provide potential future directions to improve object detection performances in paintings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

All data analysed during this study are available at locations cited in the reference section.

References

  1. Achlioptas P, Ovsjanikov M, Haydarov K, et al (2021) ArtEmis: Affective language for visual art. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 11569–11579. https://doi.org/10.1109/cvpr46437.2021.01140

  2. Al-Yasiri D, Obaid AJ (2018) A new approach for object detection, recognition and retrieving in painting images. Journal of Advance Research in Dynamic and Control System 10(2):2345–2359

    Google Scholar 

  3. Amura A, Tonazzini A, Salerno E et al (2020) Color segmentation and neural networks for automatic graphic relief of the state of conservation of artworks. Cultura e Scienza del Colore-Color Culture and Science 12(02):07–15

    Google Scholar 

  4. Arora RS, Elgammal A (2012) Towards automated classification of fine-art painting style: A comparative study. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), IEEE, pp 3541–3544

  5. Artistic-faces dataset (2019). https://faculty.runi.ac.il/arik/site/foa/artistic-faces-dataset.asp, Accessed: 2023-03-06

  6. Bai Y, Guo Y, Wei J, et al (2020) Fake generated painting detection via frequency analysis. 2020 IEEE International Conference on Image Processing (ICIP) pp 1256–1260

  7. Barnard K, Duygulu P, Forsyth D (2001) Clustering art. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, IEEE, pp II–II

  8. Bekkouch IEI, Constantin ND, Eyharabide V, et al (2021) Adversarial domain adaptation for medieval instrument recognition. In: Lecture Notes in Networks and Systems. Springer International Publishing, pp 674–687. https://doi.org/10.1007/978-3-030-82196-8_50

  9. Bengamra. S, Mzoughi. O, Bigand. A, et al (2023) Towards explainability in using deep learning for face detection in paintings. In: Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods - ICPRAM,, INSTICC. SciTePress, pp 832–841. https://doi.org/10.5220/0011670300003411

  10. Bengamra S, Mzoughi. O, Bigand. A, et al (2021) New challenges of face detection in paintings based on deep learning. In: Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP,, INSTICC. SciTePress, pp 311–320. https://doi.org/10.5220/0010243703110320

  11. Bilen H, Vedaldi A (2016) Weakly supervised deep detection networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2846–2854

  12. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3d faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques SIGGRAPH ’99, pp 187–194

  13. Bochkovskiy A, Wang CY, Liao HYM (2020) Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934

  14. Bourdev L, Malik J (2009) Poselets: Body part detectors trained using 3d human pose annotations. In: 2009 IEEE 12th International Conference on Computer Vision. IEEE, pp 1365–1372. https://doi.org/10.1109/iccv.2009.5459303

  15. Brachmann A, Redies C (2017) Computational and experimental approaches to visual aesthetics. Front Comput Neurosci 11. https://doi.org/10.3389/fncom.2017.00102

  16. Bradski G (2000) The opencv library. Dr Dobb’s Journal: Software Tools for the Professional Programmer 25(11):120–123

    Google Scholar 

  17. Bredow T, Alder N, Büßemeyer M (2021) Image retrieval. In: Deep learning for computer vision in the art domain: proceedings of the master seminar on practical introduction to deep learning for computer vision, HPI WS 20/21, Universitätsverlag Potsdam, p 59

  18. Brochu E, Cora VM, De Freitas N (2010) A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599

  19. Brueghel dataset (2019). https://imagine.enpc.fr/~shenx/ArtMiner/, Accessed: 2023-03-06

  20. Buchana P, Cazan I, Diaz-Granados M, et al (2016) Simultaneous forgery identification and localization in paintings using advanced correlation filters. 2016 IEEE International Conference on Image Processing (ICIP) pp 146–150

  21. Cai Z, Vasconcelos N (2018) Cascade r-cnn: Delving into high quality object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6154–6162

  22. Cai H, Wu Q, Corradi T, et al (2015a) The cross-depiction problem: Computer vision algorithms for recognising objects in artwork and in photographs. arXiv preprint arXiv:1505.00110

  23. Cai H, Wu Q, Hall P (2015b) Beyond photo-domain object recognition: Benchmarks for the cross-depiction problem. In: Proceedings of the IEEE international conference on computer vision workshops, pp 1–6. https://doi.org/10.1109/iccvw.2015.19

  24. Carneiro G, da Silva NP, Bue AD, et al (2012) Artistic image classification: An analysis on the PRINTART database. In: Computer Vision – ECCV 2012. Springer Berlin Heidelberg, pp 143–157. https://doi.org/10.1007/978-3-642-33765-9_11

  25. Caspa dataset (2018). https://people.cs.pitt.edu/~chris/artistic_objects/, Accessed: 2023-03-08

  26. Castellano G, Vessio G (2022) A deep learning approach to clustering visual arts. Int J Comput Vision 130(11):2590–2605

    Article  Google Scholar 

  27. Castellano G, Lella E, Vessio G (2021) Visual link retrieval and knowledge discovery in painting datasets. Multimedia Tools and Applications 80(5):6599–6616

    Article  Google Scholar 

  28. Castellano G, Vessio G (2020) Towards a tool for visual link retrieval and knowledge discovery in painting datasets. In: Italian research conference on digital libraries, Springer, pp 105–110

  29. Castellano G, Vessio G (2021) A brief overview of deep learning approaches to pattern extraction and recognition in paintings and drawings. In: International Conference on Pattern Recognition, Springer, pp 487–501

  30. Cetinic E (2021a) Iconographic image captioning for artworks. In: International Conference on Pattern Recognition, Springer, pp 502–516

  31. Cetinic E (2021b) Towards generating and evaluating iconographic image captions of artworks. Journal of Imaging 7(8):123

  32. Cetinic E, She J (2022) Understanding and creating art with AI: Review and outlook. ACM Trans Multimed Comput Commun Appl 18(2):1–22. https://doi.org/10.1145/3475799

    Article  Google Scholar 

  33. Cetinic E, Lipic T, Grgic S (2018) Fine-tuning convolutional neural networks for fine art classification. Expert Syst Appl 114:107–118. https://doi.org/10.1016/j.eswa.2018.07.026

    Article  Google Scholar 

  34. Cetinic E, Lipic T, Grgic S (2019) A deep learning perspective on beauty, sentiment, and remembrance of art. IEEE Access 7:73694–73710. https://doi.org/10.1109/access.2019.2921101

    Article  Google Scholar 

  35. Cetinic E, Grgic S (2013) Automated painter recognition based on image feature extraction. In: Proceedings ELMAR-2013, IEEE, pp 19–22

  36. Chen X, Xu C, Yang X et al (2019) Gated-gan: Adversarial gated networks for multi-collection style transfer. IEEE Trans Image Process 28:546–560

    Article  MathSciNet  Google Scholar 

  37. Chen W, Huang H, Peng S et al (2021) Yolo-face: a real-time face detector. Vis Comput 37:805–813

    Article  Google Scholar 

  38. Chen Y, Li W, Sakaridis C, et al (2018) Domain adaptive faster r-cnn for object detection in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3339–3348

  39. Chu WT, Motomura H, Tsumura N et al (2019) [invited papers] a survey on multimedia artworks analysis and attractiveness computing in multimedia. ITE Transactions on Media Technology and Applications 7(2):60–67

    Article  Google Scholar 

  40. Crowley EJ, Zisserman A (2016) The art of detection. In: European conference on computer vision, Springer, pp 721–737

  41. Crowley E, Zisserman A (2014) The state of the art: Object retrieval in paintings using discriminative regions. In: Proceedings of the British Machine Vision Conference 2014. British Machine Vision Association. https://doi.org/10.5244/c.28.38

  42. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), IEEE Computer Society, pp 886–893. https://doi.org/10.1109/cvpr.2005.177

  43. Del Chiaro R, Bagdanov AD, Del Bimbo A (2019) Webly-supervised zero-shot learning for artwork instance recognition. Pattern Recogn Lett 128:420–426

    Article  Google Scholar 

  44. Dominguez V, Messina P, Parra D, et al (2017) Comparing neural and attractiveness-based visual features for artwork recommendation. In: Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems. ACM, pp 55–59. https://doi.org/10.1145/3125486.3125495

  45. Elgammal AM, Saleh B (2015) Quantifying creativity in art networks. CoRR abs/1506.00711

  46. Elgammal A, Liu B, Kim D, et al (2018) The shape of art history in the eyes of the machine. Proceedings of the AAAI Conference on Artificial Intelligence 32(1). https://doi.org/10.1609/aaai.v32i1.11894

  47. Falomir Z, Museros L, Sanz I et al (2018) Categorizing paintings in art styles based on qualitative color descriptors, quantitative global features and machine learning (QArt-learn). Expert Syst Appl 97:83–94. https://doi.org/10.1016/j.eswa.2017.11.056

    Article  Google Scholar 

  48. Felzenszwalb PF, Girshick RB, McAllester D et al (2009) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645

    Article  Google Scholar 

  49. Fiorucci M, Khoroshiltseva M, Pontil M et al (2020) Machine learning for cultural heritage: A survey. Pattern Recogn Lett 133:102–108. https://doi.org/10.1016/j.patrec.2020.02.017

    Article  Google Scholar 

  50. Florea C, Badea M, Florea L, et al (2017) Domain transfer for delving into deep networks capacity to de-abstract art. In: Scandinavian Conference on Image Analysis, Springer, pp 337–349

  51. Foka A (2021) Computer vision applications for art history: Reflections and paradigms for future research. In: Proceedings of EVA London 2021. BCS Learning & Development, pp 73–80. https://doi.org/10.14236/ewic/eva2021.12

  52. Folego G, Gomes O, Rocha A (2016) From impressionism to expressionism: Automatically identifying van gogh’s paintings. 2016 IEEE International Conference on Image Processing (ICIP) pp 141–145

  53. Frank SJ (2021) State of the art: This convolutional neural network can tell you whether a painting is a fake. IEEE Spectr 58(9):26–31. https://doi.org/10.1109/MSPEC.2021.9531029

    Article  Google Scholar 

  54. Fujiyoshi H, Hirakawa T, Yamashita T (2019) Deep learning-based image recognition for autonomous driving. IATSS research 43(4):244–252

    Article  Google Scholar 

  55. Ganin Y, Lempitsky V (2015) Unsupervised domain adaptation by backpropagation. In: International conference on machine learning, PMLR, pp 1180–1189

  56. Gao X, Tian Y, Qi Z (2020) Rpd-gan: Learning to draw realistic paintings with generative adversarial network. IEEE Trans Image Process 29:8706–8720

    Article  Google Scholar 

  57. Garcia N, Vogiatzis G (2019) How to read paintings: Semantic art understanding with multi-modal retrieval. In: Lecture Notes in Computer Science. Springer International Publishing, pp 676–691. https://doi.org/10.1007/978-3-030-11012-3_52

  58. Gatys LA, Ecker AS, Bethge M (2016a) Image style transfer using convolutional neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp 2414–2423

  59. Gatys LA, Ecker AS, Bethge M (2016b) Image style transfer using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2414–2423

  60. Geirhos R, Rubisch P, Michaelis C, et al (2019) Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In: International Conference on Learning Representations,

  61. Geman D, Geman S, Hallonquist N et al (2015) Visual turing test for computer vision systems. Proc Natl Acad Sci 112(12):3618–3623

    Article  Google Scholar 

  62. Ginosar S, Haas D, Brown T et al (2015) Detecting people in cubist art. AI Matters 1(3):16–18. https://doi.org/10.1145/2735392.2735398

    Article  Google Scholar 

  63. Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448

  64. Girshick R, Donahue J, Darrell T, et al (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  65. Goenaga MA (2020) A critique of contemporary artificial intelligence art: Who is edmond de belamy? AusArt 8(1):51–66. https://doi.org/10.1387/ausart.21490

    Article  Google Scholar 

  66. Gonthier N, Ladjal S, Gousseau Y (2022) Multiple instance learning on deep features for weakly supervised object detection with extreme domain shifts. Comput Vis Image Underst 214(103):299

    Google Scholar 

  67. Gonthier N, Gousseau Y, Ladjal S, et al (2019) Weakly supervised object detection in artworks. In: Lecture Notes in Computer Science. Springer International Publishing, pp 692–709. https://doi.org/10.1007/978-3-030-11012-3_53

  68. Goodfellow IJ, Pouget-Abadie J, Mirza M, et al (2014) Generative adversarial networks. arXiv preprint arXiv:1406.2661

  69. Gultepe E, Conturo TE, Makrehchi M (2018) Predicting and grouping digitized paintings by style using unsupervised feature learning. J Cult Herit 31:13–23

    Article  Google Scholar 

  70. Gupta S, Kumar M, Garg A (2019) Improved object recognition results using sift and orb feature detector. Multimedia Tools and Applications 78:34157–34171

  71. Hayn-Leichsenring GU, Lehmann T, Redies C (2017) Subjective ratings of beauty and aesthetics: Correlations with statistical image properties in western oil paintings. i-Perception 8(3):204166951771,547. https://doi.org/10.1177/2041669517715474

  72. Hearst MA, Dumais ST, Osuna E et al (1998) Support vector machines. IEEE Intelligent Systems and their applications 13(4):18–28

    Article  Google Scholar 

  73. He K, Gkioxari G, Dollar P, et al (2017) Mask r-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, pp 2961–2969. https://doi.org/10.1109/iccv.2017.322

  74. Hosain MK, Harun-Ur-Rashid, Taher TB, et al (2020) Genre recognition of artworks using convolutional neural network. In: 2020 23rd International Conference on Computer and Information Technology (ICCIT). IEEE, pp 1–5. https://doi.org/10.1109/iccit51783.2020.9392688

  75. Hu X (2018) Tensorflow implementation of cyclegan. https://github.com/xhujoy/CycleGAN-tensorflow

  76. Hu M, Wang H, Wang X et al (2019) Video facial emotion recognition based on local enhanced motion history image and cnn-ctslstm networks. J Vis Commun Image Represent 59:176–185

    Article  Google Scholar 

  77. Ibrahim BIE, Eyharabide V, Page VL et al (2022) Few-shot object detection: Application to medieval musicological studies. Journal of Imaging 8(2):18. https://doi.org/10.3390/jimaging8020018

    Article  Google Scholar 

  78. Iconart dataset (2018). https://wsoda.telecom-paristech.fr/downloads/dataset/, Accessed: 2023-03-08

  79. Iliadis LA, Nikolaidis S, Sarigiannidis P et al (2021) Artwork style recognition using vision transformers and mlp mixer. Technologies 10(1):2

    Article  Google Scholar 

  80. Inoue N, Furuta R, Yamasaki T, et al (2018) Cross-domain weakly-supervised object detection through progressive domain adaptation. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, pp 5001–5009. https://doi.org/10.1109/cvpr.2018.00525

  81. Jeon HJ, Jung S, Choi YS, et al (2020) Object detection in artworks using data augmentation. In: 2020 International Conference on Information and Communication Technology Convergence (ICTC). IEEE, pp 1312–1314. https://doi.org/10.1109/ictc49870.2020.9289321

  82. Johnson MK, Stork DG, Biswas S, et al (2008) Inferring illumination direction estimated from disparate sources in paintings: an investigation into jan vermeer’s girl with a pearl earring. In: Computer image analysis in the study of art, International Society for Optics and Photonics, p 68100I

  83. Junger A, Metzenthin E, Wullenweber P (2021) Object detection. In: Deep learning for computer vision in the art domain: proceedings of the master seminar on practical introduction to deep learning for computer vision, HPI WS 20/21, Universitätsverlag Potsdam, p 33

  84. Kadish D, Risi S, Lovlie AS (2021) Improving object detection in art images using only style transfer. In: 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, pp 1–8. https://doi.org/10.1109/ijcnn52387.2021.9534264

  85. Kantorov V, Oquab M, Cho M, et al (2016) Contextlocnet: Context-aware deep network models for weakly supervised localization. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, Springer, pp 350–365

  86. Keren D (2002) Painter identification using local features and naive bayes. In: Object recognition supported by user interaction for service robots. IEEE Comput. Soc, pp 474–477. https://doi.org/10.1109/icpr.2002.1048341

  87. Khalili A, Bouchachia H (2021) An information theory approach to aesthetic assessment of visual patterns. Entropy 23(2):153. https://doi.org/10.3390/e23020153

    Article  Google Scholar 

  88. Kotenseki dataset (2019). http://codh.rois.ac.jp/pmjt/, Accessed: 2023-03-14

  89. Kumar KK, Venkateswara Reddy H (2022) Crime activities prediction system in video surveillance by an optimized deep learning framework. Concurrency and Computation: Practice and Experience 34(11):e6852

    Article  Google Scholar 

  90. Lang S, Ommer B (2018) Attesting similarity: Supporting the organization and study of art image collections with computer vision. Digital Scholarship in the Humanities 33(4):845–856. https://doi.org/10.1093/llc/fqy006

    Article  Google Scholar 

  91. Lecoutre A, Negrevergne B, Yger F (2017) Recognizing art style automatically in painting with deep learning. In: Zhang ML, Noh YK (eds) Proceedings of the Ninth Asian Conference on Machine Learning, Proceedings of Machine Learning Research, vol 77. PMLR, Yonsei University, Seoul, Republic of Korea, pp 327–342

  92. Lin TY, Goyal P, Girshick R, et al (2017) Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, pp 2980–2988. https://doi.org/10.1109/iccv.2017.324

  93. Lin Y (2020) Sentiment analysis of painting based on deep learning. In: International Conference on Application of Intelligent Systems in Multi-modal Information Analytics, Springer, pp 651–655. https://doi.org/10.1007/978-3-030-51556-0_96

  94. Liu Y (2021) Improved generative adversarial network and its application in image oil painting style transfer. Image Vis Comput 105(104):087

    Google Scholar 

  95. Liu W, Anguelov D, Erhan D, et al (2016) Ssd: Single shot multibox detector. In: European conference on computer vision, Springer, pp 21–37

  96. Liu Z, Lin Y, Cao Y, et al (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10012–10022

  97. Lu Y, Guo C, Dai X et al (2022) Data-efficient image captioning of fine art paintings via virtual-real semantic alignment training. Neurocomputing 490:163–180

    Article  Google Scholar 

  98. Madhu P, Kosti R, Mührenberg L, et al (2019) Recognizing characters in art history using deep learning. In: Proceedings of the 1st Workshop on Structuring and Understanding of Multimedia heritAge Contents, pp 15–22

  99. MAFD-150 dataset (2018). https://github.com/andeeptoor/mafd-150, Accessed: 2023-03-06

  100. Ma D, Gao F, Bai Y, et al (2017) From part to whole: Who is behind the painting? In: Proceedings of the 25th ACM international conference on Multimedia. ACM, pp 1174–1182. https://doi.org/10.1145/3123266.3123325

  101. Maji B, Swain M, Mustaqeem (2022) Advanced fusion-based speech emotion recognition system using a dual-attention mechanism with conv-caps and bi-gru features. Electronics 11(9). https://doi.org/10.3390/electronics11091328

  102. Mao H, Cheung M, She J (2017) Deepart: Learning joint representations of visual arts. In: Proceedings of the 25th ACM international conference on Multimedia. ACM, pp 1183–1191. https://doi.org/10.1145/3123266.3123405

  103. Marinescu MC, Reshetnikov A, López JM (2020) Improving object detection in paintings based on time contexts. In: 2020 International Conference on Data Mining Workshops (ICDMW), IEEE, pp 926–932

  104. Mensink T, Van Gemert J (2014) The rijksmuseum challenge: Museum-centered visual recognition. In: Proceedings of International Conference on Multimedia Retrieval, pp 451–454

  105. Mermet A, Kitamoto A, Suzuki C, et al (2020) Face detection on pre-modern japanese artworks using r-CNN and image patching for semi-automatic annotation. In: Proceedings of the 2nd Workshop on Structuring and Understanding of Multimedia heritAge Contents. ACM, pp 23–31. https://doi.org/10.1145/3423323.3423412

  106. Messina P, Dominquez V, Parra D, et al (2017) Exploring content-based artwork recommendation with metadata and visual features. ArXiv abs/1706.05786

  107. Mohammad SM, Kiritchenko S (2018) Wikiart emotions: An annotated dataset of emotions evoked by art. In: Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018)

  108. Moutafidou A, Fudos I, Adamopoulos G, et al (2018) Reconstruction and visualization of cultural heritage artwork objects. In: International Conference on Transdisciplinary Multispectral Modeling and Cooperation for the Preservation of Cultural Heritage, Springer, pp 141–149

  109. Mustaqeem, Kwon S (2020) Clstm: Deep feature-based speech emotion recognition using the hierarchical convlstm network. Mathematics 8(12). https://doi.org/10.3390/math8122133

  110. Mustaqeem, Kwon S (2021a) 1d-cnn: Speech emotion recognition system using a stacked network with dilated cnn features. Cmc-computers Materials & Continua 67:4039–4059

  111. Mustaqeem, Kwon S (2021b) Att-net: Enhanced emotion recognition system using lightweight self-attention module. Applied Soft Computing 102:107101. https://doi.org/10.1016/j.asoc.2021.107101

  112. Mustaqeem Kwon S (2021) Optimal feature selection based speech emotion recognition using two-stream deep convolutional neural network. Int J Intell Syst 36:5116–5135

    Article  Google Scholar 

  113. Mustaqeem Ishaq M, Kwon S (2022) A cnn-assisted deep echo state network using multiple time-scale dynamic learning reservoirs for generating short-term solar energy forecasting. Sustainable Energy Technol Assess 52:102275. https://doi.org/10.1016/j.seta.2022.102275

    Article  Google Scholar 

  114. Mzoughi O, Bigand A, Renaud C (2018) Face detection in painting using deep convolutional neural networks. In: Advanced Concepts for Intelligent Vision Systems. Springer International Publishing, pp 333–341. https://doi.org/10.1007/978-3-030-01449-0_28

  115. Nasir IM, Raza M, Shah JH, Wang SH, Tariq U, Khan MA (2022) Harednet: A deep learning based architecture for autonomous video surveillance by recognizing human actions. Comput Electr Eng 99:107805. https://doi.org/10.1016/j.compeleceng.2022.107805

    Article  Google Scholar 

  116. Paintings dataset (2014). https://www.robots.ox.ac.uk/~vgg/data/paintings/, Accessed: 2023-03-06

  117. Pasqualino G, Furnari A, Farinella GM (2021a) Unsupervised domain adaptation for object detection in cultural sites. In: 2020 25th International Conference on Pattern Recognition (ICPR). IEEE. https://doi.org/10.1109/icpr48806.2021.9412661

  118. Pasqualino G, Furnari A, Signorello G, et al (2021b) An unsupervised domain adaptation scheme for single-stage artwork recognition in cultural sites. Image and Vision Computing 107:104098

  119. Peleshko D, Soroka K (2013) Research of usage of haar-like features and adaboost algorithm in viola-jones method of object detection. In: 2013 12th International Conference on the Experience of Designing and Application of CAD Systems in Microelectronics (CADSM), IEEE, pp 284–286

  120. PeopleArt dataset (2014). https://github.com/BathVisArtData/PeopleArt, Accessed: 2023-03-06

  121. PhotoArt50 dataset (2016). https://github.com/BathVisArtData/PhotoArt50, Accessed: 2023-03-14

  122. Polatkan G, Jafarpour S, Brasoveanu A, et al (2009) Detection of forgery in paintings using supervised learning. 2009 16th IEEE International Conference on Image Processing (ICIP) pp 2921–2924

  123. Ranjgar B, Azar MK, Sadeghi-Niaraki A et al (2019) A novel method for emotion extraction from paintings based on luscher’s psychological color test: Case study iranian-islamic paintings. IEEE Access 7:120857–120871. https://doi.org/10.1109/access.2019.2936896

    Article  Google Scholar 

  124. Redmon J, Divvala S, Girshick R, et al (2016) You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 779–788. https://doi.org/10.1109/cvpr.2016.91

  125. Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7263–7271

  126. Ren S, He K, Girshick R et al (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28:91–99

    Google Scholar 

  127. Rodrigues JB, Ferreira AVM, Maia IMO, et al (2018) Image processing of artworks for construction of 3d models accessible to the visually impaired. In: International Conference on Applied Human Factors and Ergonomics, Springer, pp 243–253

  128. Rombach R, Blattmann A, Lorenz D, et al (2022) High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://github.com/CompVis/latent-diffusion, https://arxiv.org/abs/2112.10752

  129. Sabatelli M, Kestemont M, Daelemans W, et al (2019) Deep transfer learning for art classification problems. In: Lecture Notes in Computer Science. Springer International Publishing, pp 631–646. https://doi.org/10.1007/978-3-030-11012-3_48

  130. Saito K, Ushiku Y, Harada T, et al (2019) Strong-weak distribution alignment for adaptive object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6956–6965

  131. Saleh B, Elgammal A (2015) Large-scale classification of fine-art paintings: Learning the right metric on the right feature. ArXiv abs/1505.00855

  132. Sargentis GF, Dimitriadis P, Koutsoyiannis D (2020) Aesthetical issues of leonardo da vinci’s and pablo picasso’s paintings with stochastic evaluation. Heritage 3(2):283–305. https://doi.org/10.3390/heritage3020017

    Article  Google Scholar 

  133. Sarı C, Salah AA, Akdag Salah AA (2019) Automatic detection and visualization of garment color in western portrait paintings. Digital Scholarship in the Humanities 34(Supplement_1):i156–i171

  134. Schlecht J, Carqué B, Ommer B (2011) Detecting gestures in medieval images. In: 2011 18th IEEE International Conference on Image Processing, IEEE, pp 1285–1288

  135. Seguin B, Striolo C, Kaplan F, et al (2016) Visual link retrieval in a database of paintings. In: European conference on computer vision, Springer, pp 753–767

  136. Shen X, Efros AA, Aubry M (2019) Discovering visual patterns in art collections with spatially-consistent feature learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9278–9287

  137. Sheng S, Moens MF (2019) Generating captions for images of ancient artworks. In: Proceedings of the 27th ACM International Conference on Multimedia, pp 2478–2486

  138. Sindel A, Maier A, Christlein V (2023) Artfacepoints: High-resolution facial landmark detection in paintings and prints. In: Karlinsky L, Michaeli T, Nishino K (eds) Computer Vision - ECCV 2022 Workshops. Springer Nature Switzerland, Cham, pp 298–313

  139. Sirirattanapol C, Matsui Y, Satoh S, et al (2017) Deep image retrieval applied on kotenseki ancient japanese literature. In: 2017 IEEE International Symposium on Multimedia (ISM). IEEE, pp 495–499. https://doi.org/10.1109/ism.2017.98

  140. Smirnov S, Eguizabal A (2018) Deep learning for object detection in fine-art paintings. In: 2018 Metrology for Archaeology and Cultural Heritage (MetroArchaeo), IEEE, pp 45–49. https://doi.org/10.1109/MetroArchaeo43810.2018.9089828

  141. Song Y, Ren S, Lu Y, et al (2022) Deep learning-based automatic segmentation of images in cardiac radiography: a promising challenge. Computer Methods and Programs in Biomedicine p 106821

  142. Spehr M, Wallraven C, Fleming RW (2009) Image statistics for clustering paintings according to their visual appearance. Computational Aesthetics 2009: Eurographics Workshop on Computational Aesthetics in Graphics. Visualization and Imaging, Eurographics, pp 57–64

    Google Scholar 

  143. Srinivasan R, Rudolph C, Roy-Chowdhury AK (2015) Computerized face recognition in renaissance portrait art: A quantitative measure for identifying uncertain subjects in ancient portraits. IEEE Signal Process Mag 32(4):85–94. https://doi.org/10.1109/msp.2015.2410783

    Article  Google Scholar 

  144. Srinivasan R, Roy-Chowdhury A, Rudolph C, et al (2013) Recognizing the royals: Leveraging computerized face recognition for identifying subjects in ancient artworks. In: Proceedings of the 21st ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, MM ’13, p 581–584. https://doi.org/10.1145/2502081.2502153

  145. Stork DG (2011) Computer analysis of lighting style in fine art: steps towards inter-artist studies. In: Computer Vision and Image Analysis of Art II, vol 7869. SPIE, p 786903. https://doi.org/10.1117/12.873190

  146. Stork D (2009) Computer vision and computer graphics analysis of paintings and drawings: An introduction to the literature. International Conference on Computer Analysis of Images and Patterns. Springer, CAIP, pp 9–24

    Chapter  Google Scholar 

  147. Stork D, Johnson MK (2006) Computer vision, image analysis, and master art: Part 2. IEEE Multimedia 13:12–17

    Article  Google Scholar 

  148. Strezoski G, Worring M (2017) Omniart: Multi-task deep learning for artistic data analysis. ArXiv abs/1708.00684

  149. Surapaneni S, Syed S, Lee LY (2020) Exploring themes and bias in art using machine learning image analysis. In: 2020 Systems and Information Engineering Design Symposium (SIEDS), IEEE, pp 1–6

  150. Tan WR, Chan CS, Aguirre HE, et al (2016) Ceci n’est pas une pipe: A deep convolutional network for fine-art paintings classification. In: 2016 IEEE international conference on image processing (ICIP), IEEE, pp 3703–3707. https://doi.org/10.1109/ICIP.2016.7533051

  151. Tan WR, Chan CS, Aguirre HE, et al (2017) Artgan: Artwork synthesis with conditional categorical gans. 2017 IEEE International Conference on Image Processing (ICIP) pp 3760–3764

  152. Tan W, Wang J, Wang Y et al (2018) Cnn models for classifying emotions evoked by paintings. Technical Report, SVL Lab, Stanford University, USA, Tech. rep

    Google Scholar 

  153. Tian Y, Suzuki C, Clanuwat T, et al (2020) Kaokore: A pre-modern japanese art facial expression dataset. arXiv preprint arXiv:2002.08595

  154. Tyler CW, Smith WAP, Stork DG (2012) In search of Leonardo: computer-based facial image analysis of Renaissance artworks for identifying Leonardo as subject. In: Rogowitz BE, Pappas TN, de Ridder H (eds) Human Vision and Electronic Imaging XVII, International Society for Optics and Photonics, vol 8291. SPIE, pp 407 – 413

  155. Van Noord N, Hendriks E, Postma E (2015) Toward discovery of the artist’s style: Learning to recognize artists by their artworks. IEEE Signal Process Mag 32(4):46–54

    Article  Google Scholar 

  156. Vedaldi A, Lenc K (2015) Matconvnet: Convolutional neural networks for matlab. In: Proceedings of the 23rd ACM international conference on Multimedia, pp 689–692

  157. Volpe Y, Furferi R, Governi L et al (2014) Computer-based methodologies for semi-automatic 3d model generation from paintings. International Journal of Computer Aided Engineering and Technology 6(1):88–112

    Article  Google Scholar 

  158. Wechsler H, Toor AS (2019) Modern art challenges face detection. Pattern Recogn Lett 126:3–10. https://doi.org/10.1016/j.patrec.2018.02.014

    Article  Google Scholar 

  159. Westlake N, Cai H, Hall P (2016) Detecting people in artwork with CNNs. In: Lecture Notes in Computer Science. Springer International Publishing, pp 825–841. https://doi.org/10.1007/978-3-319-46604-0_57

  160. Wikiart: visual art encyclopedia (2010). https://www.wikiart.org/, Accessed: 2023-03-06

  161. Wikicommons (2004). https://commons.wikimedia.org/wiki/Main Page, Accessed: 2023-03-08

  162. Winarno E, Hadikurniawati W, Nirwanto AA, et al (2018) Multi-view faces detection using viola-jones method. In: Journal of Physics: Conference Series, IOP Publishing, p 012068

  163. Winston JJ, Hemanth DJ, Angelopoulou A, et al (2022) Hybrid deep convolutional neural models for iris image recognition. Multimedia Tools and Applications pp 1–23

  164. Wu Q, Cai H, Hall P (2014) Learning graphs to model visual objects across different depictive styles. In: European Conference on Computer Vision, Springer, pp 313–328. https://doi.org/10.1007/978-3-319-10584-0_21

  165. Xie J, Girshick R, Farhadi A (2016) Unsupervised deep embedding for clustering analysis. In: International conference on machine learning, PMLR, pp 478–487

  166. Yakar M, Doğan Y (2018) Gis and three-dimensional modeling for cultural heritages. International Journal of Engineering and Geosciences 3(2):50–55

    Article  Google Scholar 

  167. Yang Z (2021) Classification of picture art style based on VGGNET. J Phys: Conf Ser 1774(1):012043. https://doi.org/10.1088/1742-6596/1774/1/012043

    Article  Google Scholar 

  168. Yang H, Min K (2019) Classification of basic artistic media based on a deep convolutional approach. The Visual Computer 36(3):559–578. https://doi.org/10.1007/s00371-019-01641-6

    Article  MathSciNet  Google Scholar 

  169. Yang H, Min K (2019b) A deep approach for classifying artistic media from artworks. KSII Trans Internet Inf Syst 13:2558–2573

  170. Yaniv J, Newman Y, Shamir A (2019) The face of art: landmark detection and geometric style in portraits. ACM Transactions on graphics (TOG) 38(4):1–15

    Article  Google Scholar 

  171. Yanulevskaya V, Uijlings J, Bruni E, et al (2012) In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. In: Proceedings of the 20th ACM international conference on multimedia, pp 349–358

  172. Yi R, Liu YJ, Lai YK, et al (2019) Apdrawinggan: Generating artistic portrait drawings from face photos with hierarchical gans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10743–10752

  173. Yoloface (2019). https://github.com/sthanhng/yoloface, Accessed: 2023-03-08

  174. Yolo-v5 (2023). https://github.com/ultralytics/yolov5, Accessed: 2023-03-08

  175. Young-Min K (2019) Feature visualization in comic artist classification using deep neural networks. Journal of Big Data 6(1):1–18. https://doi.org/10.1186/s40537-019-0222-3

    Article  Google Scholar 

  176. Zhang C, Lei K, Jia J, et al (2018a) Ai painting: an aesthetic painting generation system. In: Proceedings of the 26th ACM international conference on Multimedia, pp 1231–1233

  177. Zhang H, Li Q, Sun Z, et al (2018b) Combining data-driven and model-driven methods for robust facial landmark detection. IEEE Transactions on Information Forensics and Security 13(10):2409–2422

  178. Zhao L, Shang M, Gao F et al (2020) Representation learning of image composition for aesthetic prediction. Comput Vis Image Underst 199:103024. https://doi.org/10.1016/j.cviu.2020.103024

    Article  Google Scholar 

  179. Zhu JY, Park T, Isola P, et al (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, pp 2242–2251. https://doi.org/10.1109/iccv.2017.244

  180. Zhu Y, Yan WQ (2022) Traffic sign recognition based on deep learning. Multimedia Tools and Applications 81(13):17779–17791

    Article  Google Scholar 

  181. Zujovic J, Gandy L, Friedman S, et al (2009) Classifying paintings by artistic genre: An analysis of features & classifiers. In: 2009 IEEE International Workshop on Multimedia Signal Processing. IEEE, pp 1–5. https://doi.org/10.1109/mmsp.2009.5293271

Download references

Author information

Authors and Affiliations

  1. LIMTIC laboratory, Higher Institute of Computer Science, University of Tunis El-Manar, 2, Rue Abou Raihan El Bayrouni, Ariana, 2080, Tunisia

    Siwar Bengamra & Ezzeddine Zagrouba

  2. LISIC laboratory, University of the Littoral Opal Coast (ULCO), 50, Rue Ferdinand Buisson, Calais, 62228, France

    Siwar Bengamra & André Bigand

  3. Department of Computer Sciences, Prince Sattam Bin Abdulaziz University, Al-Aflaj, 16733, Kingdom of Saudi Arabia

    Olfa Mzoughi

Authors
  1. Siwar Bengamra
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Olfa Mzoughi
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. André Bigand
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Ezzeddine Zagrouba
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Siwar Bengamra.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bengamra, S., Mzoughi, O., Bigand, A. et al. A comprehensive survey on object detection in Visual Art: taxonomy and challenge. Multimed Tools Appl 83, 14637–14670 (2024). https://doi.org/10.1007/s11042-023-15968-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15968-9

Keywords