An integrated framework for the interaction and 3D visualization of cultural heritage | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

An integrated framework for the interaction and 3D visualization of cultural heritage

  • 1161: Multimedia Alternate Realities
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this study, the aim is to design and develop a 3D acquisition, visualization, and interaction framework to preserve cultural heritage and provide new ways to enable museum visitors and cultural audiences to virtually interact with cultural objects. Indeed, cultural assets are nowadays at higher risk and most cultural institutions prohibit visitors from physically manipulating their collections. The main motivation behind our framework is to enable end-user interaction with high valuable cultural objects while addressing cost-effectiveness concerns as well as minimizing the time required to digitize and generate 3D models of cultural heritage objects. The design idea of our framework is to allow interaction with the protected assets’ 3D representation using a real-world 3D screen equipped with a depth sensor namely the leap motion controller. Our framework is an end-to-end solution that optimizes all the stages of the 3D acquisition, pre-processing, visualization, and interaction pipeline while providing contributions to its stages. It achieves good quality results thanks to the use of machine learning in the acquisition and modeling stages. Indeed, we adapted a prior preprocessing work that performs super-resolution and motion interpolation on the acquired data. The preprocessed data is then used for the generation of the 3D models using photogrammetry, which optimizes the quality of the resulting 3D models. The created 3D models are then adapted for the visualization and interaction stages. A novel visualization and interaction paradigm is introduced to enable a real-world experience for museum visitors through a 3D screen called “the Looking Glass”. The interaction with the 3D content is achieved through a motion sensor used to design our new interaction component of the framework. We propose two new interaction systems suitable for various user profiles focusing on their experience in dealing with motion sensors. The end-to-end framework tested in a museum environment was evaluated by cultural heritage curators and multimedia experts and found to provide an alternate reality tool for asset exhibition and a cost-effective alternative for asset exchange between cultural institutions. For the evaluation, we compared the end-user experience of our framework using various setups where users are visualizing the content through 2D screens and through the Looking glass while enabling and disabling motion interaction. The results of the evaluation suggest that the looking glass paired with the Leap motion sensor using our framework as a backend enables an alternate reality experience for museum visitors and new ways of interacting with cultural content, sharing of cultural knowledge, cultural education, and much more.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Agus M, Bettio F, Giachetti A, Gobbetti E, Guitián JAI, Marton F, Nilsson J, Pintore G (2009) An interactive 3D medical visualization system based on a light field display. Visual Comput 25(9):883–893

    Article  Google Scholar 

  2. Bachmann D, Weichert F, Rinkenauer G (2018) Review of three-dimensional human-computer interaction with focus on the leap motion controller. Sensors 18(7):2194

    Article  Google Scholar 

  3. Barbieri L, Bruno F, Muzzupappa M (2017) Virtual museum system evaluation through user studies. J Cult Herit 26:101–108

    Article  Google Scholar 

  4. Bastanlar Y, Grammalidis N, Zabulis X, Yilmaz E, Yardimci Y, Triantafyllidis G (2008) 3D reconstruction for a cultural heritage virtual tour system. The international archives of the photogrammetry, remote sensing and spatial information sciences, Beijing 37, pp 1023–1028

  5. Belhi A, Bouras A, Foufou S (2017) Digitization and preservation of cultural heritage: the CEPROQHA approach. In: Software, Knowledge, Information Management and Applications (SKIMA), 2017 11th International Conference on. IEEE, pp 1–7

  6. Belhi A, Bouras A, Foufou S (2018) Towards a hierarchical multitask classification framework for cultural heritage. In: 2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), IEEE, pp 1–7

  7. Belhi A, Bouras A, Alfaqheri T, Aondoakaa AS, Sadka AH (2019) Investigating 3D holoscopic visual content upsampling using super-resolution for cultural heritage digitization. Sig Process Image Commun 75:188–198

    Article  Google Scholar 

  8. Belhi A, Al-Ali AK, Bouras A, Foufou S, Yu X, Zhang H (2020) Investigating low-delay deep learning-based cultural image reconstruction. J Real-Time Image Proc 17(6):1911–1926

    Article  Google Scholar 

  9. Belhi A, Bouras A, Al-Ali AK, Foufou S (2020) A machine learning framework for enhancing digital experiences in cultural heritage. J Enterp Inf Manag:1–13

  10. Belhi A, Gasmi H, Bouras A, Alfaqheri T, Aondoakaa AS, Sadka AH, Foufou S (2020) Machine learning and digital heritage: the CEPROQHA project perspective. In: Fourth international congress on information and communication technology. Springer, pp 363–374

  11. Boher P, Leroux T, Bignon T, Collomb-Patton V (2012) Optical characterization of different types of 3D displays. In: Advances in display technologies II. International Society for Optics and Photonics, p 82800A

  12. Buyuksalih I, Bayburt S, Buyuksalih G, Baskaraca A, Karim H, Rahman AA (2017) 3D modelling and visualization based on the Unity Game Engine–Advantages and Challenges. ISPRS Annals of the Photogrammetry. Remote Sens Spat Inform Sci 4:161

    Google Scholar 

  13. De la Barré R, Jurk S (2011) Improvements of aid free 3D presentation. In: Electronic displays conference, Nuremberg

  14. Diaper D, Stanton N (2003) The handbook of task analysis for human-computer interaction. CRC Press

  15. Drossis G, Birliraki C, Stephanidis C (2018) Interaction with immersive cultural heritage environments using virtual reality technologies. In: International conference on human-computer interaction. Springer, pp 177–183

  16. Factory LG (2021) Looking Glass Factory. https://lookingglassfactory.com/. Accessed 13 Aug 2021

  17. Freire LL, Arezes PM, Campos JC (2012) A literature review about usability evaluation methods for e-learning platforms. Work 41(Supplement 1):1038–1044

    Article  Google Scholar 

  18. Ghaoui C (2005) Encyclopedia of human computer interaction. IGI Global

  19. Gonizzi Barsanti S, Caruso G, Micoli L, Covarrubias Rodriguez M, Guidi G (2015) 3D visualization of cultural heritage artefacts with virtual reality devices. In: 25th International CIPA Symposium 2015, vol 5W7. Copernicus Gesellschaft mbH, pp 165–172

  20. Gray WD, Salzman MC (1998) Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum–Comput Interact 13(3):203–261

    Article  Google Scholar 

  21. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334

    Article  Google Scholar 

  22. Jacko JA (2012) Human computer interaction handbook: fundamentals, evolving technologies, and emerging applications. CRC press

  23. Jiang H, Sun D, Jampani V, Yang M-H, Learned-Miller E, Kautz J (2018) Super slomo: high quality estimation of multiple intermediate frames for video interpolation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9000–9008

  24. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z (2017) Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint

  25. Liang W (2019) Scene art design based on human-computer interaction and multimedia information system: an interactive perspective. Multimed Tools Appl 78(4):4767–4785

    Article  Google Scholar 

  26. Matterandform.net The Matter and, Scanner F https://matterandform.net/. Accessed 29 Apr 2020

  27. Mustafa AW, Ismail AF (2018) 3D virtual pottery environment using hand gesture interaction. In: Proceedings International conference on virtual systems and multimedia, pp 1–6

  28. Pavlidis G, Koutsoudis A, Arnaoutoglou F, Tsioukas V, Chamzas C (2007) Methods for 3D digitization of cultural heritage. J Cult Herit 8(1):93–98

    Article  Google Scholar 

  29. Pieraccini M, Guidi G, Atzeni C (2001) 3D digitizing of cultural heritage. J Cult Herit 2(1):63–70

    Article  Google Scholar 

  30. Preece J, Rogers Y, Sharp H, Benyon D, Holland S, Carey T (1994) Human-computer interaction. Addison-Wesley Longman Ltd.

  31. Rojas-Sola JI, Castro-Garcia M, del Carranza-Canadas P M (2011) Content management system incorporated in a virtual museum hosting. J Cult Herit 12(1):74–81

  32. Rubin J, Chisnell D (2008) Handbook of usability testing: how to plan, design and conduct effective tests. Wiley

  33. Scopigno R, Callieri M, Cignoni P, Corsini M, Dellepiane M, Ponchio F, Ranzuglia G (2011) 3D models for cultural heritage: beyond plain visualization. Computer 7:48–55

    Article  Google Scholar 

  34. Singh G (2014) CultLab3D: digitizing cultural heritage. IEEE Comput Graph Appl 34(3):4–5

    Article  Google Scholar 

  35. Spring AP, Peters C, Minns T (2010) Using mid-range laser scanners to digitize cultural-heritage sites. IEEE Comput Graph Appl 30(3):15–19. https://doi.org/10.1109/MCG.2010.62

    Article  Google Scholar 

  36. Ssemugabi S, De Villiers (2007) R A comparative study of two usability evaluation methods using a web-based e-learning application. In: Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, pp 132–142

  37. Tomar S (2006) Converting video formats with FFmpeg. Linux J 2006(146):10

    Google Scholar 

  38. Vosinakis S, Koutsabasis P, Makris D, Sagia E (2016) A kinesthetic approach to digital heritage using leap motion: the Cycladic sculpture application. In: 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), IEEE, pp 1–8

  39. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393

    Article  Google Scholar 

  40. Yanez-Gomez R, Cascado-Caballero D, Sevillano J-L (2017) Academic methods for usability evaluation of serious games: a systematic review. Multimed Tools Appl 76(4):5755–5784

    Article  Google Scholar 

  41. Yang L, Dong H, Alelaiwi A, El Saddik A (2016) See in 3D: state of the art of 3D display technologies. Multimed Tools Appl 75(24):17121–17155

    Article  Google Scholar 

Download references

Acknowledgements

This publication was made possible by NPRP grant 9-181-1-036 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors (www.ceproqha.qa).

The authors would also like to thank Mr. Marc Pelletreau, the MIA Multimedia team, the Art Curators and the management staff of the Museum of Islamic art, Doha Qatar for their help and contribution in the data acquisition.

Author information

Authors and Affiliations

  1. Joaan Bin Jassim Academy for Defence Studies, Al Khor, Qatar

    Abdelhak Belhi

  2. Brunel University, London, UK

    Hosameldin Osman Ahmed, Taha Alfaqheri & Abdul H. Sadka

  3. CSE, Qatar University, Doha, Qatar

    Abdelaziz Bouras

  4. Department of Computer Science, The University of Sharjah, Sharjah, United Arab Emirates

    Sebti Foufou

Authors
  1. Abdelhak Belhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hosameldin Osman Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Taha Alfaqheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abdelaziz Bouras
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abdul H. Sadka
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sebti Foufou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdelhak Belhi.

Ethics declarations

Conflict of interest

 The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Belhi, A., Ahmed, H.O., Alfaqheri, T. et al. An integrated framework for the interaction and 3D visualization of cultural heritage. Multimed Tools Appl 83, 46653–46681 (2024). https://doi.org/10.1007/s11042-023-14341-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14341-0

Keywords

Search

Navigation