{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,12,19]],"date-time":"2023-12-19T23:01:50Z","timestamp":1703026910867},"reference-count":25,"publisher":"MIT Press","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,12,1]]},"abstract":"Abstract<\/jats:title>\n Volumetric video recordings of storytellers, when experienced in immersive virtual reality, can elicit a sense of copresence between the user and the storyteller. Combining a volumetric storyteller with an appropriate virtual environment presents a compelling experience that can convey the story with a depth that is hard to achieve with traditional forms of media. Volumetric video production remains difficult, time-consuming, and expensive, often excluding cultural groups who would benefit most. The difficulty is partly due to ever-increasing levels of visual detail in computer graphics, and resulting hardware and software requirements. A high level of detail is not a requirement for convincing immersive experiences, and by reducing the level of detail, experiences can be produced and delivered using readily available, nonspecialized equipment. By reducing computational requirements in this way, storytelling scenes can be created ad hoc and experienced immediately\u2014this is what we are addressing with our approach. We present our portable real-time volumetric capture system, and our framework for using it to produce immersive storytelling experiences. The real-time capability of the system, and the low data rates resulting from lower levels of visual detail, allow us to stream volumetric video in real time to enrich experiences with embodiment (seeing oneself) and with copresence (seeing others). Our system has supported collaborative research with M\u0101ori partners with the aim of reconnecting the dispersed M\u0101ori population in Aotearoa, New Zealand to their ancestral land through immersive storytelling. We present our system in the context of this collaborative work.<\/jats:p>","DOI":"10.1162\/pres_a_00364","type":"journal-article","created":{"date-parts":[[2023,2,3]],"date-time":"2023-02-03T19:31:56Z","timestamp":1675452716000},"page":"5-29","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":2,"title":["Voxel-Based Immersive Mixed Reality: A Framework for Ad Hoc Immersive Storytelling"],"prefix":"10.1162","volume":"30","author":[{"given":"Stuart","family":"Duncan","sequence":"first","affiliation":[{"name":"Department of Information Science University of Otago Dunedin 9016, New Zealand"}]},{"given":"Noel","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Information Science University of Otago Dunedin 9016, New Zealand"}]},{"given":"Claudia","family":"Ott","sequence":"additional","affiliation":[{"name":"Department of Information Science University of Otago Dunedin 9016, New Zealand"}]},{"given":"Tobias","family":"Langlotz","sequence":"additional","affiliation":[{"name":"Department of Information Science University of Otago Dunedin 9016, New Zealand"}]},{"given":"Holger","family":"Regenbrecht","sequence":"additional","affiliation":[{"name":"Department of Information Science University of Otago Dunedin 9016, New Zealand"}]}],"member":"281","published-online":{"date-parts":[[2021,12,1]]},"reference":[{"key":"2023062114284403300_B1","volume-title":"Volume graphics","author":"Chen","year":"2012"},{"issue":"3","key":"2023062114284403300_B2","doi-asserted-by":"publisher","first-page":"467","DOI":"10.1007\/s00779-020-01431-1","article-title":"Point-cloud avatars to improve spatial communication in immersive collaborative virtual environments","volume":"25","author":"Gamelin","year":"2021","journal-title":"Personal and Ubiquitous Computing"},{"issue":"3","key":"2023062114284403300_B3","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1162\/pres.17.3.293","article-title":"Temporal and spatial variations in presence: Qualitative analysis of interviews from an experiment on breaks in presence","volume":"17","author":"Garau","year":"2008","journal-title":"Presence: Teleoperators and Virtual Environments"},{"key":"2023062114284403300_B4","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1145\/3458305.3459595","article-title":"Vrcomm: An end-to-end web system for real-time photorealistic social VR communication","author":"Gunkel","year":"2021","journal-title":"Proceedings of the 12th ACM Multimedia Systems Conference"},{"key":"2023062114284403300_B5","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1109\/VR.2009.4811006","article-title":"Exploring co-presence for next generation technical support","author":"Guven","year":"2009","journal-title":"2009 IEEE Virtual Reality Conference"},{"key":"2023062114284403300_B6","first-page":"189","article-title":"Social presence in two- and three-dimensional videoconferencing","author":"Hauber","year":"2005","journal-title":"Presence 2005: The 8th Annual International Workshop on Presence"},{"key":"2023062114284403300_B7","doi-asserted-by":"publisher","first-page":"109","DOI":"10.2312\/hpg.20161197","article-title":"GVDB: Raytracing sparse Voxel database structures on the GPU","author":"Hoetzlein","year":"2016","journal-title":"Proceedings of High Performance Graphics"},{"issue":"3","key":"2023062114284403300_B8","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1177\/027836499501400301","article-title":"Hand-eye calibration","volume":"14","author":"Horaud","year":"1995","journal-title":"The International Journal of Robotics Research"},{"key":"2023062114284403300_B9","doi-asserted-by":"crossref","first-page":"778","DOI":"10.1109\/ICRA.2012.6224647","article-title":"Real-time compression of point cloud streams","author":"Kammerl","year":"2012","journal-title":"2012 IEEE International Conference on Robotics and Automation"},{"issue":"4","key":"2023062114284403300_B10","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1162\/pres_a_00124","article-title":"The sense of embodiment in virtual reality","volume":"21","author":"Kilteni","year":"2012","journal-title":"Presence: Teleoperators and Virtual Environments"},{"key":"2023062114284403300_B11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3173574.3173703","article-title":"Remixed reality: Manipulating space and time in augmented reality","author":"Lindlbauer","year":"2018","journal-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems"},{"issue":"6","key":"2023062114284403300_B12","doi-asserted-by":"publisher","first-page":"248:1","DOI":"10.1145\/2816795.2818013","article-title":"SMPL: A skinned multi-person linear model","volume":"34","author":"Loper","year":"2015","journal-title":"ACM Transactions on Graphics"},{"key":"2023062114284403300_B13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/IVCNZ48456.2019.8961016","article-title":"Towards a M\u0101ori telepresence system","author":"Park","year":"2019","journal-title":"2019 International Conference on Image and Vision Computing New Zealand"},{"key":"2023062114284403300_B14","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1109\/VR51125.2022.00033","article-title":"Mixed reality co-design for indigenous culture preservation and continuation","author":"Park","year":"2022","journal-title":"2022 IEEE Conference on Virtual Reality and 3D User Interfaces"},{"issue":"05","key":"2023062114284403300_B15","doi-asserted-by":"publisher","DOI":"10.4236\/jsea.2019.125011","article-title":"Resolutions and network latencies concerning a voxel telepresence experience","volume":"12","author":"Park","year":"2019","journal-title":"Journal of Software Engineering and Applications"},{"key":"2023062114284403300_B16","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1109\/ISMAR.2017.26","article-title":"Mixed Voxel reality: Presence and embodiment in low fidelity, visually coherent, mixed reality environments","author":"Regenbrecht","year":"2017","journal-title":"2017 IEEE International Symposium on Mixed and Augmented Reality"},{"key":"2023062114284403300_B17","doi-asserted-by":"publisher","first-page":"68185","DOI":"10.1109\/ACCESS.2021.3076488","article-title":"Voxelvideos for entertainment, education, and training","volume":"9","author":"Regenbrecht","year":"2021","journal-title":"IEEE Access"},{"key":"2023062114284403300_B18","doi-asserted-by":"publisher","DOI":"10.3389\/fict.2019.00007","article-title":"Preaching voxels: An alternative approach to mixed reality","volume":"6","author":"Regenbrecht","year":"2019","journal-title":"Frontiers in ICT"},{"issue":"1","key":"2023062114284403300_B19","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1109\/MTS.2022.3147525","article-title":"Atea presence---Enabling virtual storytelling, presence, and tele-co-presence in an indigenous setting","volume":"41","author":"Regenbrecht","year":"2022","journal-title":"IEEE Technology and Society Magazine"},{"issue":"3","key":"2023062114284403300_B20","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/j.cognition.2004.04.003","article-title":"Illusion of sense of self-agency: Discrepancy between the predicted and actual sensory consequences of actions modulates the sense of self-agency, but not the sense of self-ownership","volume":"94","author":"Sato","year":"2005","journal-title":"Cognition"},{"issue":"5","key":"2023062114284403300_B21","doi-asserted-by":"publisher","first-page":"18","DOI":"10.5594\/JMI.2019.2906835","article-title":"Advanced volumetric capture and processing","volume":"128","author":"Schreer","year":"2019","journal-title":"SMPTE Motion Imaging Journal"},{"issue":"3","key":"2023062114284403300_B22","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1162\/105474601300343603","article-title":"The experience of presence: Factor analytic insights","volume":"10","author":"Schubert","year":"2001","journal-title":"Presence: Teleoperators and Virtual Environments"},{"key":"2023062114284403300_B23","first-page":"298","article-title":"Bundle adjustment\u2014A modern synthesis","author":"Triggs","year":"1999","journal-title":"International Workshop on Vision Algorithms"},{"issue":"4","key":"2023062114284403300_B24","doi-asserted-by":"publisher","first-page":"1643","DOI":"10.1109\/TVCG.2018.2794629","article-title":"The impact of avatar personalization and immersion on virtual body ownership, presence, and emotional response","volume":"24","author":"Waltemate","year":"2018","journal-title":"IEEE Transactions on Visualization and Computer Graphics"},{"key":"2023062114284403300_B25","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1016\/B978-0-32-391755-1.00027-4","article-title":"Volumetric video as a novel medium for creative storytelling","volume-title":"Immersive Video Technologies","author":"Young","year":"2023"}],"container-title":["PRESENCE: Virtual and Augmented Reality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/pvar\/article-pdf\/doi\/10.1162\/pres_a_00364\/2133926\/pres_a_00364.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/pvar\/article-pdf\/doi\/10.1162\/pres_a_00364\/2133926\/pres_a_00364.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,21]],"date-time":"2023-06-21T14:30:36Z","timestamp":1687357836000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/pvar\/article\/doi\/10.1162\/pres_a_00364\/114611\/Voxel-Based-Immersive-Mixed-Reality-A-Framework"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":25,"URL":"https:\/\/doi.org\/10.1162\/pres_a_00364","relation":{},"ISSN":["1531-3263"],"issn-type":[{"value":"1531-3263","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}