Abstract
Panoramic images and videos have become widely popular media formats. However, there are challenges in dealing with the lack of six degrees-of-freedom (6DOF) motion in panoramas. Recent advancements in novel view synthesis techniques have shown promising outcomes in indoor settings characterized by geometric structures. Nevertheless, the translation of these advancements to complex outdoor panorama, especially with single panorama as input, remains a formidable task. In this study, we propose a novel view synthesis pipeline that takes a single outdoor panorama as input. The method employs a dual-branch design that downsamples the input image to capture the global information of the complex outdoor scene and utilizes Multi-Sphere Images (MSI) for MSI-RGBA representation inference of the input panorama. To represent complex geometric shapes and multi-scale details, we introduce a high-resolution refinement branch to optimize the fine edges in the panorama, resulting in high-quality synthesized novel outdoor panorama. Our method has achieved significant performance improvements in single-image synthesis using the CARLA datasets, and it can be generalized to real outdoor panorama datasets. These endeavors contribute to advancing panoramic media towards a more comfortable immersive experience, ultimately enhancing the realism of immersive panoramic 6DOF roaming.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Attal, B., Ling, S., Gokaslan, A., Richardt, C., Tompkin, J.: Matryodshka: real-time 6dof video view synthesis using multi-sphere images. In: European Conference on Computer Vision, pp. 441–459. Springer (2020)
Boukerch, I., Takarli, B., Saidi, K., Karich, M., Meguenni, M.: Development of panoramic virtual tours system based on low cost devices. Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 43, 869–874 (2021)
Chen, X., Liang, H., Xu, H., Ren, S., Cai, H., Wang, Y.: Virtual view synthesis based on asymmetric bidirectional DIBR for 3d video and free viewpoint video. Appl. Sci. 10(5), 1562 (2020)
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)
Fehn, C.: Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Stereoscopic Displays and Virtual Reality Systems XI, vol. 5291, pp. 93–104. SPIE (2004)
Gu, S., Zhang, W., Wang, R.: Enhanced DIBR framework for free viewpoint video. In: 2021 7th International Conference on Computer and Communications (ICCC), pp. 911–916. IEEE (2021)
Guo, S., Hu, J., Zhou, K., Wang, J., Song, L., Xie, R., Zhang, W.: Real-time free viewpoint video synthesis system based on DIBR and a depth estimation network. IEEE Trans. Multimed. (2024)
Hannuksela, M.M., Wang, Y.K.: An overview of omnidirectional media format (OMAF). Proc. IEEE 109(9), 1590–1606 (2021)
Hore, A., Ziou, D.: Image quality metrics: PSNR versus SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)
Li, D., Zhang, Y., Häne, C., Tang, D., Varshney, A., Du, R.: Omnisyn: synthesizing 360 videos with wide-baseline panoramas. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 670–671. IEEE (2022)
Li, J., Feng, Z., She, Q., Ding, H., Wang, C., Lee, G.H.: Mine: Towards continuous depth MPI with nerf for novel view synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12578–12588 (2021)
Li, J., He, Y., Hu, Y., Han, Y., Wen, J.: Learning to compose 6-DoF omnidirectional videos using multi-sphere images. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 3298–3302. IEEE (2021)
Li, J., He, Y., Jiao, J., Hu, Y., Han, Y., Wen, J.: Extending 6-DoF VR experience via multi-sphere images interpolation. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4632–4640 (2021)
Li, Y., Yabuki, N., Fukuda, T.: Measuring visual walkability perception using panoramic street view images, virtual reality, and deep learning. Sustain. Urban Areas 86, 104140 (2022)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)
Morita, H., Hild, M., Miura, J., Shirai, Y.: Panoramic view-based navigation in outdoor environments based on support vector learning. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2302–2307. IEEE (2006)
Osman, A., Iskak, N.I., Wahab, N.A., Ibrahim, N.: Interactive virtual campus tour using panoramic video: a heuristic evaluation. J. Comput. Res. Innov. 5(4), 1–7 (2020)
Pan, Q., Zeng, Q., Zhuang, Y., Yin, Z., Hai, J., Chen, G., Liang, J., Xiao, H., Yang, S.: OJUMP: optimization for joint unicast-multicast panoramic VR live streaming system. Trans. Emerg. Telecommun. Technol. 35(1), e4924 (2024)
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in Pytorch (2017)
Pintore, G., Bettio, F., Agus, M., Gobbetti, E.: Deep scene synthesis of Atlanta-world interiors from a single omnidirectional image. IEEE Trans. Vis. Comput. Graph. (2023)
Tsai, H.H., Hou, X.Y., Chang, C.T., Tsai, C.Y., Yu, P.T., Roan, J.S., Chiou, K.C.: Interactive contents with 360\(^\circ \) panorama virtual reality for soil and water conservation outdoor classroom. In: 2020 International Symposium on Educational Technology (ISET), pp. 78–82. IEEE (2020)
Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 551–560 (2020)
Waidhofer, J., Gadgil, R., Dickson, A., Zollmann, S., Ventura, J.: Panosynthvr: toward light-weight 360-degree view synthesis from a single panoramic input. In: 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 584–592. IEEE (2022)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Wiles, O., Gkioxari, G., Szeliski, R., Johnson, J.: Synsin: end-to-end view synthesis from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7467–7477 (2020)
Xu, J., Zheng, J., Xu, Y., Tang, R., Gao, S.: Layout-guided novel view synthesis from a single indoor panorama. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16438–16447 (2021)
Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578–4587 (2021)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Zheng, D., Wei, L., Liu, Y., Wang, Y.: Metamars: 3dof+ roaming with panoramic stitching for Tianwen-1 mission. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images (2018). arXiv:1805.09817
Zink, M., Sitaraman, R., Nahrstedt, K.: Scalable 360 video stream delivery: challenges, solutions, and opportunities. Proc. IEEE 107(4), 639–650 (2019)
Acknowledgements
This work was supported by the National Natural Science Foundation of China (62277035,62332017) and by the Shandong Province Youth Entrepreneurship Technology Support Program for Higher Education Institutions(2022KJN028).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Luan, H., Wang, L., Luan, X., Gai, W., Yang, C. (2025). Immersive 6DOF Roaming with Novel View Synthesis from Single Outdoor Panorama. In: Lin, Z., et al. Pattern Recognition and Computer Vision. PRCV 2024. Lecture Notes in Computer Science, vol 15039. Springer, Singapore. https://doi.org/10.1007/978-981-97-8692-3_12
Download citation
DOI: https://doi.org/10.1007/978-981-97-8692-3_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-8691-6
Online ISBN: 978-981-97-8692-3
eBook Packages: Computer ScienceComputer Science (R0)