Abstract
Modern-day display systems demand high-quality rendering. However, rendering at higher resolution requires a large number of data samples and is computationally expensive. Recent advances in deep learning-based image and video super-resolution techniques motivate us to investigate such networks for high fidelity upscaling of frames rendered at a lower resolution to a higher resolution. While our work focuses on super-resolution of medical volume visualization performed with direct volume rendering, it is also applicable for volume visualization with other rendering techniques. We propose a learning-based technique where our proposed system uses color information along with other supplementary features gathered from our volume renderer to learn efficient upscaling of a low resolution rendering to a higher resolution space. Furthermore, to improve temporal stability, we also implement the temporal reprojection technique for accumulating history samples in volumetric rendering. Our method allows high-quality reconstruction of images from highly aliased input as shown in Fig. 1.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anagun, Y., Isik, S., Seke, E.: Srlibrary: comparing different loss functions for super-resolution over various convolutional architectures. J. Vis. Commun. Image Represent. 61, 178–187 (2019). https://doi.org/10.1016/j.jvcir.2019.03.027
Chan, K.C.K., Wang, X., Yu, K., Dong, C., Loy, C.C.: BasicVSR: the search for essential components in video super-resolution and beyond. CoRR abs/2012.02181 (2020). https://arxiv.org/abs/2012.02181
Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1st International Conference on Image Processing, vol. 2, pp. 168–172 (1994). https://doi.org/10.1109/ICIP.1994.413553
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). https://doi.org/10.1109/TPAMI.2015.2439281
Edelsten, A., Jukarainen, P., Patney, A.: Truly next-gen: adding deep learning to games and graphics. In: NVIDIA Sponsored Sessions (Game Dev Conference) (2019)
Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 27. Curran Associates, Inc (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE CVPR, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hofmann, N., Martschinke, J., Engel, K., Stamminger, M.: Neural denoising for path tracing of medical volumetric data. Proc. ACM Comput. Graph. Interact. Tech. 3(2), 1–18 (2020). https://doi.org/10.1145/3406181
Isobe, T., Jia, X., Gu, S., Li, S., Wang, S., Tian, Q.: Video super-resolution with recurrent structure-detail network. ArXiv abs/2008.00455 (2020)
Jo, Y., Oh, S.W., Kang, J., Kim, S.J.: Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In: IEEE CVPR, pp. 3224–3232 (2018). https://doi.org/10.1109/CVPR.2018.00340
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE CVPR, pp. 1646–1654 (2016). https://doi.org/10.1109/CVPR.2016.182
Kim, T.H., Sajjadi, M.S.M., Hirsch, M., Schölkopf, B.: Spatio-temporal transformer network for video restoration. In: ECCV (2018)
Koskela, M., et al.: Blockwise multi-order feature regression for real-time path-tracing reconstruction. ACM Trans. Graph. 38(5), 1–14 (2019). https://doi.org/10.1145/3269978
Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 105–114 (07 2017). https://doi.org/10.1109/CVPR.2017.19
Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. In: IEEE CVPR Workshops, pp. 1132–1140 (2017). https://doi.org/10.1109/CVPRW.2017.151
Lottes, T.: TSSAA temporal super sampling AA (2011). https://timothylottes.blogspot.com/2011/04/tssaa-temporal-super-sampling-aa.html
Pedersen, L.J.F.P.: Temporal reprojection anti-aliasing in inside (2018). https://s3.amazonaws.com/arena-attachments/655504/c5c71c5507f0f8bf344252958254fb7d.pdf?1468341463
Sardana, A.: Accelerating inference up to 6x faster in pytorch with torch-tensorrt (2021). https://developer.nvidia.com/blog/accelerating-inference-up-to-6x-faster-in-pytorch-with-torch-tensorrt/
Weiss, S., Chu, M., Thuerey, N., Westermann, R.: Volumetric isosurface rendering with deep learning-based super-resolution. IEEE Trans. Vis. Comput. Graph. 27(6), 3064–3078 (2021). https://doi.org/10.1109/TVCG.2019.2956697
Xiao, L., Nouri, S., Chapman, M., Fix, A., Lanman, D., Kaplanyan, A.: Neural supersampling for real-time rendering. ACM Trans. Graph. 39(4), 142:1–142:2 (2020). https://doi.org/10.1145/3386569.3392376
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: IEEE CVPR, pp. 2472–2481 (2018). https://doi.org/10.1109/CVPR.2018.00262
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Devkota, S., Pattanaik, S. (2022). Deep Learning Based Super-Resolution for Medical Volume Visualization with Direct Volume Rendering. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13598. Springer, Cham. https://doi.org/10.1007/978-3-031-20713-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-20713-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20712-9
Online ISBN: 978-3-031-20713-6
eBook Packages: Computer ScienceComputer Science (R0)