{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,27]],"date-time":"2025-03-27T18:57:19Z","timestamp":1743101839200},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2020,8,31]]},"abstract":"Due to higher resolutions and refresh rates, as well as more photorealistic effects, real-time rendering has become increasingly challenging for video games and emerging virtual reality headsets. To meet this demand, modern graphics hardware and game engines often reduce the computational cost by rendering at a lower resolution and then upsampling to the native resolution. Following the recent advances in image and video superresolution in computer vision, we propose a machine learning approach that is specifically tailored for high-quality upsampling of rendered content in real-time applications. The main insight of our work is that in rendered content, the image pixels are point-sampled, but precise temporal dynamics are available. Our method combines this specific information that is typically available in modern renderers (i.e., depth and dense motion vectors) with a novel temporal network design that takes into account such specifics and is aimed at maximizing video quality while delivering real-time performance. By training on a large synthetic dataset rendered from multiple 3D scenes with recorded camera motion, we demonstrate high fidelity and temporally stable results in real-time, even in the highly challenging 4 \u00d7 4 upsampling scenario, significantly outperforming existing superresolution and temporal antialiasing work.<\/jats:p>","DOI":"10.1145\/3386569.3392376","type":"journal-article","created":{"date-parts":[[2020,8,12]],"date-time":"2020-08-12T11:44:27Z","timestamp":1597232667000},"update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":70,"title":["Neural supersampling for real-time rendering"],"prefix":"10.1145","volume":"39","author":[{"given":"Lei","family":"Xiao","sequence":"first","affiliation":[{"name":"Facebook Reality Labs"}]},{"given":"Salah","family":"Nouri","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}]},{"given":"Matt","family":"Chapman","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}]},{"given":"Alexander","family":"Fix","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}]},{"given":"Douglas","family":"Lanman","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}]},{"given":"Anton","family":"Kaplanyan","sequence":"additional","affiliation":[{"name":"Facebook Reality Labs"}]}],"member":"320","published-online":{"date-parts":[[2020,8,12]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Mart\u00edn Abadi et al. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. https:\/\/www.tensorflow.org\/"},{"key":"e_1_2_2_2_1","doi-asserted-by":"crossref","unstructured":"Kurt Akeley. 1993. Reality engine graphics. In Proceeding of Computer Graphics and Interactive Techniques (SIGGRAPH). 109--116.","DOI":"10.1145\/166117.166131"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.304"},{"key":"e_1_2_2_4_1","article-title":"Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder","volume":"36","author":"Alla Chaitanya Chakravarty R.","year":"2017","unstructured":"Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 36, 4 (2017), 98:1--98:12.","journal-title":"ACM Transactions on Graphics (Proceedings of SIGGRAPH)"},{"key":"e_1_2_2_5_1","volume-title":"Temporally coherent gans for video super-resolution (tecogan). arXiv preprint arXiv:1811.09393","author":"Chu Mengyu","year":"2018","unstructured":"Mengyu Chu, You Xie, Laura Leal-Taix\u00e9, and Nils Thuerey. 2018. Temporally coherent gans for video super-resolution (tecogan). arXiv preprint arXiv:1811.09393 (2018)."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2015.2439281"},{"key":"e_1_2_2_7_1","volume-title":"NVIDIA Sponsored Sessions (Game Developers Conference).","author":"Edelsten Andrew","year":"2019","unstructured":"Andrew Edelsten, Paula Jukarainen, and Anjul Patney. 2019. Truly next-gen: Adding deep learning to games and graphics. In NVIDIA Sponsored Sessions (Game Developers Conference)."},{"key":"e_1_2_2_8_1","unstructured":"Epic Games. 2020. Unreal engine. https:\/\/www.unrealengine.com"},{"key":"e_1_2_2_9_1","unstructured":"Weifeng Ge Bingchen Gong and Yizhou Yu. 2018. Image super-resolution via deterministic-stochastic synthesis and local statistical rectification. In SIGGRAPH Asia 2018 Technical Papers. ACM 260."},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of Artificial Intelligence and Statistics. 249--256","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of Artificial Intelligence and Statistics. 249--256."},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00402"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980254"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_2_2_15_1","volume-title":"SMAA: Enhanced subpixel morphological antialiasing. Comput. Graph. Forum 31, 2pt1","author":"Jimenez Jorge","year":"2012","unstructured":"Jorge Jimenez, Jose I. Echevarria, Tiago Sousa, and Diego Gutierrez. 2012. SMAA: Enhanced subpixel morphological antialiasing. Comput. Graph. Forum 31, 2pt1 (2012), 355--364."},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00340"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46475-6_43"},{"volume-title":"Proceedings of High Performance Graphics (HPG). 151--162","author":"Kaplanyan A. S.","key":"e_1_2_2_18_1","unstructured":"A. S. Kaplanyan, S. Hill, A. Patney, and A. Lefohn. 2016. Filtering distributions of normals for shading antialiasing. In Proceedings of High Performance Graphics (HPG). 151--162."},{"key":"e_1_2_2_19_1","article-title":"DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos","volume":"38","author":"Kaplanyan Anton S.","year":"2019","unstructured":"Anton S. Kaplanyan, Anton Sochenov, Thomas Leimkuehler, Mikhail Okunev, Todd Goodall, and Rufo Gizem. 2019. DeepFovea: Neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Trans. Graph. (Proceedings of SIGGRAPH Asia) 38, 4 (2019), 212:1--212:13.","journal-title":"ACM Trans. Graph. (Proceedings of SIGGRAPH Asia)"},{"key":"e_1_2_2_20_1","article-title":"High quality temporal anti-aliasing. In ACM","author":"Karis Brian","year":"2014","unstructured":"Brian Karis. 2014. High quality temporal anti-aliasing. In ACM Trans. Graph. (Advances in Real-Time Rendering). Article 4.","journal-title":"Trans. Graph. (Advances in Real-Time Rendering). Article 4."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.182"},{"key":"e_1_2_2_22_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.618"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.19"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.151"},{"volume-title":"Technical Report","author":"Lottes T.","key":"e_1_2_2_26_1","unstructured":"T. Lottes. 2009. FXAA. Technical Report. NVIDIA Corp. 3 pages."},{"key":"e_1_2_2_27_1","unstructured":"Microsoft. 2019. Directx variable rate shading. https:\/\/microsoft.github.io\/DirectX-Specs\/d3d\/VariableRateShading.html."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3322980"},{"key":"e_1_2_2_29_1","unstructured":"Nvidia Corporation. 2017--2018. TensorRT. https:\/\/developer.nvidia.com\/tensorrt."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2980246"},{"key":"e_1_2_2_31_1","volume-title":"Cook","author":"Reeves William T.","year":"1987","unstructured":"William T. Reeves, David Salesin, and Robert L. Cook. 1987. Rendering antialiased shadows with depth maps. In ACM Transactions on Graphics (Proceedings of SIGGRAPH)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1572769.1572787"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00693"},{"key":"e_1_2_2_34_1","first-page":"1","article-title":"Spatiotemporal variance-guided filtering: Real-time reconstruction for path-traced global illumination. In Proc. High Performance Graphics (HPG)","volume":"2","author":"Schied Christoph","year":"2017","unstructured":"Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal variance-guided filtering: Real-time reconstruction for path-traced global illumination. In Proc. High Performance Graphics (HPG). Article 2, 2:1--2:12 pages.","journal-title":"Article"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.207"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2012.2214933"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.479"},{"key":"e_1_2_2_38_1","unstructured":"Unity Technologies. 2005--2020. Unity engine. http:\/\/unity3d.com."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2019.00247"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/800059.801126"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306346.3323024"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1618452.1618481"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2019.2919431"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01234-2_18"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00262"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3386569.3392376","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,5]],"date-time":"2024-09-05T00:17:49Z","timestamp":1725495469000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3386569.3392376"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,12]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,8,31]]}},"alternative-id":["10.1145\/3386569.3392376"],"URL":"https:\/\/doi.org\/10.1145\/3386569.3392376","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"type":"print","value":"0730-0301"},{"type":"electronic","value":"1557-7368"}],"subject":[],"published":{"date-parts":[[2020,8,12]]},"assertion":[{"value":"2020-08-12","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}