SNeRV: Spectra-Preserving Neural Representation for Video | SpringerLink
Skip to main content
Springer Nature Link
Log in

SNeRV: Spectra-Preserving Neural Representation for Video

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15113))

Included in the following conference series:

  • 60 Accesses

Abstract

Neural representation for video (NeRV), which employs a neural network to parameterize video signals, introduces a novel methodology in video representations. However, existing NeRV-based methods have difficulty in capturing fine spatial details and motion patterns due to spectral bias, in which a neural network learns high-frequency (HF) components at a slower rate than low-frequency (LF) components. In this paper, we propose spectra-preserving NeRV (SNeRV) as a novel approach to enhance implicit video representations by efficiently handling various frequency components. SNeRV uses 2D discrete wavelet transform (DWT) to decompose video into LF and HF features, preserving spatial structures and directly addressing the spectral bias issue. To balance the compactness, we encode only the LF components, while HF components that include fine textures are generated by a decoder. Specialized modules, including a multi-resolution fusion unit (MFU) and a high-frequency restorer (HFR), are integrated into a backbone to facilitate the representation. Furthermore, we extend SNeRV to effectively capture temporal correlations between adjacent video frames, by casting the extension as additional frequency decomposition to a temporal domain. This approach allows us to embed spatio-temporal LF features into the network, using temporally extended up-sampling blocks (TUBs). Experimental results demonstrate that SNeRV outperforms existing NeRV models in capturing fine details and achieves enhanced reconstruction, making it a promising approach in the field of implicit video representations. The codes are available at https://github.com/qwertja/SNeRV.

J. Kim and J. Lee—Authors contribute equally.

Jihoo Lee is currently with SoC R&D Center, LG Electronics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 8465
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 10581
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Arora, S., Du, S., Hu, W., Li, Z., Wang, R.: Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In: International Conference on Machine Learning (2019)

    Google Scholar 

  2. Bai, Y., Dong, C., Wang, C., Yuan, C.: PS-NeRV: patch-wise stylized neural representations for videos. In: Proceedings of the IEEE Conference on Image Processing (2023)

    Google Scholar 

  3. Balle, J., Minnen, D., Singh, S., Hwang, S.J., Johnston, N.: Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436 (2018)

  4. Basri, R., Galun, M., Geifman, A., Jacobs, D., Kasten, Y., Kritchman, S.: Frequency bias in neural networks for input of non-uniform density. arXiv preprint arXiv:2003.04560 (2020)

  5. Bross, B., et al.: Overview of the versatile video coding (VVC) standard and its applications. IEEE Trans. Circuits Syst. Video Technol. 31(10), 3736–3764 (2021)

    Article  Google Scholar 

  6. Cao, Y., Fang, Z., Wu, Y., Zhou, D.X., Gu, Q.: Towards understanding the spectral bias of deep learning. arXiv preprint arXiv:1912.01198 (2019)

  7. Chen, H., Gwilliam, M., Lim, S., Shrivastava, A.: HNERV: a hybrid neural representation for videos. In: IEEE Conference on Computer Vision Pattern Recognition, pp. 10270–10279 (2023)

    Google Scholar 

  8. Chen, H., He, B., Wang, H., Ren, Y., Lim, S.N., Shrivastava, A.: NeRV: neural representations for videos. In: Advances in Neural Information Processing Systems, vol. 34, pp. 21557–21568 (2021)

    Google Scholar 

  9. Chen, Y.H., Chen, S.C., Lin, Y.Y., Peng, W.H.: MoTIF: learning motion trajectories with local implicit neural functions for continuous space-time video super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23131–23141 (2023)

    Google Scholar 

  10. Chen, Z., et al.: VideoINR: learning video implicit neural representation for continuous space-time super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2047–2057 (2022)

    Google Scholar 

  11. Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939–5948 (2019)

    Google Scholar 

  12. Choi, Y.J., Lee, Y.W., Kim, B.G.: Wavelet attention embedding networks for video super-resolution. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 7314–7320. IEEE (2021)

    Google Scholar 

  13. Djelouah, A., Campos, J., Schaub-Meyer, S., Schroers, C.: Neural inter-frame compression for video coding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6421–6429 (2019)

    Google Scholar 

  14. He, B., et al.: Towards scalable neural representation for diverse videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6132–6142 (2023)

    Google Scholar 

  15. Hu, Z., Lu, G., Xu, D.: FVC: a new framework towards deep video compression in feature space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1502–1511 (2021)

    Google Scholar 

  16. Huang, H., He, R., Sun, Z., Tan, T.: Wavelet domain generative adversarial network for multi-scale face hallucination. Int. J. Comput. Vis. 127(6–7), 763–784 (2019)

    Article  Google Scholar 

  17. Jin, B., et al.: Exploring spatial-temporal multi-frequency analysis for high-fidelity and temporal-consistency video prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4554–4563 (2020)

    Google Scholar 

  18. Jung, H., et al.: Anyflow: arbitrary scale optical flow with implicit neural representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5455–5465 (2023)

    Google Scholar 

  19. Kong, L., et al.: Dynamic frame interpolation in wavelet domain. IEEE Trans. Image Process. (2023)

    Google Scholar 

  20. Kwan, H.M., Gao, G., Zhang, F., Gower, A., Bull, D.: HiNeRV: video compression with hierarchical encoding-based neural representation. In: Advances in Neural Information Processing Systems, vol. 36 (2024)

    Google Scholar 

  21. Lee, J.C., Rho, D., Ko, J.H., Park, E.: FFNeRV: flow-guided frame-wise neural representations for videos. arXiv preprint arXiv:2212.12294 (2022)

  22. Lee, J.K., Kim, N., Cho, S., Kang, J.W.: Deep video prediction network-based inter-frame coding in HEVC. IEEE Access 8, 95906–95917 (2020)

    Article  Google Scholar 

  23. Li, J., Li, B., Lu, Y.: Deep contextual video compression. In: Advances in Neural Information Processing Systems, vol. 34, pp. 18114–18125 (2021)

    Google Scholar 

  24. Li, Z., Wang, M., Pi, H., Xu, K., Mei, J., Liu, Y.: E-NeRV: expedite neural video representation with disentangled spatial-temporal context. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13695, pp. 267–284. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19833-5_16

    Chapter  Google Scholar 

  25. Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., Gao, Z.: DVC: an end-to-end deep video compression framework. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11006–11015 (2019)

    Google Scholar 

  26. Lu, Y., Wang, Z., Liu, M., Wang, H., Wang, L.: Learning spatial-temporal implicit neural representations for event-guided video super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1557–1567 (2023)

    Google Scholar 

  27. Maiya, S.R., et al.: Nirvana: neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14378–14387 (2023)

    Google Scholar 

  28. Mallat, S.: A Wavelet Tour of Signal Processing. Elsevier (1999)

    Google Scholar 

  29. Mercat, A., Viitanen, M., Vanne, J.: UVG dataset: 50/120fps 4k sequences for video codec analysis and development. In: Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297–302 (2020)

    Google Scholar 

  30. Pan, Z., Yi, X., Zhang, Y., Jeon, B., Kwong, S.: Efficient in-loop filtering based on enhanced deep convolutional neural networks for HEVC. IEEE Trans. Image Process. 29, 5352–5366 (2020)

    Article  Google Scholar 

  31. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174 (2019)

    Google Scholar 

  32. Perazzi, F., Pont-Tuset, J., McWilliams, B., Gool, L.V., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 724–732 (2016)

    Google Scholar 

  33. Ramamonjisoa, M., Firman, M., Watson, J., Lepetit, V., Turmukhambetov, D.: Single image depth prediction with wavelet decomposition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11089–11098 (2021)

    Google Scholar 

  34. Rho, D., Cho, J., Ko, J.H., Park, E.: Neural residual flow fields for efficient video representations. In: Proceedings of the Asian Conference on Computer Vision, pp. 3447–3463 (2022)

    Google Scholar 

  35. Rho, D., Lee, B., Nam, S., Lee, J.C., Ko, J.H., Park, E.: Masked wavelet representation for compact neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20680–20690 (2023)

    Google Scholar 

  36. Ronen, B., Jacobs, D., Kasten, Y., Kritchman, S.: The convergence rate of neural networks for learned functions of different frequencies. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  37. Saragadam, V., LeJeune, D., Tan, J., Balakrishnan, G., Veeraraghavan, A., Baraniuk, R.G.: Wire: wavelet implicit neural representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507–18516 (2023)

    Google Scholar 

  38. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

    Google Scholar 

  39. Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Advances in Neural Information Processing Systems, vol. 33, pp. 7462–7473 (2020)

    Google Scholar 

  40. Sullivan, G.J., Ohm, J.R., Han, W.J., Wiegand, T.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12), 1649–1668 (2012)

    Article  Google Scholar 

  41. Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. In: Advances in Neural Information Processing Systems, vol. 33 (2020)

    Google Scholar 

  42. Taubman, D.S., Marcellin, M.W., Rabbani, M.: JPEG 2000: image compression fundamentals, standards and practice. J. Electron. Imaging 11(2), 286–287 (2002)

    Article  Google Scholar 

  43. Unser, M., Blu, T.: Mathematical properties of the JPEG2000 wavelet filters. IEEE Trans. Image Process. 12(9), 1080–1090 (2003)

    Article  MathSciNet  Google Scholar 

  44. Wang, P., Fan, Z., Chen, T., Wang, Z.: Neural implicit dictionary learning via mixture-of-expert training. In: International Conference on Machine Learning, pp. 22613–22624. PMLR (2022)

    Google Scholar 

  45. Wiegand, T., Sullivan, G.J., Bjontegaard, G., Luthra, A.: Overview of the h. 264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13(7), 560–576 (2003)

    Google Scholar 

  46. Xin, J., Li, J., Jiang, X., Wang, N., Huang, H., Gao, X.: Wavelet-based dual recursive network for image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. 33(2), 707–720 (2020)

    Article  Google Scholar 

  47. Yuce, G., Ortiz-Jimenez, G., Besbinar, B., Frossard, P.: A structured dictionary perspective on implicit neural representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19228–19238 (2022)

    Google Scholar 

  48. Zhao, Q., Asif, M.S., Ma, Z.: DNeRV: modeling inherent dynamics via difference neural representation for videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2031–2040 (2023)

    Google Scholar 

Download references

Acknowledgments

This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS-2021-II212068 and RS-2022-00167169) and was partly supported by the NRF grant funded by MSIT (No. NRF-2022R1A2C4002052).

Author information

Authors and Affiliations

  1. Department of Electronic and Electrical Engineering and Graduate Program in Smart Factory, Ewha Womans University, Seoul, South Korea

    Jina Kim, Jihoo Lee & Je-Won Kang

Authors
  1. Jina Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jihoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Je-Won Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Je-Won Kang .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 10238 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kim, J., Lee, J., Kang, JW. (2025). SNeRV: Spectra-Preserving Neural Representation for Video. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15113. Springer, Cham. https://doi.org/10.1007/978-3-031-73001-6_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73001-6_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73000-9

  • Online ISBN: 978-3-031-73001-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation