Burst super-resolution with adaptive feature refinement and enhanced group up-sampling | Applied Intelligence Skip to main content

Advertisement

Springer Nature Link
Log in

Burst super-resolution with adaptive feature refinement and enhanced group up-sampling

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Burst super-resolution (Burst SR) has achieved significant performance improvements through burst images captured by modern handheld devices. However, Burst SR still encounters several challenges. Firstly, misalignment between burst images is caused by hand tremors. Secondly, aligned features should be effectively combined and up-sampled to harness the rich information of the burst images. As numerous studies have attempted to solve these problems, the complexity of alignment and fusion strategies has been increased and the significance of up-sampling has been overlooked. It exacerbates the trade-off between performance and computational cost and leads to a problem where information from multiple frames is not effectively exploited in the up-sampling process. In this paper, we present a Burst SR network with adaptive feature refinement and enhanced group up-sampling strategies. To feature refine adaptively, we introduce a novel denoising and correction module to elaborately perform the alignment with deformable convolution instead of a pre-trained network. Furthermore, we propose the enhanced group up-sampling module to merge burst features without the loss of salient details effectively. With these two strategies, our proposed method achieves state-of-the-art performance and alleviates the trade-off between performance and computational cost compared to the existing Burst SR approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

The datasets generated during and/or analysed during the current study are available in the [SyntheticBurst dataset, real-world burst SR dataset] repository, https://github.com/goutamgmb/deep-burst-sr.

Notes

  1. The target SR scale in this work is \(\times 4\). \(\times 8\) contains an extra \(\times 2\) upscale due to the mosaicked RAW image.

References

  1. Bhat G, Danelljan M, Van Gool L et al (2021a) Deep burst super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9209–9218

  2. Bhat G, Danelljan M, Yu F et al (2021b) Deep reparametrization of multi-frame super-resolution and denoising. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2460–2470

  3. Brooks T, Mildenhall B, Xue T et al (2019) Unprocessing images for learned raw denoising. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11036–11045

  4. Chen L, Chu X, Zhang X et al (2022) Simple baselines for image restoration. In: European conference on computer vision. Springer, pp 17–33

  5. Chen Y, Liu L, Phonevilay V et al (2021) Image super-resolution reconstruction based on feature map attention mechanism. Appl Int 51:4367–4380

    Article  Google Scholar 

  6. Cho W, Son S, Kim DS (2021) Weighted multi-kernel prediction network for burst image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 404–413

  7. Dai T, Cai J, Zhang Y et al (2019) Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  8. Del Gallego NP, Ilao J (2017) Multiple-image super-resolution on mobile devices: an image warping approach. EURASIP J Image Vid Process 1:1–15

  9. Deudon M, Kalaitzis A, Goytom I et al (2020) Highres-net: recursive fusion for multi-frame super-resolution of satellite imagery. arXiv:2002.06460

  10. Dong C, Loy CC, He K et al (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  11. Dudhane A, Zamir SW, Khan S et al (2022) Burst image restoration and enhancement. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5759–5768

  12. Farsiu S, Robinson MD, Elad M et al (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344

    Article  Google Scholar 

  13. Freeman WT, Jones TR, Pasztor EC (2002) Example-based super-resolution. IEEE Comput Graph Appl 22(2):56–65

    Article  Google Scholar 

  14. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  15. Hendrycks D, Gimpel K (2016) Gaussian error linear units (gelus). arXiv:1606.08415

  16. Howard A, Sandler M, Chu G et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1314–1324

  17. Ignatov A, Van Gool L, Timofte R (2020) Replacing mobile camera isp with a single deep learning model. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 536–537

  18. Lecouat B, Ponce J, Mairal J (2021) Lucas-kanade reloaded: end-to-end super-resolution from raw image bursts. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 2370–2379

  19. Ledig C, Theis L, Huszár F et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4681–4690

  20. Li Y, Zhang K, Timofte R et al (2022) Ntire 2022 challenge on efficient super-resolution: methods and results. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops

  21. Lian W, Peng S (2023) Kernel-aware burst blind super-resolution. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 4892–4902

  22. Liu C, Sun X, Chen C et al (2019) Multi-scale residual hierarchical dense networks for single image super-resolution. IEEE Access 7:60572–60583

    Article  Google Scholar 

  23. Loshchilov I, Hutter F (2016) Sgdr: stochastic gradient descent with warm restarts. arXiv:1608.03983

  24. Lu E, Hu X (2022) Image super-resolution via channel attention and spatial attention. Appl Intell 52(2):2260–2268

    Article  MathSciNet  Google Scholar 

  25. Mehta N, Dudhane A, Murala S et al (2022) Adaptive feature consolidation network for burst super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1279–1286

  26. Molini AB, Valsesia D, Fracastoro G et al (2019) Deepsum: deep neural network for super-resolution of unregistered multitemporal images. IEEE Trans Geosci Remote Sens 58(5):3644-3656

    Article  Google Scholar 

  27. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Icml

  28. Niu B, Wen W, Ren W et al (2020) Single image super-resolution via a holistic attention network. In: Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16. Springer, pp 191–207

  29. Qin J, Liu F, Liu K et al (2022) Lightweight hierarchical residual feature fusion network for single-image super-resolution. Neurocomputing 478:104–123

    Article  Google Scholar 

  30. Shi W, Caballero J, Huszár F et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  31. Sun D, Yang X, Liu MY et al (2018) Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8934–8943

  32. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  33. Wang X, Yu K, Wu S et al (2018) Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops, pp 0–0

  34. Wang Z, Cun X, Bao J et al (2022) Uformer: a general u-shaped transformer for image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 17683–17693

  35. Wronski B, Garcia-Dorado I, Ernst M et al (2019) Handheld multi-frame super-resolution. ACM Trans Graph (TOG) 38(4):1–18

    Article  Google Scholar 

  36. Yang J, Wright J, Huang TS et al (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  Google Scholar 

  37. Zamir SW, Arora A, Khan S et al (2022) Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5728–5739

  38. Zeng K, Ding S, Jia W (2019) Single image super-resolution using a polymorphic parallel cnn. Appl Intell 49:292–300

    Article  Google Scholar 

  39. Zhang K, Gu S, Timofte R et al (2019) Aim 2019 challenge on constrained super-resolution: methods and results. In: Proceedings of the IEEE/CVF international conference on computer vision workshops

  40. Zhang K, Danelljan M, Li Y et al (2020) Aim 2020 challenge on efficient super-resolution: methods and results. In: Proceedings of the European conference on computer vision workshops

  41. Zhang Y, Li K, Li K et al (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European conference on computer vision (ECCV), pp 286–301

  42. Zhu X, Hu H, Lin S et al (2019) Deformable convnets v2: more deformable, better results. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9308–9316

Download references

Acknowledgements

This research has been supported by the Samsung Electronics Corporation. (IO201209-07893-01)

Funding

This research has been supported by the Samsung Electronics Corporation. (IO201209-07893-01)

Author information

Author notes

  1. Minchan Kang and Woojin Jeong contributed equally to this work.

Authors and Affiliations

  1. School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), 291 Daehak-ro, Yuseong-gu, Daejeon, 34141, Republic of Korea

    Minchan Kang, Woojin Jeong, Sanghyeok Son, Gyeongdo Ham & Dae-shik Kim

Authors
  1. Minchan Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Woojin Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanghyeok Son
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gyeongdo Ham
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dae-shik Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by [Minchan Kang], [Woojin Jeong] and [Sanghyeok Son], [Dae-shik Kim]. The first draft of the manuscript was written by [Minchan Kang], [Woojin Jeong], [Gyeongdo Ham], [Dae-shik Kim] and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Dae-shik Kim.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kang, M., Jeong, W., Son, S. et al. Burst super-resolution with adaptive feature refinement and enhanced group up-sampling. Appl Intell 53, 30940–30953 (2023). https://doi.org/10.1007/s10489-023-05127-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05127-w

Keywords

Search

Navigation