Image super-resolution using only low-resolution images | The Visual Computer Skip to main content

Advertisement

Springer Nature Link
Log in

Image super-resolution using only low-resolution images

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Currently, most image super-resolution methods use the paired training dataset which is difficult to collect. So bicubic mode is often used to generate paired low-resolution and high-resolution images, but this does not match the actual degradation mode which is often unknown. This results in deteriorated performance of these methods. Considering that only low-resolution images can actually be acquired, we propose a novel model architecture named Cycle-SRGAN which uses only low-resolution images to achieve image super-resolution. The Cycle-SRGAN consists of three components: an up-sampler, a down-sampler and a discriminator. The up-sampler generates high-resolution images, and the down-sampler learns the degradation mode of the images. The discriminator is adopted to help learn the downscaling mode. And we apply two cycle losses, an adversarial loss and a regularization term to implement training. Our method can realize downscaling kernel estimation and image super-resolution at the meantime. In the end, the experimental results indicate the images generated by Cycle-SRGAN own higher image quality than some other methods. And our method has good stability and generalization performance for unknown degradation modes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The datasets used in this paper are public datasets and can be obtained by contacting the relevant providers.

References

  1. Shi, W., Caballero, J., Ledig, C., Zhuang, X., Bai, W., Bhatia, K., de Marvao, A.M.S.M., Dawes, T., O’Regan, D., Rueckert, D.: Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) Medical Image Computing and Computer-Assisted Intervention - MICCAI 2013, pp. 9–16. Springer, Berlin, Heidelberg (2013)

  2. Jiang, K., Wang, Z., Yi, P., Jiang, J., Xiao, J., Yao, Y.: Deep distillation recursive network for remote sensing imagery super-resolution. Remote Sens. 10(11), 1700 (2018). https://doi.org/10.3390/rs10111700

    Article  Google Scholar 

  3. Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP, Graph. Models Image Process. 53(3), 231–239 (1991)

    Article  Google Scholar 

  4. Keys, R.G.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. ASSP–29(6), 1153–1160 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  5. Kim, K.I., Kwon, Y.: Single-image super-resolution using sparse regression and natural image prior. IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1127–1133 (2010). https://doi.org/10.1109/TPAMI.2010.25

    Article  Google Scholar 

  6. Timofte, R., De, V., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution, pp. 1920–1927 (2013). https://doi.org/10.1109/ICCV.2013.241. http://ieeexplore.ieee.org/document/6751349/

  7. Timofte, R., De Smet, V., Van Gool, L.: A+: adjusted anchored neighborhood regression for fast super-resolution. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) Computer Vision—ACCV 2014, pp. 111–126. Springer, Cham (2015)

  8. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  9. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654 (2016). https://doi.org/10.1109/CVPR.2016.182

  10. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

  11. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

  12. Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2020). https://doi.org/10.1109/TPAMI.2020.2982166

    Article  Google Scholar 

  13. Hu, Y., Li, J., Huang, Y., Gao, X.: Channel-wise and spatial feature modulation network for single image super-resolution. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3911–3927 (2020). https://doi.org/10.1109/TCSVT.2019.2915238

    Article  Google Scholar 

  14. Zhu, X., Guo, K., Ren, S., Hu, B., Hu, M., Fang, H.: Lightweight image super-resolution with expectation-maximization attention mechanism. IEEE Trans. Circuits Syst. Video Technol. 32(3), 1273–1284 (2021). https://doi.org/10.1109/TCSVT.2021.3078436

    Article  Google Scholar 

  15. Amaranageswarao, G., Deivalakshmi, S., Ko, S.-B.: Joint restoration convolutional neural network for low-quality image super resolution. Vis. Comput. 38, 31–50 (2022). https://doi.org/10.1007/s00371-020-01998-z

    Article  Google Scholar 

  16. Shi, W., Du, H., Mei, W., Ma, Z.: (SARN) spatial-wise attention residual network for image super-resolution. Vis. Comput. 37, 1569–1580 (2021). https://doi.org/10.1007/s00371-020-01903-8

    Article  Google Scholar 

  17. Chen, S., Han, Z., Dai, E., Jia, X., Liu, Z., Xing, L., Zou, X., Xu, C., Liu, J., Tian, Q.: Unsupervised image super-resolution with an indirect supervised path. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2020)

  18. Hou, Y., Zhang, J.: Unsupervised remote sensing image super-resolution method based on adaptive domain distance measurement network, pp. 256–259 (2020). https://doi.org/10.1109/AEMCSE50948.2020.00062. https://ieeexplore.ieee.org/document/9131243/

  19. Yuan, Y., Liu, S., Zhang, J., Zhang, Y., Dong, C., Lin, L.: Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks, pp. 814–81409 (2018). https://doi.org/10.1109/CVPRW.2018.00113. https://ieeexplore.ieee.org/document/8575264/

  20. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27 (2014). https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

  21. Wang, P., Zhang, H., Zhou, F., Jiang, Z.: Unsupervised remote sensing image super-resolution using cycle CNN, pp. 3117–3120 (2019). https://doi.org/10.1109/IGARSS.2019.8898648

  22. Shocher, A., Cohen, N., Irani, M.: “zero-shot” super-resolution using deep internal learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  23. Ahn, N., Yoo, J., Sohn, K.-A.: Simusr: a simple but strong baseline for unsupervised image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2020)

  24. Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-gan. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook (2019)

  25. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  26. Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  27. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  28. Dai, T., Cai, J., Zhang, Y., Xia, S.-T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  29. Yang, X., Zhu, Y., Guo, Y., Zhou, D.: An image super-resolution network based on multi-scale convolution fusion. Vis. Comput. 1–11 (2021). https://doi.org/10.1007/s00371-021-02297-x

  30. Hu, X., Mu, H., Zhang, X., Wang, Z., Tan, T., Sun, J.: Meta-sr: a magnification-arbitrary network for super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

  31. Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  32. Chudasama, V., Upla, K., Raja, K., Raghavendra, R., Busch, C.: Compact and progressive network for enhanced single image super-resolution-ComPrESRNet. Vis. Comput. 1–23 (2021). https://doi.org/10.1007/s00371-021-02193-4

  33. Song, D., Wang, Y., Chen, H., Xu, C., Xu, C., Tao, D.: Addersr: towards energy efficient image super-resolution. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15643–15652 (2021). https://doi.org/10.1109/CVPR46437.2021.01539

  34. Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

  35. Zhang, Y., Liu, S., Dong, C., Zhang, X., Yuan, Y.: Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution. IEEE Trans. Image Process. 29, 1101–1112 (2020). https://doi.org/10.1109/TIP.2019.2938347

    Article  MathSciNet  MATH  Google Scholar 

  36. Guo, Y., Chen, J., Wang, J., Chen, Q., Cao, J., Deng, Z., Xu, Y., Tan, M.: Closed-loop matters: dual regression networks for single image super-resolution, pp. 5406–5415 (2020). https://doi.org/10.1109/CVPR42600.2020.00545. https://ieeexplore.ieee.org/document/9157622/

  37. Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super-resolution with iterative kernel correction. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1604–1613 (2019). https://doi.org/10.1109/CVPR.2019.00170

  38. Mirza, M., Osindero, S.: Conditional generative adversarial nets. CoRR. arXiv:1411.1784 (2014)

  39. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)

  40. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 214–223 (2017). https://proceedings.mlr.press/v70/arjovsky17a.html

  41. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein gans. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30 (2017). https://proceedings.neurips.cc/paper/2017/file/892c3b1c6dccd52936e27cbd0ff683d6-Paper.pdf

  42. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science (New York) 313, 504–7 (2006). https://doi.org/10.1126/science.1127647

    Article  MathSciNet  MATH  Google Scholar 

  43. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)

  44. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Press, San Francisco, California, USA (2017)

  45. Shi, W., Caballero, J., Huszar, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  46. Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. CoRR. arXiv:1505.00853 (2015)

  47. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.-H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)

  48. Bevilacqua, M., Roumy, A., Guillemot, C., Morel, M.-l.A.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the British Machine Vision Conference (2012). https://doi.org/10.5244/C.26.135

  49. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., Chenin, P., Cohen, A., Gout, C., Lyche, T., Mazure, M.-L., Schumaker, L. (eds.) Curves and Surfaces, pp. 711–730. Springer, Berlin, Heidelberg (2012)

    Chapter  Google Scholar 

  50. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 416–423 (2001). https://doi.org/10.1109/ICCV.2001.937655

  51. Huang, J.-B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

  52. Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K.: Dslr-quality photos on mobile devices with deep convolutional networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3297–3305 (2017). https://doi.org/10.1109/ICCV.2017.355

  53. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  54. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)

  55. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012). https://doi.org/10.1109/TIP.2012.2214050

    Article  MathSciNet  MATH  Google Scholar 

  56. N, V., D, P., Bh, M.C., Channappayya, S.S., Medasani, S.S.: Blind image quality evaluation using perception based features. In: 2015 21st National Conference on Communications (NCC), pp. 1–6 (2015). https://doi.org/10.1109/NCC.2015.7084843

  57. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018). https://doi.org/10.1109/CVPR.2018.00068

  58. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007). https://doi.org/10.1109/TIP.2007.901238

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, University of Science and Technology of China, No. 96, JinZhai Road Baohe District, Hefei, 230026, Anhui, People’s Republic of China

    Fan Wang & Dong Yin

  2. School of Cyber Science and Technology, University of Science and Technology of China, No. 96, JinZhai Road Baohe District, Hefei, 230026, Anhui, People’s Republic of China

    Ruiyuan Song

Authors
  1. Fan Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Dong Yin
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ruiyuan Song
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dong Yin.

Ethics declarations

Conflict of interest

All authors have no relevant financial or non-financial interests to disclose. And the authors have no competing interests to declare that are relevant to the content of this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, F., Yin, D. & Song, R. Image super-resolution using only low-resolution images. Vis Comput 39, 5069–5084 (2023). https://doi.org/10.1007/s00371-022-02646-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02646-4

Keywords