HST: Hierarchical Swin Transformer for Compressed Image Super-Resolution | SpringerLink
Skip to main content
Springer Nature Link
Log in

HST: Hierarchical Swin Transformer for Compressed Image Super-Resolution

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 Workshops (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13802))

Included in the following conference series:

Abstract

Compressed Image Super-resolution has achieved great attention in recent years, where images are degraded with compression artifacts and low-resolution artifacts. Since the complex hybrid distortions, it is hard to restore the distorted image with the simple cooperation of super-resolution and compression artifacts removing. In this paper, we take a step forward to propose the Hierarchical Swin Transformer (HST) network to restore the low-resolution compressed image, which jointly captures the hierarchical feature representations and enhances each-scale representation with Swin transformer, respectively. Moreover, we find that the pretraining with Super-resolution (SR) task is vital in compressed image super-resolution. To explore the effects of different SR pretraining, we take the commonly-used SR tasks (e.g., bicubic and different real super-resolution simulations) as our pretraining tasks, and reveal that SR plays an irreplaceable role in the compressed image super-resolution. With the cooperation of HST and pre-training, our HST achieves the fifth place in AIM 2022 challenge on the low-quality compressed image super-resolution track, with the PSNR of 23.51 dB. Extensive experiments and ablation studies have validated the effectiveness of our proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 19447
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 24309
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://clic.compression.cc/2021/tasks/index.html.

References

  1. Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: dataset and study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 126–135 (2017)

    Google Scholar 

  2. Bell-Kligler, S., Shocher, A., Irani, M.: Blind super-resolution kernel estimation using an internal-GAN. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  3. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding (2012)

    Google Scholar 

  4. Bross, B., Chen, J., Ohm, J.R., Sullivan, G.J., Wang, Y.K.: Developments in international video coding standardization after AVC, with an overview of versatile video coding (VVC). Proc. IEEE 109(9), 1463–1493 (2021)

    Article  Google Scholar 

  5. Cai, J., Zeng, H., Yong, H., Cao, Z., Zhang, L.: Toward real-world single image super-resolution: a new benchmark and a new model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3086–3095 (2019)

    Google Scholar 

  6. Cavigelli, L., Hager, P., Benini, L.: CAS-CNN: a deep convolutional neural network for image compression artifact suppression. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 752–759. IEEE (2017)

    Google Scholar 

  7. Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1st International Conference on Image Processing, vol. 2, pp. 168–172. IEEE (1994)

    Google Scholar 

  8. Chen, H., et al.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021)

    Google Scholar 

  9. Chen, Y., et al.: Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3435–3444 (2019)

    Google Scholar 

  10. Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)

    Google Scholar 

  11. Dong, C., Deng, Y., Loy, C.C., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 576–584 (2015)

    Google Scholar 

  12. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  13. Fu, X., Zha, Z.J., Wu, F., Ding, X., Paisley, J.: JPEG artifacts reduction via deep convolutional sparse coding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2501–2510 (2019)

    Google Scholar 

  14. Gu, J., Lu, H., Zuo, W., Dong, C.: Blind super-resolution with iterative kernel correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1604–1613 (2019)

    Google Scholar 

  15. Guo, J., Chao, H.: Building dual-domain representations for compression artifacts reduction. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 628–644. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_38

    Chapter  Google Scholar 

  16. Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)

    Google Scholar 

  17. Ji, X., Cao, Y., Tai, Y., Wang, C., Li, J., Huang, F.: Real-world super-resolution via kernel estimation and noise injection. In: proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 466–467 (2020)

    Google Scholar 

  18. Jolicoeur-Martineau, A.: The relativistic discriminator: a key element missing from standard GAN. arXiv preprint arXiv:1807.00734 (2018)

  19. Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645 (2016)

    Google Scholar 

  20. Kim, J., Choi, Y., Uh, Y.: Feature statistics mixing regularization for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11294–11303 (2022)

    Google Scholar 

  21. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4681–4690 (2017)

    Google Scholar 

  22. Li, X., Jin, X., Fu, J., Yu, X., Tong, B., Chen, Z.: Few-shot real image restoration via distortion-relation guided transfer learning. arXiv preprint arXiv:2111.13078 (2021)

  23. Li, X., et al.: Learning disentangled feature representation for hybrid-distorted image restoration. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 313–329. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_19

    Chapter  Google Scholar 

  24. Li, X., et al.: Learning omni-frequency region-adaptive representations for real image super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1975–1983 (2021)

    Google Scholar 

  25. Li, X., Shi, J., Chen, Z.: Task-driven semantic coding via reinforcement learning. IEEE Trans. Image Process. 30, 6307–6320 (2021)

    Article  Google Scholar 

  26. Li, X., Sun, S., Zhang, Z., Chen, Z.: Multi-scale grouped dense network for VVC intra coding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 158–159 (2020)

    Google Scholar 

  27. Li, Y., Jin, P., Yang, F., Liu, C., Yang, M.H., Milanfar, P.: COMISR: compression-informed video super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2543–2552 (2021)

    Google Scholar 

  28. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: image restoration using Swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833–1844 (2021)

    Google Scholar 

  29. Liang, J., Zhang, K., Gu, S., Van Gool, L., Timofte, R.: Flow-based kernel prior with application to blind super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10601–10610 (2021)

    Google Scholar 

  30. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference On Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  31. Liu, J., Li, X., Peng, Y., Yu, T., Chen, Z.: SwinIQA: learned Swin distance for compressed image quality assessment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1795–1799 (2022)

    Google Scholar 

  32. Liu, J., Lin, J., Li, X., Zhou, W., Liu, S., Chen, Z.: LIRA: lifelong image restoration from unknown blended distortions. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12363, pp. 616–632. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58523-5_36

    Chapter  Google Scholar 

  33. Liu, P., Zhang, H., Lian, W., Zuo, W.: Multi-level wavelet convolutional neural networks. IEEE Access 7, 74973–74985 (2019)

    Article  Google Scholar 

  34. Lu, M., Chen, T., Liu, H., Ma, Z.: Learned image restoration for VVC intra coding. In: CVPR Workshops (2019)

    Google Scholar 

  35. Lu, Y., et al.: RTN: reinforced transformer network for coronary CT angiography vessel-level image quality assessment. arXiv preprint arXiv:2207.06177 (2022)

  36. Luo, Z., Huang, H., Yu, L., Li, Y., Fan, H., Liu, S.: Deep constrained least squares for blind image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17642–17652 (2022)

    Google Scholar 

  37. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 2, pp. 416–423. IEEE (2001)

    Google Scholar 

  38. Matsui, Y., et al.: Sketch-based manga retrieval using manga109 dataset. Multimed. Tools Appl. 76(20), 21811–21838 (2017)

    Article  Google Scholar 

  39. Pang, Y., et al.: FAN: frequency aggregation network for real image super-resolution. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 468–483. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_28

    Chapter  Google Scholar 

  40. Pennebaker, W.B., Mitchell, J.L.: JPEG: Still Image Data Compression Standard. Springer Science & Business Media, New York (1992). https://link.springer.com/book/9780442012724

  41. Rabbani, M., Joshi, R.: An overview of the JPEG 2000 still image compression standard. Signal Process. Image Commun. 17(1), 3–48 (2002)

    Article  Google Scholar 

  42. Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

    Google Scholar 

  43. Svoboda, P., Hradis, M., Barina, D., Zemcik, P.: Compression artifacts removal using convolutional neural networks. arXiv preprint arXiv:1605.00366 (2016)

  44. Timofte, R., Agustsson, E., Van Gool, L., Yang, M.H., Zhang, L.: Ntire 2017 challenge on single image super-resolution: methods and results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114–125 (2017)

    Google Scholar 

  45. Wang, L., et al.: Unsupervised degradation representation learning for blind super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10581–10590 (2021)

    Google Scholar 

  46. Wang, X., Xie, L., Dong, C., Shan, Y.: Real-ESRGAN: training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1905–1914 (2021)

    Google Scholar 

  47. Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)

    Google Scholar 

  48. Wang, Z., Liu, D., Chang, S., Ling, Q., Yang, Y., Huang, T.S.: D3: deep dual-domain based fast restoration of JPEG-compressed images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2764–2772 (2016)

    Google Scholar 

  49. Wei, P., et al.: AIM 2020 challenge on real image super-resolution: methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 392–422. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_24

    Chapter  Google Scholar 

  50. Wei, P., et al.: Component divide-and-conquer for real-world image super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 101–117. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_7

    Chapter  Google Scholar 

  51. Wu, Y., Wang, X., Li, G., Shan, Y.: AnimeSR: learning real-world super-resolution models for animation videos. arXiv preprint arXiv:2206.07038 (2022)

  52. Wu, Y., Li, X., Zhang, Z., Jin, X., Chen, Z.: Learned block-based hybrid image compression. IEEE Trans. Circ. Syst. Video Technol. 32, 3978–3990 (2021)

    Article  Google Scholar 

  53. Yang, R., Timofte, R., et al.: AIM 2022 challenge on super-resolution of compressed image and video: dataset, methods and results. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)

    Google Scholar 

  54. Zeyde, R., Elad, M., Protter, M.: On single image scale-up using sparse-representations. In: Boissonnat, J.-D., et al. (eds.) Curves and Surfaces 2010. LNCS, vol. 6920, pp. 711–730. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27413-8_47

    Chapter  Google Scholar 

  55. Zhang, K., Liang, J., Van Gool, L., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4791–4800 (2021)

    Google Scholar 

  56. Zhang, X., Yang, W., Hu, Y., Liu, J.: DMCNN: dual-domain multi-scale convolutional neural network for compression artifacts removal. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 390–394. IEEE (2018)

    Google Scholar 

  57. Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18

    Chapter  Google Scholar 

  58. Zheng, M., et al.: Progressive training of a two-stage framework for video restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1024–1031 (2022)

    Google Scholar 

Download references

Acknowledgement

This work was supported in part by NSFC under Grant U1908209, 62021001 and the National Key Research and Development Program of China 2018AAA0101400.

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, 230027, China

    Bingchen Li, Xin Li, Yiting Lu, Sen Liu, Ruoyu Feng & Zhibo Chen

Authors
  1. Bingchen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiting Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruoyu Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhibo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhibo Chen .

Editor information

Editors and Affiliations

  1. IBM Research - MIT-IBM Watson AI Lab, Massachusetts, USA

    Leonid Karlinsky

  2. Technion – Israel Institute of Technology, Haifa, Israel

    Tomer Michaeli

  3. Kyoto University, Kyoto, Japan

    Ko Nishino

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, B., Li, X., Lu, Y., Liu, S., Feng, R., Chen, Z. (2023). HST: Hierarchical Swin Transformer for Compressed Image Super-Resolution. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13802. Springer, Cham. https://doi.org/10.1007/978-3-031-25063-7_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25063-7_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25062-0

  • Online ISBN: 978-3-031-25063-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation