Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration | SpringerLink
Skip to main content
Springer Nature Link
Log in

Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15074))

Included in the following conference series:

  • 117 Accesses

Abstract

How to explore useful features from images as prompts to guide the deep image restoration models is an effective way to solve image restoration. In contrast to mining spatial relations within images as prompt, which leads to characteristics of different frequencies being neglected and further remaining subtle or undetectable artifacts in the restored image, we develop a Frequency Prompting image restoration method, dubbed FPro, which can effectively provide prompt components from a frequency perspective to guild the restoration model address these differences. Specifically, we first decompose input features into separate frequency parts via dynamically learned filters, where we introduce a gating mechanism for suppressing the less informative elements within the kernels. To propagate useful frequency information as prompt, we then propose a dual prompt block, consisting of a low-frequency prompt modulator (LPM) and a high-frequency prompt modulator (HPM), to handle signals from different bands respectively. Each modulator contains a generation process to incorporate prompting components into the extracted frequency maps, and a modulation part that modifies the prompt feature with the guidance of the decoder features. Experimental results on several popular datasets have demonstrated the favorable performance of our pipeline against SOTA methods on 5 image restoration tasks, including deraining, deraindrop, demoiréing, deblurring, and dehazing. The source code is available at https://github.com/joshyZhou/FPro.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 8465
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 10581
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Brown, T.B., et al.: Language models are few-shot learners. In: NeurIPS (2020)

    Google Scholar 

  2. Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: Dehazenet: an end-to-end system for single image haze removal. TIP 25, 5187–5198 (2016)

    MathSciNet  Google Scholar 

  3. Chantas, G., Galatsanos, N.P., Molina, R., Katsaggelos, A.K.: Variational bayesian image restoration with a product of spatially weighted total variation image priors. TIP 19, 351–362 (2009)

    MathSciNet  Google Scholar 

  4. Chen, H., et al.: Pre-trained image processing transformer. In: CVPR (2021)

    Google Scholar 

  5. Chen, L., Chu, X., Zhang, X., Sun, J.: Simple baselines for image restoration. In: ECCV 2022, pp. 17–33. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-20071-7_2

  6. Chen, L., Lu, X., Zhang, J., Chu, X., Chen, C.: Hinet: half instance normalization network for image restoration. In: CVPR Workshops (2021)

    Google Scholar 

  7. Chen, X., Li, H., Li, M., Pan, J.: Learning a sparse transformer network for effective image deraining. In: CVPR (2023)

    Google Scholar 

  8. Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., Liu, Z.: Dynamic convolution: attention over convolution kernels. In: CVPR (2020)

    Google Scholar 

  9. Cho, S.J., Ji, S.W., Hong, J.P., Jung, S.W., Ko, S.J.: Rethinking coarse-to-fine approach in single image deblurring. In: ICCV (2021)

    Google Scholar 

  10. Deng, X., Dragotti, P.L.: Deep convolutional neural network for multi-modal image restoration and fusion. TPAMI 43, 3333–3348 (2021)

    Article  Google Scholar 

  11. Dong, J., Pan, J., Yang, Z., Tang, J.: Multi-scale residual low-pass filter network for image deblurring. In: ICCV (2023)

    Google Scholar 

  12. Dong, Y., Liu, Y., Zhang, H., Chen, S., Qiao, Y.: Fd-gan: generative adversarial networks with fusion-discriminator for single image dehazing. In: AAAI (2020)

    Google Scholar 

  13. Dosovitskiy, A., et al.: An image is worth 16\(\times \)16 words: transformers for image recognition at scale. In: ICLR (2021)

    Google Scholar 

  14. d’Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G., Sagun, L.: Convit: improving vision transformers with soft convolutional inductive biases. In: ICML (2021)

    Google Scholar 

  15. Eigen, D., Krishnan, D., Fergus, R.: Restoring an image taken through a window covered with dirt or rain. In: ICCV (2013)

    Google Scholar 

  16. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: CVPR (2017)

    Google Scholar 

  17. Gan, Y., et al.: Decorate the newcomers: visual domain prompt for continual test time adaptation. In: AAAI (2023)

    Google Scholar 

  18. Guo, Y., Xiao, X., Chang, Y., Deng, S., Yan, L.: From sky to the ground: a large-scale benchmark and simple baseline towards real rain removal. In: ICCV (2023)

    Google Scholar 

  19. He, B., Wang, C., Shi, B., Duan, L.: Mop moiré patterns using mopnet. In: ICCV (2019)

    Google Scholar 

  20. He, B., Wang, C., Shi, B., Duan, L.-Y.: Fhd\(\text{ e}^{2}\)net: full high definition demoireing network. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12367, pp. 713–729. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58542-6_43

    Chapter  Google Scholar 

  21. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. TPAMI 33, 2341–2353 (2010)

    Google Scholar 

  22. Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)

  23. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017)

    Google Scholar 

  24. Jia, M., et al.: Visual prompt tuning. In: ECCV 2022, pp. 709–727. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19827-4_41

  25. Jiang, L., Dai, B., Wu, W., Loy, C.C.: Focal frequency loss for image reconstruction and synthesis. In: CVPR (2021)

    Google Scholar 

  26. Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., Khan, F.S.: Maple: multi-modal prompt learning. In: CVPR (2023)

    Google Scholar 

  27. Kong, L., Dong, J., Ge, J., Li, M., Pan, J.: Efficient frequency domain-based transformers for high-quality image deblurring. In: CVPR (2023)

    Google Scholar 

  28. Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: Aod-net: all-in-one dehazing network. In: ICCV (2017)

    Google Scholar 

  29. Li, B., et al.: Benchmarking single-image dehazing and beyond. TIP 28, 492–505 (2018)

    MathSciNet  Google Scholar 

  30. Li, B., Liu, X., Hu, P., Wu, Z., Lv, J., Peng, X.: All-in-one image restoration for unknown corruption. In: CVPR (2022)

    Google Scholar 

  31. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: CVPR (2016)

    Google Scholar 

  32. Li, Z., Lei, Y., Ma, C., Zhang, J., Shan, H.: Prompt-in-prompt learning for universal image restoration. arXiv preprint arXiv:2312.05038 (2023)

  33. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: Swinir: image restoration using swin transformer. In: ICCV Workshops (2021)

    Google Scholar 

  34. Liu, D., Wen, B., Fan, Y., Loy, C.C., Huang, T.S.: Non-local recurrent network for image restoration. In: NeurIPS (2018)

    Google Scholar 

  35. Liu, J., Yan, M., Zeng, T.: Surface-aware blind image deblurring. TPAMI 43, 1041–1055 (2021)

    Article  Google Scholar 

  36. Liu, L., et al.: Wavelet-based dual-branch network for image demoiréing. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 86–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_6

    Chapter  Google Scholar 

  37. Liu, L., et al.: Tape: task-agnostic prior embedding for image restoration. In: ECCV 2022, pp. 447–464. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19797-0_26

  38. Liu, X., Suganuma, M., Sun, Z., Okatani, T.: Dual residual networks leveraging the potential of paired operations for image restoration. In: CVPR (2019)

    Google Scholar 

  39. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: ICCV (2021)

    Google Scholar 

  40. Ma, J., Cheng, T., Wang, G., Zhang, Q., Wang, X., Zhang, L.: Prores: exploring degradation-aware visual prompt for universal image restoration. arXiv preprint arXiv:2306.13653 (2023)

  41. Mei, Y., Fan, Y., Zhou, Y.: Image super-resolution with non-local sparse attention. In: CVPR (2021)

    Google Scholar 

  42. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind" image quality analyzer. IEEE SPL 20, 209–212 (2012)

    Google Scholar 

  43. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR (2017)

    Google Scholar 

  44. Narasimhan, S.G., Nayar, S.K.: Contrast restoration of weather degraded images. TPAMI 25, 713–724 (2003)

    Article  Google Scholar 

  45. Niu, B., et al.: Single image super-resolution via a holistic attention network. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 191–207. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_12

    Chapter  Google Scholar 

  46. Oppenheim, A.: Discrete-Time Signal Processing. Prentice-Hall, Upper Saddle River (1999)

    Google Scholar 

  47. Özdenizci, O., Legenstein, R.: Restoring vision in adverse weather conditions with patch-based denoising diffusion models. TPAMI 45, 10346–10357 (2023)

    Article  Google Scholar 

  48. Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C.C., Luo, P.: Exploiting deep generative prior for versatile image restoration and manipulation. TPAMI 44, 7474–7489 (2022)

    Article  Google Scholar 

  49. Park, N., Kim, S.: How do vision transformers work? In: ICLR (2022)

    Google Scholar 

  50. Potlapalli, V., Zamir, S.W., Khan, S., Khan, F.S.: Promptir: prompting for all-in-one blind image restoration. In: NeurIPS (2023)

    Google Scholar 

  51. Purohit, K., Suin, M., Rajagopalan, A., Boddeti, V.N.: Spatially-adaptive image restoration using distortion-guided networks. In: ICCV (2021)

    Google Scholar 

  52. Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: CVPR (2018)

    Google Scholar 

  53. Qu, Y., Chen, Y., Huang, J., Xie, Y.: Enhanced pix2pix dehazing network. In: CVPR (2019)

    Google Scholar 

  54. Quan, Y., Deng, S., Chen, Y., Ji, H.: Deep learning for seeing through window with raindrops. In: ICCV (2019)

    Google Scholar 

  55. Rabiner, L.R., Gold, B.: Theory and Application of Digital Signal Processing. Prentice-Hall, Englewood Cliffs (1975)

    Google Scholar 

  56. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: CVPR (2019)

    Google Scholar 

  57. Ren, W., et al.: Single image dehazing via multi-scale convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 154–169. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_10

    Chapter  Google Scholar 

  58. Ren, W., et al.: Deblurring dynamic scenes via spatially varying recurrent neural networks. TPAMI 44, 3974–3987 (2022)

    Google Scholar 

  59. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  60. Song, X., et al.: Tusr-net: triple unfolding single image dehazing with self-regularization and dual feature to pixel attention. TIP 32, 1231–1244 (2023)

    Google Scholar 

  61. Sun, Y., Yu, Y., Wang, W.: Moiré photo restoration using multiresolution convolutional neural networks. TIP 27, 4160–4178 (2018)

    Google Scholar 

  62. Tu, Z., et al.: Maxim: multi-axis mlp for image processing. In: CVPR (2022)

    Google Scholar 

  63. Valanarasu, J.M.J., Yasarla, R., Patel, V.M.: Transweather: transformer-based restoration of images degraded by adverse weather conditions. In: CVPR (2022)

    Google Scholar 

  64. Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)

    Google Scholar 

  65. Voigtman, E., Winefordner, J.D.: Low-pass filters for signal averaging. Rev. Sci. Inst. 57, 957–966 (1986)

    Article  Google Scholar 

  66. Wang, C., He, B., Wu, S., Wan, R., Shi, B., Duan, L.Y.: Coarse-to-fine disentangling demoiréing framework for recaptured screen images. TPAMI 45, 9439–9453 (2023)

    Article  Google Scholar 

  67. Wang, C., Pan, J., Lin, W., Dong, J., Wu, X.M.: Selfpromer: self-prompt dehazing transformers with depth-consistency. arXiv preprint arXiv:2303.07033 (2023)

  68. Wang, C., et al.: Promptrestorer: a prompting image restoration method with degradation perception. In: NeurIPS (2023)

    Google Scholar 

  69. Wang, H., Xie, Q., Zhao, Q., Meng, D.: A model-driven deep neural network for single image rain removal. In: CVPR (2020)

    Google Scholar 

  70. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: CVPR (2019)

    Google Scholar 

  71. Wang, W., et al.: Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. In: ICCV (2021)

    Google Scholar 

  72. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: Uformer: a general u-shaped transformer for image restoration. In: CVPR (2022)

    Google Scholar 

  73. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. TIP 13, 600–612 (2004)

    Google Scholar 

  74. Wu, R., Yang, T., Sun, L., Zhang, Z., Li, S., Zhang, L.: Seesr: towards semantics-aware real-world image super-resolution. arXiv preprint arXiv:2311.16518 (2023)

  75. Xiao, J., Fu, X., Liu, A., Wu, F., Zha, Z.J.: Image de-raining transformer. TPAMI 45, 12978–12995 (2022)

    Article  Google Scholar 

  76. Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollar, P., Girshick, R.: Early convolutions help transformers see better. In: NeurIPS (2021)

    Google Scholar 

  77. Yang, W., Tan, R.T., Feng, J., Guo, Z., Yan, S., Liu, J.: Joint rain detection and removal from a single image with contextualized deep networks. TPAMI 42, 1377–1393 (2019)

    Article  Google Scholar 

  78. Yu, F., et al.: Scaling up to excellence: practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint arXiv:2401.13627 (2024)

  79. Yu, K., Wang, X., Dong, C., Tang, X., Loy, C.C.: Path-restore: learning network path selection for image restoration. TPAMI 44, 7078–7092 (2022)

    Article  Google Scholar 

  80. Yue, H., Mao, Y., Liang, L., Xu, H., Hou, C., Yang, J.: Recaptured screen image demoiréing. TCSVT (2021)

    Google Scholar 

  81. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. In: CVPR (2022)

    Google Scholar 

  82. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.-H., Shao, L.: Learning enriched features for real image restoration and enhancement. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12370, pp. 492–511. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58595-2_30

    Chapter  Google Scholar 

  83. Zamir, S.W., et al.: Multi-stage progressive image restoration. In: CVPR (2021)

    Google Scholar 

  84. Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., Timofte, R.: Plug-and-play image restoration with deep denoiser prior. TPAMI 44, 6360–6376 (2021)

    Article  Google Scholar 

  85. Zhao, H., Gou, Y., Li, B., Peng, D., Lv, J., Peng, X.: Comprehensive and delicate: an efficient transformer for image restoration. In: CVPR (2023)

    Google Scholar 

  86. Zheng, B., Yuan, S., Slabaugh, G., Leonardis, A.: Image demoireing with learnable bandpass filters. In: CVPR (2020)

    Google Scholar 

  87. Zheng, C., Zhang, Y., Gu, J., Zhang, Y., Kong, L., Yuan, X.: Cross aggregation transformer for image restoration. In: NeurIPS (2022)

    Google Scholar 

  88. Zhou, S., Chen, D., Pan, J., Shi, J., Yang, J.: Adapt or perish: adaptive sparse transformer with attentive feature refinement for image restoration. In: CVPR (2024)

    Google Scholar 

  89. Zou, X., Xiao, F., Yu, Z., Li, Y., Lee, Y.J.: Delving deeper into anti-aliasing in convnets. IJCV 131, 67–81 (2023)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Natural Science Foundation of Tianjin, China (No. 20JCJQJC00020), the National Natural Science Foundation of China (Nos. U22B2049, 62302240), Fundamental Research Funds for the Central Universities, and Supercomputing Center of Nankai University (NKSC).

Author information

Authors and Affiliations

  1. VCIP & TMCC & DISSec, College of Computer Science, Nankai University, Tianjin, China

    Shihao Zhou, Jinglei Shi, Duosheng Chen, Lishen Qu & Jufeng Yang

  2. Nankai International Advanced Research Institute (SHENZHEN FUTIAN), Tianjin, China

    Shihao Zhou & Jufeng Yang

  3. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China

    Jinshan Pan

Authors
  1. Shihao Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinglei Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Duosheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lishen Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jufeng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jinglei Shi or Jufeng Yang .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhou, S., Pan, J., Shi, J., Chen, D., Qu, L., Yang, J. (2025). Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15074. Springer, Cham. https://doi.org/10.1007/978-3-031-72640-8_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72640-8_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72639-2

  • Online ISBN: 978-3-031-72640-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation