Multi-aggregation network based on non-separable lifting wavelet for single image deraining | Multimedia Systems Skip to main content
Springer Nature Link
Log in

Multi-aggregation network based on non-separable lifting wavelet for single image deraining

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Recently, many methods have utilized Haar wavelet to extract frequency-domain information for image deraining. However, Haar wavelet and other tensor product wavelets only capture high frequencies in horizontal, vertical, and diagonal directions, leading to the loss of high-frequency details in deraining methods. To address this issue, we propose a multi-aggregation network (MAGNet) based on non-separable lifting wavelet transform (NLWT), where NLWT is employed to capture high-frequency rain streaks in various directions. MAGNet aggregates neighboring features and incorporates outer skip connections based on the U-Net architecture, effectively utilizing the complementary information of rain patterns in different features. Additionally, a gated fusion module is used to fuse the aggregated features, capturing important rain pattern information and reducing feature redundancy. Moreover, MAGNet employs a scale-guide progressive fusion module to exploit the similarity between rain patterns at adjacent scales for deraining. Experiments on rainy datasets and a joint rain removal and object detection task demonstrate that our MAGNet outperforms advanced methods. The code is available at https://github.com/fashyon/MAGNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author upon reasonable request.

References

  1. Liu, J., Yang, W., Yang, S., et al.: Erase or fill? deep joint recurrent rain removal and reconstruction in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3233–3242 (2018)

  2. Yang, W., Liu, J., Feng, J.: Frame-consistent recurrent video deraining with dual-level flow. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1661–1670 (2019)

  3. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  4. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

  5. Chen, Y.L., Hsu, C.T.: A generalized low-rank appearance model for spatio-temporally correlated rain streaks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1968–1975 (2013)

  6. Li, Y., Tan, R.T., Guo, X., et al.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

  7. Wang, C., Xing, X., Wu, Y., et al.: DCSFN: deep cross-scale fusion network for single image rain removal. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1643–1651 (2020)

  8. Liu, B., Liu, W.: The lifting factorization of 2D 4-channel nonseparable wavelet transforms. Inf. Sci. 456, 113–130 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  9. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241. Springer, Cham (2015)

  10. Lin, H., Jing, C., Huang, Y., et al.: A2Net: adjacent aggregation networks for image raindrop remova. IEEE Access 8, 60769–60779 (2020)

    Article  Google Scholar 

  11. Park, Y., Jeon, M., Lee, J., et al.: MCW-Net: single image deraining with multi-level connections and wide regional non-local blocks. Signal Process. Image Commun. 105, 116701 (2022)

    Article  Google Scholar 

  12. Qin, X., Wang, Z., Bai, Y., et al.: FFA-Net: feature fusion attention network for single image dehazing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, pp. 11908–11915 (2020)

  13. Wang, C., Zhu, H., Fan, W., et al.: Single image rain removal using recurrent scale-guide networks. Neurocomputing 467, 242–255 (2022)

    Article  Google Scholar 

  14. Wang, Q., Wu, B., Zhu, P., et al.: ECA-Net: efficient channel attention for deep convolutional neural networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE (2020)

  15. Woo, S., Park, J., Lee, J.Y., et al.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  16. Gu, S., Meng, D., Zuo, W., et al.: Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1708–1716 (2017)

  17. Fu, X., Huang, J., Ding, X., et al.: Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans. Image Process. 26(6), 2944–2956 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  18. Li, X., Wu, J., Lin, Z., et al.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 254–269 (2018)

  19. Wang, T., Yang, X., Xu, K., et al.: Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12270–12279 (2019)

  20. Li, R., Cheong, L.F., Tan, R.T.: Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1633–1642 (2019)

  21. Wang, H., Xie, Q., Zhao, Q., et al.: A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3103–3112 (2020)

  22. Chen, H., Wang, Y., Guo, T., et al.: Pre-trained image processing transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021)

  23. Yi, Q., Li, J., Dai, Q., et al.: Structure-preserving deraining with residue channel prior guidance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4238–4247 (2021)

  24. Li, Y., Monno, Y., Okutomi, M.: Single image deraining network with rain embedding consistency and layered LSTM. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4060–4069 (2022)

  25. Yang, W., Tan, R.T., Feng, J., et al.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

  26. Jiang, K., Wang, Z., Yi, P., et al.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8346–8355 (2020)

  27. Lin, T.Y., Dollár, P., Girshick, R., et al.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

  28. Liu, B., Peng, J.X.: Fusion method of multi-spectral image and panchromatic image based on four channels non-sperable additive wavelets. Chin. J. Comput. 32(2), 350–356 (2009). (in Chinese)

    Article  Google Scholar 

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., et al.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  30. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 695–704 (2018)

  31. Huynh-Thu, Q., Ghanbari, M.: Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008)

    Article  Google Scholar 

  32. Kingma, D.P., Ba, J.: ADAM: a method for stochastic optimization (2014). arXiv preprint arXiv:1412.6980

  33. Ren, D., Zuo, W., Hu, Q., et al.: Progressive image deraining networks: a better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

  34. Ren, D., Shang, W., Zhu, P., et al.: Single image deraining using bilateral recurrent network. IEEE Trans. Image Process. 29, 6852–6863 (2020)

    Article  MATH  Google Scholar 

  35. Guo, Q., Sun, J., Juefei-Xu, F., et al.: EfficientDeRain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining. In: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, no. 2, pp. 1487–1495 (2021)

  36. Cui, X., Wang, C., Ren, D., et al.: Semi-supervised image deraining using knowledge distillation. IEEE Trans. Circuits Syst. Video Technol. 32, 8327–8341 (2022)

    Article  Google Scholar 

  37. Ge, Z., Liu, S., Wang, F., et al.: YOLOX: exceeding yolo series in 2021 (2021). arXiv preprint arXiv:2107.08430

Download references

Acknowledgements

This research is supported by the National Natural Science Foundation of China (No. 61471160).

Funding

This work was supported by the National Natural Science Foundation of China (No. 61471160).

Author information

Authors and Affiliations

  1. School of Computer and Information Engineering, Hubei University, Wuhan, 430062, Hubei, China

    Bin Liu & Siyan Fang

Authors
  1. Bin Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Siyan Fang
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

BL: Conceptualization, validation, writing—review & editing, Supervision, project administration, funding acquisition. SF: Methodology, software, formal analysis, investigation, data curation, writing—original draft, visualization.

Corresponding author

Correspondence to Siyan Fang.

Ethics declarations

Conflict of Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Communicated by J. Gao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, B., Fang, S. Multi-aggregation network based on non-separable lifting wavelet for single image deraining. Multimedia Systems 29, 3669–3684 (2023). https://doi.org/10.1007/s00530-023-01156-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01156-0

Keywords