Quality and content-aware fusion optimization mechanism of infrared and visible images | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

Quality and content-aware fusion optimization mechanism of infrared and visible images

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Infrared and visible image fusion aims to generate a single fused image that contains abundant texture details and thermal radiance information. For this purpose, many unsupervised deep learning image fusion methods have been proposed, ignoring image content and quality awareness. To address these challenges, this paper presents a quality and content-aware image fusion network, termed QCANet, capable of solving the similarity fusion optimization problems, e.g., the dependence of fusion results on source images and the weighted average fusion effect. Specifically, the QCANet is composed of three modules, i.e., Image Fusion Network (IFNet), Quality-Aware Network (QANet), and Content-Aware Network (CANet). The latter two modules, a.k.a., QANet and CANet, aim to improve the content semantic awareness and quality awareness of IFNet. In addition, a new quality-aware image fusion loss is introduced to avoid the weighted average effect caused by the traditional similarity metric optimization mechanism. Therefore, the stumbling blocks of deep learning in image fusion, i.e., similarity fusion optimization problems, are significantly mitigated. Extensive experiments demonstrate that the quality and content-aware image fusion method outperforms most state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Aardt V (2008) Jan: assessment of image fusion procedures using entropy, image quality, and multispectral classification. J Appl Remote Sens 2(1):1–28

    Google Scholar 

  2. Bavirisetti D P, Dhuli R (2016) Two-scale image fusion of visible and infrared images using saliency detection. Infrared Phys Technol 76:52–64

    Article  Google Scholar 

  3. Chen H, Varshney P K (2007) A human perception inspired quality metric for image fusion based on regional information. Inf Fusion 8(2):193–207

    Article  Google Scholar 

  4. Cui G, Feng H, Xu Z, Li Q, Chen Y (2015) Detail preserved fusion of visible and infrared images using regional saliency extraction and multi-scale image decomposition. Opt Commun 341(341):199–209

    Article  Google Scholar 

  5. Eskicioglu A M, Fisher P S (1995) Image quality measures and their performance. IEEE Trans Commun 43(12):2959–2965

    Article  Google Scholar 

  6. Fang A, Zhao X, Zhang Y (2020) Cross-modal image fusion guided by subjective visual attention. Neurocomputing 414:333–345

    Article  Google Scholar 

  7. Gu K, Zhai G, Yang X, Zhang W, Chen C W (2015) Automatic contrast enhancement technology with saliency preservation. IEEE Trans Circuits Syst Video Technol 25(9):1480–1494

    Article  Google Scholar 

  8. Gu K, Li L, Lu H, Min X, Lin W (2017) A fast reliable image quality predictor by fusing micro- and macro-structures. IEEE Trans Ind Electron 64(5):3903–3912

    Article  Google Scholar 

  9. Gu K, Jakhetiya V, Qiao J -F, Li X, Lin W, Thalmann D (2018) Model-based referenceless quality metric of 3d synthesized images using local image description. IEEE Trans Image Process 27(1):394–405

    Article  MathSciNet  MATH  Google Scholar 

  10. Ha Q, Watanabe K, Karasawa T, Ushiku Y, Harada T (2017) Mfnet: towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In: 2017 IEEE/RSJ International conference on intelligent robots and systems, pp 5108–5115

  11. Hou R, Nie R, Zhou D, Cao J, Liu D (2019) Infrared and visible images fusion using visual saliency and optimized spiking cortical model in non-subsampled shearlet transform domain. Multimed Tools Appl (78):28609–28632

  12. Jiayi M, Wei Y, Pengwei L, Chang L, Junjun J (2019) Fusiongan: a generative adversarial network for infrared and visible image fusion. Inf Fusion 48:11–26

    Article  Google Scholar 

  13. Ke G, Zhou J, Qiao J, Zhai G, Bovik A C (2017) No-reference quality assessment of screen content pictures. IEEE Trans Image Process PP(99):1–1

    MathSciNet  MATH  Google Scholar 

  14. Li C, Liang X, Lu Y, Zhao N, Tang J (2019) Rgb-t object tracking: benchmark and baseline. Pattern Recogn 96:106977

    Article  Google Scholar 

  15. Li H, Wu X -J (2019) Densefuse: a fusion approach to infrared and visible images. IEEE Trans Image Process 28(5):2614–2623

    Article  MathSciNet  Google Scholar 

  16. Li H, Wu X -J, Durrani T (2020) Nestfuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models. IEEE Trans Instrum Meas 69(12):9645–9656

    Article  Google Scholar 

  17. Li H, Wu X -J, Kittler J (2021) Rfn-nest: an end-to-end residual fusion network for infrared and visible images. Inf Fusion 73:72–86

    Article  Google Scholar 

  18. Ma J, Zhou Y (2020) Infrared and visible image fusion via gradientlet filter. Comput Vis Image Underst 197–198:103016

    Article  Google Scholar 

  19. Ma J, Liang P, Yu W, Chen C, Jiang J (2019) Infrared and visible image fusion via detail preserving adversarial learning. Inf Fusion 54:85–98

    Article  Google Scholar 

  20. Ma J, Xu H, Jiang J, Mei X, Zhang X -P (2020) Ddcgan: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans Image Process 29:1–1

    Article  MathSciNet  MATH  Google Scholar 

  21. Ma J, Tang L, Xu M, Zhang H, Xiao G (2021) Stdfusionnet: an infrared and visible image fusion network based on salient target detection. IEEE Trans Instrum Meas 70:1–13

    Google Scholar 

  22. Ma J, Zhang H, Shao Z, Liang P, Xu H (2021) Ganmcc: a generative adversarial network with multi-classification constraints for infrared and visible image fusion. IEEE Trans Instrum Meas 70:5005014

    Google Scholar 

  23. Naidu V (2011) Image fusion technique using multi-resolution singular value decomposition. Def Sci J 61(5):479–484

    Article  MathSciNet  Google Scholar 

  24. Peng C, Tian T, Chen C, Guo X, Ma J (2021) Bilateral attention decoder: a lightweight decoder for real-time semantic segmentation. Neural Netw 137:188–199

    Article  Google Scholar 

  25. Shah P, Merchant S N, Desai U B (2013) Multifocus and multispectral image fusion based on pixel significance using multiresolution decomposition. SIViP 7(1):95–109

    Article  Google Scholar 

  26. Sun Y, Zuo W, Liu M (2019) RTFNEt: RGB-thermal fusion network for semantic segmentation of urban scenes. IEEE Robot Autom Lett 4 (3):2576–2583

    Article  Google Scholar 

  27. Tang L, Yuan J, Ma J (2022) Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network. Inf Fusion 82:28–42

    Article  Google Scholar 

  28. Tang L, Yuan J, Zhang H, Jiang X, Ma J (2022) Piafusion: a progressive infrared and visible image fusion network based on illumination aware. Inf Fusion 83-84:79–92

    Article  Google Scholar 

  29. Toet A (2014) TNO image fusion dataset. Figshare Dataset

  30. Wang H, Zhong W, Wang J (2004) Research of measurement for digital image definition. J Image Graph 9(7):828–831

    Google Scholar 

  31. Wang Z, Bovik A C, Sheikh H R, Simoncelli E P (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  32. Xu H, Zhang H, Ma J (2021) Classification saliency-based rule for visible and infrared image fusion. IEEE Trans Comput Imaging 7:824–836

    Article  MathSciNet  Google Scholar 

  33. Xu H, Ma J, Jiang J, Guo X, Ling H (2022) U2fusion: a unified unsupervised image fusion network. IEEE Trans Pattern Anal Mach Intell 44(1):502–518

    Article  Google Scholar 

  34. Xydeas C S, Petrović V (2000) Objective image fusion performance measure. Electron Lett 36(4):308–309

    Article  Google Scholar 

  35. Zhang Z, Liu Y, Tan H, Yi X, Zhang M (2018) No-reference image sharpness assessment using scale and directional models. In: 2018 IEEE International conference on multimedia and expo (ICME), pp 1–6

  36. Zhang Y, Liu Y, Sun P, Yan H, Zhao X, Zhang L (2020) Ifcnn: a general image fusion framework based on convolutional neural network. Inf Fusion 54:99–118

    Article  Google Scholar 

  37. Zhang X, Ye P, Xiao G (2020) Vifb: a visible and infrared image fusion benchmark. In: CVF Conference on computer vision and pattern recognition workshops, pp 468–478

  38. Zhang H, Xu H, Tian X, Jiang J, Ma J (2021) Image fusion meets deep learning: a survey and perspective. Inf Fusion 76:323–336

    Article  Google Scholar 

  39. Zhou K, Chen L, Cao X (2020) Improving multispectral pedestrian detection by addressing modality imbalance problems. In: Vedaldi A, Bischof H, Brox T, Frahm J-M (eds) Computer vision—ECCV 2020. Springer, pp 787–803

  40. Zhou H, Wu W, Zhang Y, Ma J, Ling H (2021) Semantic-supervised infrared and visible image fusion via a dual-discriminator generative adversarial network. IEEE Trans Multimed 1–1

Download references

Acknowledgments

This work has been supported by these following projects: (1) Grant No. D5120190078, National Science and technology projects of China. (2) Grant No. D5140190006, Key R & D projects of Shaanxi Province.

Author information

Authors and Affiliations

  1. School of Software, Northwestern Polytechnical University, 1 Dongxiang Road, Xi’an, 710129, Shaanxi, China

    Weigang Li & Junsheng Wu

  2. School of Computer Science and Engineering, Northwestern Polytechnical University, 1 Dongxiang Road, Xi’an, 710129, Shaanxi, China

    Aiqing Fang & Ying Li

Authors
  1. Weigang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aiqing Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junsheng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aiqing Fang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Raw data for the dataset is not publicly available to preserve individuals’ privacy under the Northwestern Polytechnical University Data Protection Regulation.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, W., Fang, A., Wu, J. et al. Quality and content-aware fusion optimization mechanism of infrared and visible images. Multimed Tools Appl 82, 47695–47717 (2023). https://doi.org/10.1007/s11042-023-15237-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15237-9

Keywords

Search

Navigation