FusionCNN: a remote sensing image fusion algorithm based on deep convolutional neural networks | Multimedia Tools and Applications Skip to main content

Advertisement

Springer Nature Link
Log in

FusionCNN: a remote sensing image fusion algorithm based on deep convolutional neural networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In remote sensing image fusion field, traditional algorithms based on the human-made fusion rules are severely sensitive to the source images. In this paper, we proposed an image fusion algorithm using convolutional neural networks (FusionCNN). The fusion model implicitly represents a fusion rule whose inputs are a pair of source images and the output is a fused image with end-to-end property. As no datasets can be used to train FusionCNN in remote sensing field, we constructed a new dataset from a natural image set to approximate MS and Pan images. In order to obtain higher fusion quality, low frequency information of MS is used to enhance the Pan image in the pre-processing step. The method proposed in this paper overcomes the shortcomings of the traditional fusion methods in which the fusion rules are artificially formulated, because it learns an adaptive strong robust fusion function through a large amount of training data. In this paper, Landsat and Quickbird satellite data are used to verify the effectiveness of the proposed method. Experimental results show that the proposed fusion algorithm is superior to the comparative algorithms in terms of both subjective and objective evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Amro I, Mateos J, Vega M et al (2011) A survey of classical methods and new trends in pansharpening of multispectral images. EURASIP J Adv Signal Process 79:1–22. https://doi.org/10.1186/1687-6180-2011-79

    Google Scholar 

  2. Burt PJ, Adelson EH (1983) The Laplacian pyramid as a compact image code. IEEE Trans Commun COM-31(4):532–540

    Article  Google Scholar 

  3. Cheng J, Liu H, Liu T et al (2015) Remote sensing image fusion via wavelet transform and sparse representation. ISPRS J Photogramm Remote Sens 104:158–173

    Article  Google Scholar 

  4. Choi M, Kim RY, Nam MR, Kim HO (2005) Fusion of multispectral and panchromatic satellite images using the curvelet transform. IEEE Geosci Remote Sens Lett 2(2):136–140

    Article  Google Scholar 

  5. Chu H, Zhu W (2008) Fusion of IKONOS satellite imagery using IHS transform and local variation. IEEE Geosci Remote Sens Lett 5(4):653–657

    Article  MathSciNet  Google Scholar 

  6. Collobert R (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12:2493–2537

    MATH  Google Scholar 

  7. Do MN, Vetterli M (2005) The contourlet transform: an efficient directional multiresolution image representation. IEEE Trans Image Process 14(12):2091–2106

    Article  Google Scholar 

  8. Fan C, Wang L, Liu P et al (2016) Compressed sensing based remote sensing image reconstruction via employing similarities of reference images. Multimed Tools Appl 75(19):12201–12225

    Article  Google Scholar 

  9. Gangkofner UG, Pradhan PS, Holcomb DW (2008) Optimizing the high pass filter addition technique for image fusion. Photogramm Eng Remote Sens 74(9):1107–1118

    Article  Google Scholar 

  10. Ghahremani M, Ghassemian H (2015) Remote-sensing image fusion based on Curvelets and ICA. Int J Remote Sens 36(16):4131–4143

    Article  Google Scholar 

  11. Ghassemian H (2016) A review of remote sensing image fusion methods. Inform Fusion 32(PA):75–89

    Article  Google Scholar 

  12. Global Land Cover Facility. http://www.landcover.org/. Accessed 11 Nov 2018

  13. González-Audícana M, Saleta JL, Catalán RG et al (2004) Fusion of multispectral and panchromatic images using improved IHS and PCA mergers based on wavelet decomposition. IEEE Trans Geosci Remote Sens 42(6):1291–1299

    Article  Google Scholar 

  14. Hinton GE, OsinderoS TYW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554. https://doi.org/10.1162/neco.2006.18.7.1527

    Article  MathSciNet  MATH  Google Scholar 

  15. Hnatushenko VV, Vasyliev VV (2016) Remote sensing image fusion using Ica and optimized wavelet transform. Int Arch Photogramm Remote Sens Spat Inf Sci XLI-B7:653–659

    Article  Google Scholar 

  16. Ji X, Zhang G (2017) Image fusion method of SAR and infrared image based on Curvelet transform with adaptive weighting. Multimed Tools Appl 76(17):17633–17649

    Article  Google Scholar 

  17. Kong WW, Lei YJ, Lei Y et al (2011) Image fusion technique based on non-subsampled contourlet transform and adaptive unit-fast-linking pulse-coupled neural network. IET Image Process 5(2):113–121

    Article  Google Scholar 

  18. Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional neural networks. In proc. Adv Neural Inf Proces Syst 25:1090–1098

    Google Scholar 

  19. Liu Y, Wang Z (2014) A practical pan-sharpening method with wavelet transform and sparse representation. IEEE international conference on imaging systems and techniques, 288–293

  20. Luo Y, Liu R, Zhu YF (2011) Fusion of remote sensing image base on the PCA + ATROUS wavelet transform. Appl Mech Mater 353-356:172–176

    Article  Google Scholar 

  21. Nakazawa T, Kulkarni DV (2018) Wafer map defect pattern classification and image retrieval using convolutional neural network. IEEE Trans Semicond Manuf 31(2):309–314

    Article  Google Scholar 

  22. Paramanandham N, Rajendiran K (2017) Multi sensor image fusion for surveillance applications using hybrid image fusion algorithm. Multimed Tools Appl. https://doi.org/10.1007/s11042-017-4895-3

  23. Park CC, Kim Y, Kim G (2018) Retrieval of sentence sequences for an image stream via coherence recurrent convolutional networks. IEEE Trans Pattern Anal Mach Intell 40(4):945–957

    Article  Google Scholar 

  24. PohlC V (1998) Multisensor image fusion in remote sensing: concepts, methods and applications. Int J Remote Sens 19(5):823–854

    Article  Google Scholar 

  25. Shah VP, Younan NH, King RL (2008) An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Trans Geosci Remote Sens 46(5):1323–1335

    Article  Google Scholar 

  26. Shahdoosti HR, Ghassemian H (2015) Fusion of MS and PAN images preserving spectral quality. IEEE Geosci Remote Sens Lett 12(3):611–615

    Article  Google Scholar 

  27. Shahdoosti HR, Ghassemian H (2016) Combining the spectral PCA and spatial PCA fusion methods by an optimal filter. Information Fusion 27:150–160

    Article  Google Scholar 

  28. Shensa MJ (1992) The discrete wavelet transform: wedding the àtrous and Mallat algorithm. IEEE Trans Signal Process 40(10):2464–2482

    Article  MATH  Google Scholar 

  29. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. Adv Neural Inf Proces Syst 27:3104–3112

    Google Scholar 

  30. Tu TM, Su SC, Shyu HC et al (2001) A new look at ihs-like image fusion methods. Inform Fusion 2(3):177–186

    Article  Google Scholar 

  31. Tu TM, Huang PS, Hung CL et al (2004) A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci Remote Sens Lett 1(4):309–312

    Article  Google Scholar 

  32. Valizadeh SA, Ghassemian H (2012) Remote Sensing image fusion using combining HIS and Curvelet transform. In: International symposium on telecommunications, 1184–1189

  33. Wu B, Fu Q, Sun L et al (2015) Enhanced hyperspherical color space fusion technique preserving spectral and spatial content. J Appl Remote Sens 9(1):097291

    Article  Google Scholar 

  34. Yang Y, Wan W, Huang S et al (2017) Remote sensing image fusion based on adaptive IHS and multiscale guided filter. IEEE Access 4:4573–4582

    Article  Google Scholar 

  35. Zhang X, Li X, Feng Y (2017) Image fusion based on simultaneous empirical wavelet transform. Multimed Tools Appl 76(6):8175–8193

    Article  Google Scholar 

Download references

Acknowledgements

The work was supported by National Science & Technology Pillar Program of China (Grant No. 2012BAH48F02), National Natural Science Foundation of China (Grant No. 61272209, 61801190), Natural Science Foundation of Jilin Province (Grant No. 20180101055JC), Outstanding Young Talent Foundation of Jilin Province (Grant No. 20180520029JH) and China Postdoctoral Science Foundation (Grant No. 2017M611323). The authors would like to thank Dr. Shuang Yu for her help on technical editing of the manuscript, and Prof. Xiaoying Sun for scientific advices.

Author information

Authors and Affiliations

  1. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012, China

    Fajie Ye, Xiongfei Li & Xiaoli Zhang

  2. College of Computer Science and Technology, Jilin University, Changchun, 130012, China

    Fajie Ye, Xiongfei Li & Xiaoli Zhang

Authors
  1. Fajie Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiongfei Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoli Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoli Zhang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ye, F., Li, X. & Zhang, X. FusionCNN: a remote sensing image fusion algorithm based on deep convolutional neural networks. Multimed Tools Appl 78, 14683–14703 (2019). https://doi.org/10.1007/s11042-018-6850-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6850-3

Keywords

Search

Navigation