Abstract
With the development of satellite technology and airborne platforms, there are more and more methods to acquire remote sensing data. The remote sensing data acquired by multiple methods contain different information and internal structures. Nowadays, single-mode hyperspectral image (HSI) data are no longer satisfactory for researchers’ needs. How to apply and process the information of multimodal data poses a great challenge to researchers. In this paper, we propose a deep learning-based network framework for multimodal remote sensing data classification, where we construct an advanced cross-stage fusion strategy using a fully connected network as the backbone, called CSF. Like the name implies, CSF incorporated two separate stages of fusion strategies for more effective fusion of multimodal data: fusion at the pre-structure and fusion at the tail of the network. This strategy prevents the preservation of excessive redundant information in the pre-fusion and the details of information lost due to late fusion. Moreover, a plug-and-play cross-fusion module for CSF is implemented. On the Houston 2013 dataset, our model strategy outperformed the fusion strategy of each stage and the single-modal strategy, which also demonstrated that multimodal feature fusion has promising performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Liang, J., Zhou, J., Tong, L., Bai, X., Wang, B.: Material based salient object detection from hyperspectral images. Pattern Recognit. 76, 476–490 (2018). https://doi.org/10.1016/j.patcog.2017.11.024
Gao, B., et al.: Additional sampling layout optimization method for environmental quality grade classifications of farmland soil. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 10, 5350–5358 (2017). https://doi.org/10.1109/JSTARS.2017.2753467
Zadeh, M.H., Tangestani, M.H., Roldan, F.V., Yusta, I.: Mineral exploration and alteration zone mapping using mixture tuned matched filtering approach on ASTER Data at the Central Part of Dehaj-Sarduiyeh Copper Belt, SE Iran. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 7, 284–289 (2014). https://doi.org/10.1109/JSTARS.2013.2261800
Lu, B., Dao, P., Liu, J., He, Y., Shang, J.: Recent advances of hyperspectral imaging technology and applications in agriculture. Remote Sens. 12, 2659 (2020). https://doi.org/10.3390/rs12162659
Kobryn, H.T., Wouters, K., Beckley, L.E., Heege, T.: Ningaloo reef: shallow marine habitats mapped using a hyperspectral sensor. PLoS ONE 8, e70105 (2013). https://doi.org/10.1371/journal.pone.0070105
Tang, C., Liu, X., Zhu, E., Wang, L., Zomaya, A.: Hyperspectral band selection via spatial-spectral weighted region-wise multiple graph fusion-based spectral clustering. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 3038–3044. International Joint Conferences on Artificial Intelligence Organization, Montreal, Canada (2021). https://doi.org/10.24963/ijcai.2021/418
Chakraborty, T., Trehan, U.: SpectralNET: Exploring Spatial-Spectral WaveletCNN for Hyperspectral Image Classification. http://arxiv.org/abs/2104.00341 (2021)
Guo, Z., Zhang, L., Zhang, D.: A completed modeling of local binary pattern operator for texture classification. IEEE Trans. Image Process. 19, 1657–1663 (2010). https://doi.org/10.1109/TIP.2010.2044957
Liu, L., Lao, S., Fieguth, P.W., Guo, Y., Wang, X., Pietikäinen, M.: Median robust extended local binary pattern for texture classification. IEEE Trans. Image Process. 25, 1368–1381 (2016). https://doi.org/10.1109/TIP.2016.2522378
Camps-Valls, G., Bruzzone, L.: Kernel-based methods for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 43, 1351–1362 (2005). https://doi.org/10.1109/TGRS.2005.846154
Jia, S., et al.: Flexible gabor-based superpixel-level unsupervised LDA for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 59, 10394–10409 (2021). https://doi.org/10.1109/TGRS.2020.3048994
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141. IEEE, Salt Lake City, UT (2018). https://doi.org/10.1109/CVPR.2018.00745
Zhang, N., Li, J., Li, Y., Du, Y.: Global attention pyramid network for semantic segmentation. In: 2019 Chinese Control Conference (CCC), pp. 8728–8732 (2019). https://doi.org/10.23919/ChiCC.2019.8865946
Zhang, M., Li, W., Du, Q., Gao, L., Zhang, B.: Feature extraction for classification of hyperspectral and LiDAR data using patch-to-patch CNN. IEEE Trans. Cybern. 50, 100–111 (2020). https://doi.org/10.1109/TCYB.2018.2864670
You, X., Xu, J., Yuan, W., Jing, X.-Y., Tao, D., Zhang, T.: Multi-view common component discriminant analysis for cross-view classification. Pattern Recognit. 92, 37–51 (2019). https://doi.org/10.1016/j.patcog.2019.03.008
Jia, S., et al.: Multiple feature-based superpixel-level decision fusion for hyperspectral and LiDAR data classification. IEEE Trans. Geosci. Remote Sens. 59, 1437–1452 (2021). https://doi.org/10.1109/TGRS.2020.2996599
Wu, X., Hong, D., Chanussot, J.: Convolutional neural networks for multimodal remote sensing data classification. IEEE Trans. Geosci. Remote Sens. 60, 1 (2022). https://doi.org/10.1109/TGRS.2021.3124913
Hong, D., et al.: More diverse means better: multimodal deep learning meets remote-sensing imagery classification. IEEE Trans. Geosci. Remote Sens. 59, 4340–4354 (2021). https://doi.org/10.1109/TGRS.2020.3016820
Hang, R., Li, Z., Ghamisi, P., Hong, D., Xia, G., Liu, Q.: Classification of hyperspectral and LiDAR data using coupled CNNs. IEEE Trans. Geosci. Remote Sens. 58, 4939–4950 (2020). https://doi.org/10.1109/TGRS.2020.2969024
Marcos, D., Volpi, M., Kellenberger, B., Tuia, D.: Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models. ISPRS J. Photogramm. Remote Sens. 145, 96–107 (2018). https://doi.org/10.1016/j.isprsjprs.2018.01.021
Marmanis, D., Wegner, J.D., Galliani, S., Schindler, K., Datcu, M., Stilla, U.: Semantic Segmentation of Aerial Images with an Ensemble of CNNS. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, pp. 473–480. Copernicus GmbH (2016). https://doi.org/10.5194/isprs-annals-III-3-473-2016
Audebert, N., Le Saux, B., Lefèvre, S.: Beyond RGB: very high resolution urban remote sensing with multimodal deep networks. ISPRS J. Photogramm. Remote Sens. 140, 20–32 (2018). https://doi.org/10.1016/j.isprsjprs.2017.11.011
Hong, D., Hu, J., Yao, J., Chanussot, J., Zhu, X.X.: Multimodal remote sensing benchmark datasets for land cover classification with a shared and specific feature learning model. ISPRS J. Photogramm. Remote Sens. 178, 68–80 (2021). https://doi.org/10.1016/j.isprsjprs.2021.05.011
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62071157, National Key Research and Development Programme 2022YFD2000500 and Natural Science Foundation of Heilongjiang Province under Grant YQ2019F011.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Sun, Y., Wang, Z., Li, A., Jiang, H. (2023). Cross-Stage Fusion Network Based Multi-modal Hyperspectral Image Classification. In: Li, A., Shi, Y., Xi, L. (eds) 6GN for Future Wireless Networks. 6GN 2022. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 504. Springer, Cham. https://doi.org/10.1007/978-3-031-36011-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-36011-4_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36010-7
Online ISBN: 978-3-031-36011-4
eBook Packages: Computer ScienceComputer Science (R0)