A robust feature matching algorithm based on adaptive feature fusion combined with image superresolution reconstruction | Applied Intelligence Skip to main content
Springer Nature Link
Log in

A robust feature matching algorithm based on adaptive feature fusion combined with image superresolution reconstruction

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

With the development of image feature matching technology, feature matching algorithms based on deep learning have achieved excellent results, but in scenarios with low texture or extreme perspective changes, the matching accuracy is still difficult to guarantee. In this paper, a superresolution reconstruction method based on a Residual-ESPCN (efficient subpixel convolutional neural network) approach is proposed based on LoFTR (local feature matching with transformers). The superresolution method is used to improve the interpolation method used in ASFF (adaptive spatial feature fusion) to increase the image resolution, enhance the detailed information of the image, and make the extracted features richer. Then, ASFF is introduced into the local feature extraction module of LoFTR, which can alleviate the inconsistency problem of information transmission between different scale features of the feature pyramid and lessen the amount of information lost during transmission from low- to high-resolution levels. Moreover, to improve the adaptability of the algorithm to different scenarios, OTSU is introduced to adaptively calculate the threshold of feature matching. The experimental results show that in different indoor or outdoor scenarios, our proposed algorithm for matching features can effectively improve the adaptability of feature matching and can achieve good results in terms of the area under the curve (AUC), accuracy and recall.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The datasets used or analysed during the current study are available from the corresponding author upon reasonable request.

Code availability

The codes used during the current study are available from the corresponding author upon reasonable request.

References

  1. Zhang Y, Tosi F, Mattoccia S et al (2023) Go-slam: Global optimization for consistent 3d instant reconstruction[C]. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3727–3737

  2. Sharafutdinov D, Griguletskii M, Kopanev P, Kurenkov M, Ferrer G, Burkov A et al (2023) Comparison of modern open-source visual SLAM approaches. J Intell Rob Syst 107(3):43

    Article  Google Scholar 

  3. Pan T, Xu F, Yang X et al (2023) Boundary-aware backward-compatible representation via adversarial learning in image retrieval[C]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 15201–15210

  4. Van Hoorick B, Tokmakov P, Stent S et al. Tracking through containers and occluders in the Wild[C]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13802–13812

  5. Zhou Z, Tulsiani S (2023) Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction[C]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12588–12597

  6. Chen W, Liu Y, Wang W et al (2022) Deep learning for instance retrieval: A survey[J]. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp 7270–7292

  7. Gupta D K, Arya D, Gavves E (2021) Rotation equivariant siamese networks for tracking[C]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12362–12371

  8. Peng H (2021) Design of 3D image feature point detection system based on artificial intelligence[C]. In: Advanced Hybrid Information Processing: 4th EAI International Conference, ADHIP 2020, Binzhou, China, September 26-27, 2020, Proceedings, Part II 4. Springer International Publishing, pp 313–323

  9. Lin W, Zhang Z, Zhang L (2022) Infrared moving small target detection and tracking algorithm based on feature point matching. The European Physical Journal D 76(10):185

    Article  Google Scholar 

  10. Trajković M, Hedley M (1998) Fast corner detection. Image Vis Comput 16(2):75–87

    Article  Google Scholar 

  11. Lowe DG (1999) Object recognition from local scale-invariant features[C]. In: Proceedings of the Seventh IEEE International Conference on Computer Vision. IEEE 2:1150–1157

  12. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60:91–110

    Article  Google Scholar 

  13. Rublee E, Rabaud V, Konolige K et al (2011) ORB: An efficient alternative to SIFT or SURF[C]. In: 2011 International Conference on Computer Vision. IEEE pp 2564–2571

  14. Bay H, Tuytelaars T, Van Gool L (2006) Surf: Speeded up robust features[C]. In: Computer Vision–ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I 9. Springer Berlin Heidelberg, pp 404–417

  15. Harris C, Stephens M (1988) A combined corner and edge detector[C]. Alvey vision conference 15(50):10–5244

  16. Yi K M, Trulls E, Lepetit V et al (2016) Lift: Learned invariant feature transform[C]. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14. Springer International Publishing, pp 467–483

  17. Sarlin P E, DeTone D, Malisiewicz T et al (2020) Superglue: Learning feature matching with graph neural networks[C]. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4938–4947

  18. Dusmanu M, Rocco I, Pajdla T et al (2019) D2-net: A trainable cnn for joint description and detection of local features[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8092–8101

  19. Luo Z, Zhou L, Bai X, et al (2020) Aslfeat: Learning local features of accurate shape and localization[C]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6589–6598

  20. Tyszkiewicz M, Fua P, Trulls E (2020) DISK: Learning local features with policy gradient. Adv Neural Inf Process Syst 33:14254–14265

    Google Scholar 

  21. Hao W, Wang P, Ni C et al (2024) SuperGlue-based accurate feature matching via outlier filtering[J]. Vis Comput 40(5):3137–3150

  22. Li X, Han K, Li S, Prisacariu V (2020) Dual-resolution correspondence networks. Adv Neural Inf Process Syst 33:17346–17357

    Google Scholar 

  23. Rocco I, Cimpoi M, Arandjelović R, Torii A, Pajdla T, Sivic J (2018) Neighbourhood consensus networks. Adv Neural Inf Process Syst 31

  24. Rocco I, Arandjelović R, Sivic J (2020) Efficient neighbourhood consensus networks via submanifold sparse convolutions[C]. Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16. Springer International Publishing, pp 605–621

  25. Sun J, Shen Z, Wang Y, et al (2021) LoFTR: Detector-free local feature matching with transformers[C]. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8922–8931

  26. Peyré G, Cuturi M (2019) Computational optimal transport: With applications to data science[J]. Foundations and Trends® in Machine Learning 11(5–6):355–607

  27. Lin T Y, Dollár P, Girshick R et al (2017) Feature pyramid networks for object detection[C]. Proceedings of the IEEE Conference On Computer Vision and Pattern recognition, pp 2117–2125

  28. Tan M, Pang R, Le QV (2020) Efficientdet: Scalable and efficient object detection[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10781–10790

  29. Zhu M (2024) Dynamic feature pyramid networks for object detection[C]. Fifteenth International Conference on Signal Processing Systems (ICSPS 2023). SPIE 13091:503–511

  30. Guo C, Fan B, Zhang Q et al (2020) Augfpn: Improving multi-scale feature learning for object detection[C]. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12595–12604

  31. Yang G, Wang Z, Zhuang S (2021) PFF-FPN: a parallel feature fusion module based on FPN in pedestria detection[C]. 2021 International conference on computer engineering and artificial intelligence (ICCEAI). IEEE, pp 377–381

  32. Qiao S, Chen L C, Yuille A (2021) Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution[C]. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10213–10224

  33. Wang W, Xie E, Li X et al (2021) Pyramid vision transformer: A versatile backbone for dense prediction without convolutions[C]. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 568–578

  34. Wang Z, Chen J, Hoi SC (2020) Deep learning for image super-resolution: A survey. IEEE Trans Pattern Anal Mach Intell 43(10):3365–3387

    Article  Google Scholar 

  35. Prajapati K, Chudasama V, Patel H et al (2020) Unsupervised single image super-resolution network (USISResNet) for real-world data using generative adversarial network[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 464–465

  36. Zhang K, Liang J, Van Gool L et al (2021) Designing a practical degradation model for deep blind image super-resolution[C]. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 4791–4800

  37. Chen Z, Zhang Y, Gu J et al (2023) Dual aggregation transformer for image super-resolution[C]. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 12312–12321

  38. Wang Z, Gao G, Li J et al (2021) Lightweight image super-resolution with multi-scale feature interaction network[C]. In: 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, pp 1–6

  39. Li A, Zhang L, Liu Y et al (2023) Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 12514–12524

  40. Zhou Y, Li Z, Guo CL et al (2023) Srformer: Permuted self-attention for single image super-resolution[C]. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 12780–12791

  41. Zhang X, Li T, Zhao X (2023) Boosting single image super-resolution via partial channel shifting[C]. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 13223–13232

  42. Shi W, Caballero J, Huszár F et al (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]. Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1874–1883

  43. Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66

    Article  Google Scholar 

Download references

Funding

This work was partially supported by the China Postdoctoral Science Foundation (Grant No. 2021M702030) and Shandong Provincial Transportation Science and Technology Project (Grant No. 2021B120).

Author information

Authors and Affiliations

  1. School of Information Science and Electrical Engineering, Shandong Jiaotong University, Jinan, 250357, China

    Wenjun Huangfu, Cui Ni, Peng Wang & Yingying Zhang

  2. Institute of Automation, Shandong Academy of Sciences, Jinan, 250013, China

    Peng Wang

Authors
  1. Wenjun Huangfu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cui Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yingying Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Wenjun Huangfu: Conceptualization, Methodology, Software.

Peng Wang: Data curation, Writing–Original draft preparation.

Cui Ni: Visualization, Investigation.

Yingying Zhang: Supervision.

Corresponding authors

Correspondence to Cui Ni or Peng Wang.

Ethics declarations

Conflicts of interest/Competing interests

The authors declare that there are no conflicts of interest or competing interests regarding the publication of this article.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

The work described has not been published before, and its publication has been approved by the responsible authorities at the institution where the work is carried out.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huangfu, W., Ni, C., Wang, P. et al. A robust feature matching algorithm based on adaptive feature fusion combined with image superresolution reconstruction. Appl Intell 54, 8576–8591 (2024). https://doi.org/10.1007/s10489-024-05600-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05600-0

Keywords

Search

Navigation