Unsupervised non-rigid point cloud registration based on point-wise displacement learning | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

Unsupervised non-rigid point cloud registration based on point-wise displacement learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Registration of deformable objects is a fundamental prerequisite for many modern virtual reality and computer vision applications. However, due to the difficulties of acquiring labeled datasets and the inherent irregular deformation, non-rigid registration for 3D scanner-captured data remains challenging. This paper proposes an unsupervised non-rigid 3D point cloud registration network based on the self-attention mechanism. Specifically, considering the registration as the result of point drifts between the source and target shapes, a Transformer-based encoder-decoder module is utilized to estimate the point displacements. Additionally, a symmetric registration procedure is adopted with regularization loss to manage the regular deformation of points, ultimately producing reasonable registration results for real-world deformable objects. Experiments are conducted on public and synthesized datasets which simulate diversiform non-rigid 2D or 3D deformations. Numerical and qualitative experimental results demonstrate that the proposed network achieves outstanding performance and is robust in scenes with multiple interferences.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Availability of data and materials

The data that support the findings of this study are available from the corresponding author, D.Z., upon reasonable request.

Code Availability

The source code of this study is available from the corresponding author, D.Z., upon reasonable request.

Notes

  1. https://github.com/djzgroup/Non-rigid-Registration

References

  1. Agarwal S, Bhowmick B (2017) 3d point cloud registration with shape constraint. In: 2017 IEEE International Conference on Image Processing (ICIP), IEEE, pp 2199–2203

  2. Atzmon M, Maron H, Lipman Y (2018) Point convolutional neural networks by extension operators. arXiv:1803.10091

  3. Balakrishnan G, Zhao A, Sabuncu MR et al. (2018) An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9252–9260

  4. Bednarik J, Fua P, Salzmann M (2018) Learning to reconstruct texture-less deformable surfaces from a single view. In: 2018 International Conference on 3D Vision (3DV), IEEE pp 606–615

  5. Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures, Spie pp 586–606

  6. Bookstein FL (1989) Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 11(6):567–585

    Article  Google Scholar 

  7. Chui H, Rangarajan A (2000) A new algorithm for non-rigid point matching. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), IEEE pp 44–51

  8. Darbari A, Kumar K, Darbari S et al (2021) Requirement of artificial intelligence technology awareness for thoracic surgeons. The Cardiothoracic Surgeon 29(1):1–10

    Article  Google Scholar 

  9. Dosovitskiy A, Beyer L, Kolesnikov A et al. (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929

  10. Duan Y, Zheng Y, Lu J et al (2019) Structural relational reasoning of point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 949–958

  11. Guo MH, Cai JX, Liu ZN et al (2021) Pct: Point cloud transformer. Comput Vis Med 7(2):187–199

    Article  Google Scholar 

  12. Huang H, Wu S, Cohen-Or D et al (2013) L1-medial skeleton of point cloud. ACM Trans Graph 32(4):65–1

    Article  Google Scholar 

  13. Kumar K (2021) Text query based summarized event searching interface system using deep learning over cloud. Multimedia Tools and Applications 80(7):11,079–11,094

  14. Li K, Xiong H, Liu J et al. (2022) Real-time monocular joint perception network for autonomous driving. IEEE Trans Intell Transportation Syst 23(9):15,864–15,877

  15. Li X, Wang L, Fang Y (2019) Pc-net: Unsupervised point correspondence learning with neural networks. In: 2019 International Conference on 3D Vision (3DV), IEEE, pp 145–154

  16. Li Y, Bu R, Sun M et al. (2018) Pointcnn: Convolution on x-transformed points. Adv Neural Inf Process Syst 31

  17. Ma J, Zhao J, Jiang J et al. (2017) Non-rigid point set registration with robust transformation estimation under manifold regularization. In: Thirty-First AAAI Conference on Artificial Intelligence

  18. Myronenko A, Song X (2010) Point set registration: Coherent point drift. IEEE Trans Pattern Anal Mach Intell 32(12):2262–2275

    Article  PubMed  Google Scholar 

  19. Negi A, Kumar K (2021) Face mask detection in real-time video stream using deep learning. Computational intelligence and healthcare informatics, pp 255–268

  20. Qi CR, Su H, Mo K et al. (2017a) Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 652–660

  21. Qi CR, Yi L, Su H et al. (2017b) Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30

  22. Rocco I, Arandjelovic R, Sivic J (2017) Convolutional neural network architecture for geometric matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6148–6157

  23. Saini P, Kumar K, Kashid S et al (2023) Video summarization using deep learning techniques: a detailed analysis and investigation. Artificial Intelligence Review, pp 1–39

  24. Sarode V, Li X, Goforth H et al. (2019) Pcrnet: Point cloud registration network using pointnet encoding. arXiv:1908.07906

  25. Shimada S, Golyanik V, Tretschk E et al. (2019) Dispvoxnets: Non-rigid point set alignment with supervised learning proxies. In: 2019 International Conference on 3D Vision (3DV), IEEE pp 27–36

  26. Vaswani A, Shazeer N, Parmar N et al. (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  27. Verma P, Srivastava R (2020) Three stage deep network for 3d human pose reconstruction by exploiting spatial and temporal data via its 2d pose. J Vis Commun Image Representation 71(102):866

    Google Scholar 

  28. Verma P, Srivastava R (2022) Two-stage multi-view deep network for 3d human pose reconstruction using images and its 2d joint heatmaps through enhanced stack-hourglass approach. Vis Comput 38(7):2417–2430

    Article  Google Scholar 

  29. Wang L, Chen J, Li X et al. (2019a) Non-rigid point set registration networks. arXiv:1904.01428

  30. Wang L, Li X, Chen J et al. (2019b) Coherent point drift networks: Unsupervised learning of non-rigid point set registration. arXiv:1906.03039

  31. Wang Y, Sun Y, Liu Z et al (2019) Dynamic graph cnn for learning on point clouds. ACM Trans Graphics (tog) 38(5):1–12

    Article  Google Scholar 

  32. Wang Z, Delingette H (2021) Attention for image registration (air): an unsupervised transformer approach. arXiv:2105.02282

  33. Wu S, Huang H, Gong M et al (2015) Deep points consolidation. ACM Trans Graphics (ToG) 34(6):1–13

    Google Scholar 

  34. Wu Z, Song S, Khosla A et al. (2015b) 3d shapenets: A deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1912–1920

  35. Xiang R, Lai R, Zhao H (2021) A dual iterative refinement method for non-rigid shape matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 15,930–15,939

  36. Yang J, Li H, Campbell D et al (2015) Go-icp: A globally optimal solution to 3d icp point-set registration. IEEE Trans Pattern Anal Mach Intell 38(11):2241–2254

    Article  PubMed  Google Scholar 

  37. Yao Y, Deng B, Xu W et al. (2020) Quasi-newton solver for robust non-rigid registration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7600–7609

  38. Yew ZJ, Lee GH (2022) Regtr: End-to-end point cloud correspondences with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6677–6686

  39. Yin K, Huang H, Cohen-Or D et al (2018) P2p-net: Bidirectional point displacement net for shape transform. ACM Trans Graphics (TOG) 37(4):1–13

    Article  Google Scholar 

  40. Zhang D, He F, Tu Z et al (2020) Pointwise geometric and semantic learning network on 3d point clouds. Integrated Computer-Aided Eng 27(1):57–75

    Article  Google Scholar 

  41. Zhang J, Yao Y, Deng B (2021a) Fast and robust iterative closest point. IEEE Trans Pattern Anal Mach Intell

  42. Zhang Y, Wang X, Jiang X et al (2021) Marginalized graph self-representation for unsupervised hyperspectral band selection. IEEE Trans Geosci Remote Sens 60:1–12

    Google Scholar 

  43. Zhang Y, Wang Y, Chen X et al (2022) Spectral-spatial feature extraction with dual graph autoencoder for hyperspectral image clustering. IEEE Trans Circuits Syst Video Technol 32(12):8500–8511

    Article  Google Scholar 

  44. Zhang Z, Dai Y, Sun J (2020) Deep learning based point cloud registration: an overview. Virtual Reality Intell Hardware 2(3):222–246

    Article  Google Scholar 

  45. Zhao H, Jiang L, Jia J et al. (2021) Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 16,259–16,268

Download references

Funding

This work is supported by the National Natural Science Foundation of China (grant No. 61802355 and 61702350) and Hubei Key Laboratory of Intelligent Robot (HBIR 202105).

Author information

Authors and Affiliations

  1. School of Computer Science, China University of Geosciences, NO. 68 Jin Cheng Street, Wuhan, Hubei, 430078, China

    Yiqi Wu, Fang Han, Dejun Zhang & Tiantian Zhang

  2. Hubei Key Laboratory of Intelligent Robot, Wuhan Institute of Technology, No.206, Guanggu 1st road, Wuhan, Hubei, 430205, China

    Yilin Chen

Authors
  1. Yiqi Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fang Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dejun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tiantian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yilin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.W. and F.H. conceived and designed the algorithm and the experiments. F.H. analyzed the data. Y.W. and F.H. wrote the manuscript. D.Z. supervised the research. D.Z. and Y.C. provided suggestions for the proposed method and its evaluation and assisted in the preparation of the manuscript. T.Z. and Y.C. collected and sorted out the literature. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Dejun Zhang.

Ethics declarations

Conflicts of interest

The authors declared no potential conflicts of interest with respect to the research, authorship, or publication of this article.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Han, F., Zhang, D. et al. Unsupervised non-rigid point cloud registration based on point-wise displacement learning. Multimed Tools Appl 83, 24589–24607 (2024). https://doi.org/10.1007/s11042-023-16854-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16854-0

Keywords

Search

Navigation