Keyframe extraction for motion capture data via pose saliency and reconstruction error | The Visual Computer Skip to main content
Springer Nature Link
Log in

Keyframe extraction for motion capture data via pose saliency and reconstruction error

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Keyframes are a summary representation of motion capture data, which provide the basis for compression, retrieval, overview and reuse of motion capture data. In this paper, a new approach is proposed to extract keyframes from motion capture data. This approach uses the angle of rotation of limbs and the distance between joints as the feature representation of human movement and calculates limb saliency based on the multiscale saliency of each motion feature. Then the weighted sum of limb saliency is defined as pose saliency, and the frames corresponding to the local maxima on the pose saliency curve are extracted as the initial keyframes. Finally, guided by the initial keyframes, the optimal keyframes are extracted based on the reconstruction error optimization algorithm. Experiments demonstrate that this approach can effectively extract the keyframes with high visual perceptual quality and low reconstruction error, and better meet the needs of real-time analysis and compression of motion capture data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Guo, S., Southern, R., Chang, J., Greer, D., Zhang, J.J.: Adaptive motion synthesis for virtual characters: a survey. Vis. Comput. 31(5), 497–512 (2015). https://doi.org/10.1007/s00371-014-0943-4

    Article  Google Scholar 

  2. Arikan, O.: Compression of motion capture databases. ACM Trans Graph. 25(3), 890–897 (2006). https://doi.org/10.1145/1141911.1141971

    Article  MATH  Google Scholar 

  3. Costa, BF., Esperança, C.: Motion Capture Analysis and Reconstruction Using Spatial Keyframes. In: Cláudio AP, Bouatouch K, Chessa M, Paljic A, Kerren A, Hurter C, et al., editors. Computer Vision, Imaging and Computer Graphics Theory and Applications. Springer International Publishing; p. 48–70 (2020)

  4. Kim, Y., Lee, S.H.: Keyframe-based multi-contact motion synthesis. Vis. Comput. 37(7), 1949–1963 (2021). https://doi.org/10.1007/s00371-020-01956-9

    Article  Google Scholar 

  5. Assa, J., Caspi, Y., Cohen-Or, D.: Action synopsis: pose selection and illustration. ACM Trans Graph. 24(3), 667–676 (2005). https://doi.org/10.1145/1073204.1073246

    Article  Google Scholar 

  6. Qi, T., Feng, Y.F., Xiao, J., Zhuang, Y.T., Yang, X.S., Zhang, J.J.: A semantic feature for human motion retrieval. Comput. Anim. Virt. Worlds 24(3–4), 399–407 (2013). https://doi.org/10.1002/cav.1505

    Article  Google Scholar 

  7. Xia, G., Xue, P., Zhang, D., Liu, Q.: Keyframe-editable real-time motion synthesis. IEEE Trans. Circuits Syst. Video Technol. 32(7), 4538–4551 (2022). https://doi.org/10.1109/TCSVT.2021.3129478

    Article  Google Scholar 

  8. Ik Soo, L., Thalmann, D.: Key-posture extraction out of human motion data by curve simplification. In: 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society. vol. 2; p. 1167–1169 vol.2 (2001)

  9. Togawa, H., Okuda, M.: Position-Based Keyframe Selection for Human Motion Animation. In: 11th International Conference on Parallel and Distributed Systems (ICPADS’05). vol. 2; p. 182–185 (2005)

  10. Xiao, J., Zhuang, Y., Yang, T., Wu, F.: An Efficient keyframe extraction from motion capture data. In: Nishita, T., Peng, Q., Seidel, H.P. (eds.) Advances in Computer Graphics, pp. 494–501. Springer, Berlin, Heidelberg (2006)

    Chapter  Google Scholar 

  11. Liu, Y., Liu, J.: Keyframe extraction from motion capture data by optimal reconstruction error. J. Comput.-Aided Des. Graph. 22(4), 670–675 (2010)

    Article  Google Scholar 

  12. Miura, T., Kaiga, T., Shibata, T., Katsura, H., Tajima, K., Tamamoto, H.: A hybrid approach to keyframe extraction from motion capture data using curve simplification and principal component analysis. IEEE J. Trans. Electr. Electron. Eng. 9(6), 697–699 (2014). https://doi.org/10.1002/tee.22029

    Article  Google Scholar 

  13. Xu, C.X., Yu, W.J., Li, Y.R., Lu, X.Q., Wang, M.L., Yang, X.S.: KeyFrame extraction for human motion capture data via multiple binomial fitting. Comput. Anim. Virt. Worlds (2021). https://doi.org/10.1002/cav.1976

    Article  Google Scholar 

  14. Huang, K.S., Chang, C.F., Hsu, Y.Y., Yang, S.N.: Key Probe: a technique for animation keyframe extraction. Vis. Comput. 21(8), 532–541 (2005). https://doi.org/10.1007/s00371-005-0316-0

    Article  Google Scholar 

  15. Jin, C., Fevens, T., Mudur, S.: Optimized keyframe extraction for 3D character animations. Comput. Anim. Virt. Worlds 23(6), 559–568 (2012). https://doi.org/10.1002/cav.1471

    Article  Google Scholar 

  16. Liu, Xm., Hao, Am., Zhao, D.: Optimization-based key frame extraction for motion capture animation. Vis. Comput. 29(1), 85–95 (2013). https://doi.org/10.1007/s00371-012-0676-1

    Article  Google Scholar 

  17. Zhang, Q., Zhang, S.L., Zhou, D.S.: Keyframe extraction from human motion capture data based on a multiple population genetic algorithm. Symm.-Basel. 6(4), 926–937 (2014). https://doi.org/10.3390/sym6040926

    Article  Google Scholar 

  18. Chang, XJ., Yi, PF., Zhang, Q., Król, D., Madeyski, L., Nguyen, NT.: editors.: Key Frames Extraction from Human Motion Capture Data Based on Hybrid Particle Swarm Optimization Algorithm [Conference Paper]. Springer International Publishing

  19. Roberts, R., Lewis, J.P., Anjyo, K., Seo, J., Seol, Y.: Optimal and interactive keyframe selection for motion capture. Comput. Vis. Media. 5(2), 171–191 (2019). https://doi.org/10.1007/s41095-019-0138-z

    Article  Google Scholar 

  20. Xia, G.Y., Sun, H.J., Niu, X.Q., Zhang, G.Q., Feng, L.: Keyframe extraction for human motion capture data based on joint Kernel sparse representation. IEEE Trans. Industr. Electron. 64(2), 1589–1599 (2017). https://doi.org/10.1109/TIE.2016.2610946

    Article  Google Scholar 

  21. Xia, G.Y., Chen, B.J., Sun, H.J., Liu, Q.S.: Nonconvex low-rank kernel sparse subspace learning for keyframe extraction and motion segmentation. IEEE Trans. Neural Netw. Learn. Syst. 32(4), 1612–1626 (2021). https://doi.org/10.1109/TNNLS.2020.2985817

    Article  MathSciNet  Google Scholar 

  22. Liu, F., Zhuang, Y., Wu, F., Pan, Y.: 3D motion retrieval with motion index tree. Comput. Vis. Image Underst. 92(2–3), 265–284 (2003). https://doi.org/10.1016/j.cviu.2003.06.001

    Article  Google Scholar 

  23. Park, M.J., Shin, S.Y.: Example-based motion cloning. Comput. Anim. Virt. Worlds. 15(3–4), 245–257 (2004). https://doi.org/10.1002/cav.27

    Article  Google Scholar 

  24. Zhang, Q., Yu, S.P., Zhou, D.S., Wei, X.P.: An efficient method of key-frame extraction based on a cluster algorithm. J. Hum. Kinet. 39(1), 5–13 (2013). https://doi.org/10.2478/hukin-2013-0063

    Article  Google Scholar 

  25. Sun, B., Kong, D., Wang, S., Li, J., Keyframe extraction for human motion capture data based on affinity propagation. In: IEEE 9th annual information technology. Electronics and Mobile Communication Conference (IEMCON) 2018, 107–112 (2018)

  26. Voulodimos, A., Rallis, I., Doulamis, N.: Physics-based keyframe selection for human motion summarization. Multimed. Tools Appl. 79(5–6), 3243–3259 (2020). https://doi.org/10.1007/s11042-018-6935-z

    Article  Google Scholar 

  27. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998). https://doi.org/10.1109/34.730558

    Article  Google Scholar 

  28. Lee, C.H., Varshney, A., Jacobs, D.W.: Mesh saliency. ACM Trans Graph. 24(3), 659–666 (2005). https://doi.org/10.1145/1073204.1073244

    Article  Google Scholar 

  29. Bulut, E., Capin, T.: Key Frame Extraction from Motion Capture Data by Curve Saliency. In: CASA 2007 – Computer Animation and Social Agents. Hassel University Press; p. 63–67 (2007)

  30. Halit, C., Capin, T.: Multiscale motion saliency for keyframe extraction from motion capture sequences. Comput Anim. Virt Worlds 22(1), 3–14 (2011). https://doi.org/10.1002/cav.380

    Article  Google Scholar 

  31. Müller, M., Röder, T., Clausen, M.: Efficient content-based retrieval of motion capture data. ACM Trans Graph. 24(3), 677–685 (2005). https://doi.org/10.1145/1073204.1073247

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematics and Computer Science, Nanchang University, Nanchang, 330031, Jiangxi, China

    Yungen Liu, Linfeng Chen & Zhenrong Lin

Authors
  1. Yungen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linfeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenrong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenrong Lin.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Chen, L. & Lin, Z. Keyframe extraction for motion capture data via pose saliency and reconstruction error. Vis Comput 39, 4943–4953 (2023). https://doi.org/10.1007/s00371-022-02639-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02639-3

Keywords

Search

Navigation