Increasing pose comprehension through augmented reality reenactment | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

Increasing pose comprehension through augmented reality reenactment

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Standard video does not capture the 3D aspect of human motion, which is important for comprehension of motion that may be ambiguous. In this paper, we apply augmented reality (AR) techniques to give viewers insight into 3D motion by allowing them to manipulate the viewpoint of a motion sequence of a human actor using a handheld mobile device. The motion sequence is captured using a single RGB-D sensor, which is easier for a general user, but presents the unique challenge of synthesizing novel views using images captured from a single viewpoint. To address this challenge, our proposed system reconstructs a 3D model of the actor, then uses a combination of the actor’s pose and viewpoint similarity to find appropriate images to texture it. The system then renders the 3D model on the mobile device using visual SLAM to create a map in order to use it to estimate the mobile device’s camera pose relative to the original capturing environment. We call this novel view of a moving human actor a reenactment, and evaluate its usefulness and quality with an experiment and a survey.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. PTAMM is a version of Parallel Tracking and Mapping [16], which is a visual SLAM system.

  2. A visual comparison of the two rendering methods can be found at http://yokoya.naist.jp/fabian-d/arreenactment.htm

References

  1. Alexiadis DS, Zarpalas D, Daras P (2013) Real-time, full 3-D reconstruction of moving foreground objects from multiple consumer depth cameras. IEEE Trans Multimed 15(2):339–358

    Article  Google Scholar 

  2. Anderson F, Grossman T, Matejka J, Fitzmaurice G (2013) YouMove: Enhancing movement training with an augmented reality mirror. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp 311–320

  3. Azuma RT (1997) A survey of augmented reality. Presence 6(4):355–385

    Article  Google Scholar 

  4. Beck S, Kunert A, Kulik A, Froehlich B (2013) Immersive group-to-group telepresence. IEEE Trans Vis Comput Graph 19(4):616–625

    Article  Google Scholar 

  5. Carranza J, Theobalt C, Magnor M, Seidel H (2003) Free-viewpoint video of human actors. ACM Trans Graph 22(3):569–577

    Article  Google Scholar 

  6. Castle R, Klein G, Murray D (2008) Video-rate localization in multiple maps for wearable augmented reality. In: Proceedings of the IEEE International Symposium on Wearable Computers, pp 15–22

  7. Dai B, Yang X (2013) A low-latency 3D teleconferencing system with image based approach. In: Proceedings of the ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp 243–248

  8. Dayrit FL, Nakashima Y, Sato T, Yokoya N (2014) Free-viewpoint AR human-motion reenactment based on a single RGB-D video stream. In: Proceedings of the IEEE International Conference on Multimedia and Expo

  9. Debevec P, Taylor C, Malik J (1996) Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In: Proceedings of the ACM SIGGRAPH, pp 11–20

  10. de Aguiar E, Stoll C, Theobalt C, Ahmed N, Seidel H, Thrun S (2008) Performance capture from sparse multi-view video. ACM Trans Graph 27(3)

  11. Hauswiesner S, Straka M, Reitmayr G (2011) Image-based clothes transfer. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pp 169–172

  12. Henderson S, Feiner S (2011) Augmented reality in the psychomotor phase of a procedural task. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pp 191–200

  13. Hilsmann A, Fechteler P, Eisert P (2013) Pose space image based rendering. In: Proceedings of the Computer Graphics Forum, vol 32, pp 265–274

  14. Hondori H, Khademi M, Dodakian L, Cramer S, Lopes CV (2013) A spatial augmented reality rehab system for post-stroke hand rehabilitation. In: Proceedings of the Conference on Medicine Meets Virtual Reality, pp 279–285

  15. Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A (2011) KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp 559–568

  16. Klein G, Murray D (2007) Parallel tracking and mapping for small AR workspaces. In: Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality

  17. Lorensen W, Cline H (1987) Marching cubes: A high resolution 3D surface construction algorithm. In: Proceedings of the ACM SIGGRAPH, vol 21, pp 163–169

  18. Malleson C, Klaudiny M, Hilton A, Guillemaut JY (2013) Single-view RGBD-based reconstruction of dynamic human geometry. In: Proceedings of the International Workshop on Dynamic Shape Capture and Analysis, pp 307–314

  19. Matusik W, Buehler C, Raskar R, Gortler S, McMillan L (2000) Image-based visual hulls. In: Proceedings of the ACM SIGGRAPH, pp 369–374

  20. Pagés R, Berjón D, Morán F (2013) Automatic system for virtual human reconstruction with 3D mesh multi-texturing and facial enhancement. Signal Process Image Commun 28(9):1089–1099

    Article  Google Scholar 

  21. Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M, Moore R (2013) Real-time human pose recognition in parts from single depth images. Commun ACM 56(1):116–124

    Article  Google Scholar 

  22. Shum H, Kang SB (2000) Review of image-based rendering techniques. Visual Communications and Image Processing:2–13

  23. Velloso E, Bulling A, Gellersen H (2013) MotionMA: Motion modelling and analysis by demonstration. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp 1309–1318

  24. Wang Z, Ong S, Nee A (2013) Augmented reality aided interactive manual assembly design. Int J Adv Manuf Technol 69(5-8):1311–1321

    Article  Google Scholar 

  25. Waschbüsch M, Würmlin S, Cotting D, Sadlo F, Gross M (2005) Scalable 3D video of dynamic scenes. Vis Comput 21(8-10):629–638

    Article  Google Scholar 

  26. Würmlin S, Lamboray E, Staadt O, Gross M (2002) 3D video recorder. In: Proceedings of the Pacific Conference on Computer Graphics and Applications, pp 325–334

  27. Xu F, Liu Y, Stoll C, Tompkin J, Bharaj G, Dai Q, Seidel HP, Kautz J, Theobalt C (2011) Video-based characters: creating new human performances from a multi-view video database. ACM Trans Graph 30(4)

  28. Yamabe T, Nakajima T (2013) Playful training with augmented reality games: case studies towards reality-oriented system design. Multimed Tools Appl 62(1):259–286

    Article  Google Scholar 

  29. Ye G, Liu Y, Deng Y, Hasler N, Ji X, Dai Q, Theobalt C (2013) Free-viewpoint video of human actors using multiple handheld Kinects. IEEE Trans Cybern 43(5):1370–1382

    Article  Google Scholar 

  30. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  31. Zhou Z, Shu B, Zhuo S, Deng X, Tan P, Lin S (2012) Image-based clothes animation for virtual fitting. In: Proceedings of the ACM SIGGRAPH Asia

  32. Zitnick C, Kang S, Uyttendaele M, Winder S, Szeliski R (2004) High-quality video view interpolation using a layered representation. ACM Trans Graph 23(3):600–608

    Article  Google Scholar 

Download references

Acknowledgments

This work was partially supported by JSPS Grant-in-Aid for Scientific Research Nos. 23240024 and 25540086.

Author information

Authors and Affiliations

  1. Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara, 630-0192, Japan

    Fabian Lorenzo Dayrit, Yuta Nakashima, Tomokazu Sato & Naokazu Yokoya

Authors
  1. Fabian Lorenzo Dayrit
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuta Nakashima
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tomokazu Sato
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Naokazu Yokoya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabian Lorenzo Dayrit.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dayrit, F.L., Nakashima, Y., Sato, T. et al. Increasing pose comprehension through augmented reality reenactment. Multimed Tools Appl 76, 1291–1312 (2017). https://doi.org/10.1007/s11042-015-3116-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-015-3116-1

Keywords

Search

Navigation