Mixed interaction: evaluating user interactions for object manipulations in virtual space | Journal on Multimodal User Interfaces Skip to main content
Springer Nature Link
Log in

Mixed interaction: evaluating user interactions for object manipulations in virtual space

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

This paper presents an evaluation of a potential new interaction mode in virtual reality (VR) to determine whether it provides any positive impact in terms of how users interact with content. We evaluated the user experiences for 3D object manipulation across three modes of interaction. Interaction using controllers and gestures are used as baselines from which to gauge the potential value of the new mode of interaction, where a single controller and gestures are combined. This paper reports on a user study that captures quantitative and qualitative data related to a variety of object manipulation tasks in a Virtual Environment (VE). We investigated the impact of this new interaction mode with 40 participants across a number of interaction tasks, with the quantitative evaluation indicating that generally, the mixed mode of interaction resulted in task completion times consistently faster than gesture-based interaction and, in some cases, faster than with the use of controllers alone. A qualitative evaluation of the user experience indicated potential application areas for the new mode of interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Araújo T, Santos C, Miranda B, Carneiro N, Marques A, Mota M, Meiguins B (2016) Aspects of voice interaction on a mobile augmented reality application. In: Lackey S, Shumaker R (eds) Virtual, augmented and mixed reality, vol 9740. Lecture Notes in Computer Science. Springer, Cham, pp 199–210

    Chapter  Google Scholar 

  2. Bossavit B, Marzo A, Ardaiz O, De Cerio LD, Pina A (2014) Design choices and their implications for 3D mid-air manipulation techniques. Presence 23(4):377–392. https://doi.org/10.1162/PRES_a_00207

    Article  Google Scholar 

  3. Bowman D, Coquillart S, Froehlich B, Hirose M, Kitamura Y, Kiyokawa K, Stuerzlinger W (2008) 3D user interfaces: new directions and perspectives. IEEE Comput Graphics Appl 28(6):20–36. https://doi.org/10.1109/MCG.2008.109

    Article  Google Scholar 

  4. Bowman DA, Gabbard JL, Hix D (2002) A survey of usability evaluation in virtual environments: classification and comparison of methods. Presence 11(4):404–424. https://doi.org/10.1162/105474602760204309

    Article  Google Scholar 

  5. Burdea G, Richard P, Coiffet P (1996) Multimodal virtual reality: input-output devices, system integration, and human factors. Int J Hum Comput Interact 8(1):5–24. https://doi.org/10.1080/10447319609526138

    Article  Google Scholar 

  6. Cabral MC, Morimoto CH, Zuffo MK (2005) On the usability of gesture interfaces in virtual reality environments. In: Proceedings of the 2005 Latin American conference on human-computer interaction—CLIHC’05. ACM Press, Cuernavaca, pp 100–108

  7. Caputo FM, Giachetti A (2015) Evaluation of basic object manipulation modes for low-cost immersive Virtual Reality. In: Proceedings of the 11th biannual conference on Italian SIGCHI chapter. ACM, Rome, pp 74–77

  8. Chang E, Kim H-T, Yoo B (2020) Virtual reality sickness: a review of causes and measurements. Int J Hum Comput Interact 36:1–25. https://doi.org/10.1080/10447318.2020.1778351

    Article  Google Scholar 

  9. Choi I, Ofek E, Benko H, Sinclair M, Holz C (2018). CLAW: a multifunctional handheld haptic controller for grasping, touching, and triggering in virtual reality. In: Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, Montreal QC, pp 1–13

  10. Costa D, Duarte C (2011) Adapting multimodal fission to user’s abilities. 6765:347-356

  11. Dinno A (2015) Nonparametric pairwise multiple comparisons in independent groups using Dunn’s test. Stata J 15(1):292–300. https://doi.org/10.1177/1536867X1501500117

    Article  Google Scholar 

  12. Drogemuller A, Cunningham A, Walsh J, Thomas BH, Cordeil M, Ross W (2020) Examining virtual reality navigation techniques for 3D network visualisations. J Comput Lang 56:100937. https://doi.org/10.1016/j.cola.2019.100937

    Article  Google Scholar 

  13. Foottit J, Brown D, Marks S, Connor A (2016) Development of a wearable haptic game interface. EAI Endor Trans Creat Technol 3:e5. https://doi.org/10.4108/eai.25-4-2016.151165

    Article  Google Scholar 

  14. Gerhard D, Norton WJ (2022) Virtual reality usability design, 1st edn. CRC Press, Boca Raton

    Book  Google Scholar 

  15. Glonek G, Pietruszka M (2012) Natural user interfaces (NUI): review. J Appl Comput Sci 20:27–45

    Google Scholar 

  16. Hansberger J, Peng C, Blakely V, Meacham S, Cao L, Diliberti N (2019) A multimodal interface for virtual information environments, pp 59–70

  17. Iqbal H, Latif S, Yan Y, Yu C, Shi Y (2021) Reducing arm fatigue in virtual reality by introducing 3d-spatial offset. IEEE Access 9:64085–64104. https://doi.org/10.1109/ACCESS.2021.3075769

    Article  Google Scholar 

  18. Iskander J, Hossny M, Nahavandi S (2018) A review on ocular biomechanic models for assessing visual fatigue in virtual reality. IEEE Access 6:19345–19361. https://doi.org/10.1109/ACCESS.2018.2815663

    Article  Google Scholar 

  19. Ivankova N, Wingo N (2018) Applying mixed methods in action research: methodological potentials and advantages. Am Behav Sci 62(7):978–997. https://doi.org/10.1177/0002764218772673

    Article  Google Scholar 

  20. Jacob RJK, Leggett JJ, Myers BA, Pausch R (1993) Interaction styles and input/output devices. Behav Inform Technol 12(2):69–79. https://doi.org/10.1080/01449299308924369

    Article  Google Scholar 

  21. Jang S, Vitale JM, Jyung RW, Black JB (2017) Direct manipulation is better than passive viewing for learning anatomy in a three-dimensional virtual reality environment. Comput Educ 106:150–165. https://doi.org/10.1016/j.compedu.2016.12.009

    Article  Google Scholar 

  22. Jeong S, Jung ES, Im Y (2016) Ergonomic evaluation of interaction techniques and 3D menus for the practical design of 3D stereoscopic displays. Int J Ind Ergon 53:205–218. https://doi.org/10.1016/j.ergon.2016.01.001

    Article  Google Scholar 

  23. Jerald J (2016) The VR book: human-centered design for virtual reality, No 8. Association for Computing Machinery Morgan & Claypool Publishers, New York

    Google Scholar 

  24. Jones S (2017) Disrupting the narrative: immersive journalism in virtual reality. J Media Pract 18(2–3):171–185. https://doi.org/10.1080/14682753.2017.1374677

    Article  Google Scholar 

  25. Kangas J, Kumar SK, Mehtonen H, Järnstedt J, Raisamo R (2022) Trade-off between task accuracy, task completion time and naturalness for direct object manipulation in virtual reality. Multimodal Technol Interact 6(1):6. https://doi.org/10.3390/mti6010006

    Article  Google Scholar 

  26. Kwon J, Kim J-Y, Nam S (2017) Designing 3D menu interfaces for spatial interaction in virtual environments. Int J Grid Distrib Comput 10(12):31–38. https://doi.org/10.14257/ijgdc.2017.10.12.04

    Article  Google Scholar 

  27. LaValle SM (2020) Virtual reality. Cambridge University Press, Cambridge

    Google Scholar 

  28. LaViola JJ (2000) A discussion of cybersickness in virtual environments. ACM SIGCHI Bull 32(1):47–56. https://doi.org/10.1145/333329.333344

    Article  Google Scholar 

  29. Li Y, Huang J, Tian F, Wang H-A, Dai G-Z (2019) Gesture interaction in virtual reality. Virtual Real Intell Hardw 1(1):84–112. https://doi.org/10.3724/SP.J.2096-5796.2018.0006

    Article  Google Scholar 

  30. Li Y, Wu D, Huang J, Tian F, Wang H, Dai G (2019) Influence of multi-modality on moving target selection in virtual reality. Virtual Real Intell Hardw 1(3):303–315. https://doi.org/10.3724/SP.J.2096-5796.2019.0013

    Article  Google Scholar 

  31. Lidwell W, Holden K, Butler J (2003) Universal principles of design. Rockport, Gloucester, Mass

    Google Scholar 

  32. Lou X, Li XA, Hansen P, Du P (2021) Hand-adaptive user interface: improved gestural interaction in virtual reality. Virtual Real 25(2):367–382. https://doi.org/10.1007/s10055-020-00461-7

    Article  Google Scholar 

  33. Marriott K et al (eds) (2018) Immersive analytics, vol 11190. Springer, Cham

  34. Martin D, Malpica S, Gutierrez D, Masia B, Serrano A (2022) Multimodality in VR: a survey. ACM Comput Surv 54(10s):1–36. https://doi.org/10.1145/3508361

    Article  Google Scholar 

  35. Martinko M, Gardner W (2019) Beyond structured observation: methodological issues and new directions, pp 243–262

  36. McKight P, Najab J (2010) Kruskal–Wallis test, vol 1

  37. McMahan RP, Gorton D, Gresock J, McConnell W, Bowman DA (2006) Separating the effects of level of immersion and 3D interaction techniques. In: Proceedings of the ACM symposium on virtual reality software and technology. ACM, Limassol Cyprus, pp 108–111

  38. Mendes D, Fonseca F, Araujo B, Ferreira A, Jorge J (2014) Mid-air interactions above stereoscopic interactive tables. In: 2014 IEEE symposium on 3D user interfaces (3DUI). IEEE, MN, pp 3–10

  39. Mohamad Yahya Fekri A, AjuneWanis I (2019) A review on multimodal interaction in mixed reality environment. IOP Conf Ser Mater Sci Eng 551(1):012049. https://doi.org/10.1088/1757-899X/551/1/012049

    Article  Google Scholar 

  40. Moustafa F, Steed A (2018). A longitudinal study of small group interaction in social virtual reality. In: Proceedings of the 24th ACM symposium on virtual reality software and technology. ACM, Tokyo, pp 1–10

  41. Murray JH (2011) Inventing the medium: principles of interaction design as a cultural practice. The MIT Press, Cambridge

    Google Scholar 

  42. Murthy G, Jadon R (2009) A review of vision based hand gestures recognition. IJITKM 2:405–410

    Google Scholar 

  43. Nanjappan V, Liang H-N, Lu F, Papangelis K, Yue Y, Man KL (2018) User-elicited dual-hand interactions for manipulating 3D objects in virtual reality environments. HCIS 8(1):31. https://doi.org/10.1186/s13673-018-0154-5

    Article  Google Scholar 

  44. Olmedo H, Escudero D, Cardeñoso V (2015) Multimodal interaction with virtual worlds XMMVR: eXtensible language for MultiModal interaction with virtual reality worlds. J Multimodal User Interfaces 9(3):153–172. https://doi.org/10.1007/s12193-015-0176-5

    Article  Google Scholar 

  45. O’Shaughnessy D (2003) Interacting with computers by voice: automatic speech recognition and synthesis. Proc IEEE 91(9):1272–1305. https://doi.org/10.1109/JPROC.2003.817117

    Article  Google Scholar 

  46. Pamungkas DS, Ward K (2016) Electro-tactile feedback system to enhance virtual reality experience. Int J Comput Theory Eng 8(6):465–470. https://doi.org/10.7763/IJCTE.2016.V8.1090

    Article  Google Scholar 

  47. Perret J, Vander Poorten E (2018) Touching virtual reality: a review of haptic gloves. In: Actuator 2018; 16th international conference on new actuators, pp 1–5

  48. Piumsomboon T, Lee G, Lindeman R, Billinghurst M (2017) Exploring natural eye-gaze-based interaction for immersive virtual reality, pp 36–39

  49. Polit DF, Beck CT (2017) Nursing research: generating and assessing evidence for nursing practice, 10th edn. Wolters Kluwer Health Philadelphia, Philadelphia

    Google Scholar 

  50. Ramaseri Chandra AN, El Jamiy F, Reza H (2019) A review on usability and performance evaluation in virtual reality systems. In: 2019 International conference on computational science and computational intelligence (CSCI). IEEE, Las Vegas, pp 1107–1114

  51. Rantamaa H-R, Kangas J, Jordan M, Mehtonen H, Mäkelä J, Ronkainen K, Raisamo R (2022) Evaluation of virtual handles for dental implant manipulation in virtual reality implant planning procedure. Int J Comput Assisted Radiol Surg 17(9):1723–1730. https://doi.org/10.1007/s11548-022-02693-1

    Article  Google Scholar 

  52. Santos A, Zarraonandia T, Díaz P, Aedo I (2017) A comparative study of menus in virtual reality environments. In: Proceedings of the 2017 ACM international conference on interactive surfaces and spaces. ACM, Brighton, pp 294–299

  53. Schalkwyk J, Beeferman D, Beaufays F, Byrne B, Chelba C, Cohen M, Strope B (2010) your word is my command: Google search by voice: a case study. In: Neustein A (ed) Advances in speech recognition: mobile environments, call centers and clinics. Springer, US, Boston, MA, pp 61–90

  54. Seinfeld S, Feuchtner T, Maselli A, Müller J (2021) User representations in human-computer interaction. Hum Comput Interact 36(5–6):400–438. https://doi.org/10.1080/07370024.2020.1724790

  55. Song P, Goh WB, Hutama W, Fu C-W, Liu X (2012) A handle bar metaphor for virtual object manipulation with mid-air interaction. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, Austin, pp 1297–1306

  56. Spanlang B, Normand J-M, Borland D, Kilteni K, Giannopoulos E, As Pomés, Slater M (2014) How to build an embodiment lab: achieving body representation illusions in virtual reality. Front Robot AI. https://doi.org/10.3389/frobt.2014.00009

    Article  Google Scholar 

  57. Suarez Fernandez RA, Sanchez-Lopez JL, Sampedro C, Bavle H, Molina M, Campoy P (2016) Natural user interfaces for human-drone multi-modal interaction. In: 2016 International conference on unmanned aircraft systems (ICUAS). IEEE, Arlington, pp 1013–1022

  58. Sudha M, Sriraghav K, Abisheck S, Jacob S, Manisha S (2017) Approaches and applications of virtual reality and gesture recognition: a review. Int J Ambient Comput Intell 8:1–18. https://doi.org/10.4018/IJACI.2017100101

    Article  Google Scholar 

  59. Sutcliffe AG, Poullis C, Gregoriades A, Katsouri I, Tzanavari A, Herakleous K (2019) Reflecting on the design process for virtual reality applications. Int J Hum Comput Interact 35(2):168–179. https://doi.org/10.1080/10447318.2018.1443898

    Article  Google Scholar 

  60. Taherdoost H (2019) What is the best response scale for survey and questionnaire design; review of different lengths of rating scale / attitude scale / likert scale. Int J Acad Res Manag 8(1):1–10

    Google Scholar 

  61. Wagner J, Stuerzlinger W, Nedel L (2021) Comparing and combining virtual hand and virtual ray pointer interactions for data manipulation in immersive analytics. IEEE Trans Visual Comput Graphics 27(5):2513–2523. https://doi.org/10.1109/TVCG.2021.3067759

    Article  Google Scholar 

  62. Wang Y, Zhai G, Chen S, Min X, Gao Z, Song X (2019) Assessment of eye fatigue caused by head-mounted displays using eye-tracking. Biomed Eng Online 18(1):111. https://doi.org/10.1186/s12938-019-0731-5

    Article  Google Scholar 

  63. Wilson V (2014) Research methods: triangulation. Evid Based Libr Inf Pract 9:74–75. https://doi.org/10.18438/B8WW3X

    Article  Google Scholar 

  64. Yang KCC (2019) Cases on immersive virtual reality techniques. IGI Global, Hershey, PA

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Future Environments, Auckland University of Technology, Auckland, New Zealand

    Yemon Lee & Stefan Marks

  2. Independent Researcher, Auckland, New Zealand

    Andy M. Connor

Authors
  1. Yemon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andy M. Connor
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Marks
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andy M. Connor.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, Y., Connor, A.M. & Marks, S. Mixed interaction: evaluating user interactions for object manipulations in virtual space. J Multimodal User Interfaces 18, 297–311 (2024). https://doi.org/10.1007/s12193-024-00431-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-024-00431-2

Keywords

Search

Navigation