Multimodal human\u2013computer interaction has been sought to provide not only more compelling interactive experiences, but also more accessible interfaces to mobile devices. With the advance in mobile technology and in affordable sensors, multimodal research that leverages and combines multiple interaction modalities (such as speech, touch, vision, and gesture) has become more and more prominent. This article provides a framework for the key aspects in mid-air gesture and speech-based interaction for older adults. It explores the literature on multimodal interaction and older adults as technology users and summarises the main findings for this type of users. Building on these findings, a number of crucial factors to take into consideration when designing multimodal mobile technology for older adults are described. The aim of this work is to promote the usefulness and potential of multimodal technologies based on mid-air gestures and voice input for making older adults' interaction with mobile devices more accessible and inclusive.<\/p>","DOI":"10.4018\/ijmhci.2020010102","type":"journal-article","created":{"date-parts":[[2019,11,29]],"date-time":"2019-11-29T10:02:55Z","timestamp":1575021775000},"page":"22-41","source":"Crossref","is-referenced-by-count":8,"title":["Framing the Design Space of Multimodal Mid-Air Gesture and Speech-Based Interaction With Mobile Devices for Older People"],"prefix":"10.4018","volume":"12","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-9063-2540","authenticated-orcid":true,"given":"Ornella","family":"Mich","sequence":"first","affiliation":[{"name":"FBK, Trento, Italy"}]},{"given":"Gianluca","family":"Schiavo","sequence":"additional","affiliation":[{"name":"FBK, Trento, Italy"}]},{"given":"Michela","family":"Ferron","sequence":"additional","affiliation":[{"name":"FBK, Trento, Italy"}]},{"given":"Nadia","family":"Mana","sequence":"additional","affiliation":[{"name":"FBK, Trento, Italy"}]}],"member":"2432","reference":[{"key":"IJMHCI.2020010102-0","unstructured":"Aigner, R., Wigdor, D., Benko, H., Haller, M., Lindbauer, D., Ion, A., & Koh, J. T. K. V. (2012). Understanding mid-air hand gestures: A study of human preferences in usage of gesture types for HCI. Microsoft Research."},{"key":"IJMHCI.2020010102-1","article-title":"CrossY: a crossing based drawing application.","author":"G.Apitz","year":"2004","journal-title":"Proc. UIST 2004"},{"key":"IJMHCI.2020010102-2","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124887"},{"key":"IJMHCI.2020010102-3","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2012.05.009"},{"key":"IJMHCI.2020010102-4","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2013.02.006"},{"key":"IJMHCI.2020010102-5","doi-asserted-by":"publisher","DOI":"10.1145\/1936652.1936657"},{"key":"IJMHCI.2020010102-6","doi-asserted-by":"publisher","DOI":"10.4236\/jsea.2011.49059"},{"key":"IJMHCI.2020010102-7","doi-asserted-by":"publisher","DOI":"10.1145\/2325616.2325625"},{"key":"IJMHCI.2020010102-8","doi-asserted-by":"publisher","DOI":"10.1145\/800250.807503"},{"key":"IJMHCI.2020010102-9","doi-asserted-by":"publisher","DOI":"10.1145\/1738826.1738853"},{"key":"IJMHCI.2020010102-10","doi-asserted-by":"publisher","DOI":"10.1145\/191666.191733"},{"key":"IJMHCI.2020010102-11","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8535.2008.00914.x"},{"key":"IJMHCI.2020010102-12","doi-asserted-by":"publisher","DOI":"10.1109\/TSA.2004.828702"},{"key":"IJMHCI.2020010102-13","doi-asserted-by":"crossref","unstructured":"Chen, X., Schwarz, J., Harrison, C., Mankoff, J., & Hudson, S. E. (2014). Air+Touch: Interweaving Touch & In-air Gestures. Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (pp. 519\u2013525). New York: ACM. doi:10.1145\/2642918.2647392","DOI":"10.1145\/2642918.2647392"},{"key":"IJMHCI.2020010102-14","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/4528.001.0001"},{"key":"IJMHCI.2020010102-15","doi-asserted-by":"publisher","DOI":"10.1145\/2049536.2049561"},{"key":"IJMHCI.2020010102-16","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.92.22.9921"},{"key":"IJMHCI.2020010102-17","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03655-2_7"},{"key":"IJMHCI.2020010102-18","doi-asserted-by":"publisher","DOI":"10.2114\/jpa2.25.113"},{"key":"IJMHCI.2020010102-19","doi-asserted-by":"publisher","DOI":"10.1145\/2909132.2909260"},{"key":"IJMHCI.2020010102-20","doi-asserted-by":"publisher","DOI":"10.1145\/2901790.2901834"},{"key":"IJMHCI.2020010102-21","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-85729-433-3_1"},{"key":"IJMHCI.2020010102-22","doi-asserted-by":"publisher","DOI":"10.1109\/ICMB.2005.97"},{"key":"IJMHCI.2020010102-23","doi-asserted-by":"publisher","DOI":"10.5539\/gjhs.v4n2p2"},{"key":"IJMHCI.2020010102-24","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-06076-3_6"},{"key":"IJMHCI.2020010102-25","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470703"},{"key":"IJMHCI.2020010102-26","author":"A. D.Fisk","year":"2009","journal-title":"Designing for Older Adults: Principles and Creative Human Factors Approaches"},{"key":"IJMHCI.2020010102-27","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-13-3693-5_3"},{"key":"IJMHCI.2020010102-28","doi-asserted-by":"publisher","DOI":"10.1177\/0018720815581293"},{"key":"IJMHCI.2020010102-29","first-page":"133","article-title":"Is Movement Better?: Comparing Sedentary and Motion-based Game Controls for Older Adults","author":"K.Gerling","year":"2013","journal-title":"Proceedings of Graphics Interface 2013"},{"key":"IJMHCI.2020010102-30","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208324"},{"key":"IJMHCI.2020010102-31","doi-asserted-by":"publisher","DOI":"10.4324\/9781315740218"},{"key":"IJMHCI.2020010102-32","unstructured":"Grady, J. (1997). Foundations of meaning: Primary metaphors and primary scenes [Dissertation]. University of California, Berkley, CA."},{"key":"IJMHCI.2020010102-33","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2010.5598698"},{"key":"IJMHCI.2020010102-34","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979061"},{"key":"IJMHCI.2020010102-35","author":"R. A.Harris","year":"2005","journal-title":"Voice interaction design: crafting the new conversational speech systems"},{"key":"IJMHCI.2020010102-36","doi-asserted-by":"publisher","DOI":"10.1145\/67449.67496"},{"key":"IJMHCI.2020010102-37","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2036"},{"key":"IJMHCI.2020010102-38","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557130"},{"key":"IJMHCI.2020010102-39","doi-asserted-by":"publisher","DOI":"10.3389\/fnagi.2014.00259"},{"key":"IJMHCI.2020010102-40","unstructured":"Hurtienne, J., & Blessing, L. (2007, August). Design for Intuitive Use-Testing image schema theory for user interface design. In ICED (Vol. 7, pp. 1-12). Academic Press."},{"key":"IJMHCI.2020010102-41","doi-asserted-by":"publisher","DOI":"10.1093\/iwc\/iwu049"},{"key":"IJMHCI.2020010102-42","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2010.08.009"},{"key":"IJMHCI.2020010102-43","doi-asserted-by":"publisher","DOI":"10.1038\/24300"},{"key":"IJMHCI.2020010102-44","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642619"},{"key":"IJMHCI.2020010102-45","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14504901236"},{"key":"IJMHCI.2020010102-46","author":"J.Johnson","year":"2017","journal-title":"Designing user interfaces for an aging population: towards universal design"},{"key":"IJMHCI.2020010102-47"},{"key":"IJMHCI.2020010102-48","doi-asserted-by":"publisher","DOI":"10.1145\/2639189.2670267"},{"key":"IJMHCI.2020010102-49","doi-asserted-by":"crossref","unstructured":"Kendon, A. (1980). Gesticulation and Speech: Two Aspects of the Process of Utterance. The relationship of verbal and nonverbal communication, (25), 207-227.","DOI":"10.1515\/9783110813098.207"},{"key":"IJMHCI.2020010102-50","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511807572"},{"key":"IJMHCI.2020010102-51","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23774-4_9"},{"key":"IJMHCI.2020010102-52","doi-asserted-by":"publisher","DOI":"10.4324\/9781315009292"},{"key":"IJMHCI.2020010102-53","first-page":"257","article-title":"Intelligent Multimedia Interfaces","author":"D. B.Koons","year":"1993"},{"key":"IJMHCI.2020010102-54","author":"P.Kortum","year":"2008","journal-title":"HCI beyond the GUI: Design for haptic, speech, olfactory, and other nontraditional interfaces"},{"key":"IJMHCI.2020010102-55","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557122"},{"key":"IJMHCI.2020010102-56","author":"G.Lakoff","year":"2008","journal-title":"Metaphors we live by"},{"key":"IJMHCI.2020010102-57","doi-asserted-by":"publisher","DOI":"10.1016\/j.neurobiolaging.2005.05.024"},{"key":"IJMHCI.2020010102-58","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2318-11-64"},{"key":"IJMHCI.2020010102-59","doi-asserted-by":"publisher","DOI":"10.1145\/2614028.2615424"},{"key":"IJMHCI.2020010102-60","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31522-0_99"},{"key":"IJMHCI.2020010102-61","author":"J. R.Lewis","year":"2011","journal-title":"Practical speech user interface design"},{"key":"IJMHCI.2020010102-62","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-39943-0_34"},{"key":"IJMHCI.2020010102-63","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208570"},{"key":"IJMHCI.2020010102-64","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1145\/2914920.2915015","article-title":"Investigating mid-air gestures and handhelds in motion tracked environments.","author":"V.M\u00e4kel\u00e4","year":"2016","journal-title":"Proceedings of the 5th ACM International Symposium on Pervasive Displays"},{"key":"IJMHCI.2020010102-65","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2556964"},{"key":"IJMHCI.2020010102-66","doi-asserted-by":"publisher","DOI":"10.1177\/154193120104500305"},{"key":"IJMHCI.2020010102-67","doi-asserted-by":"publisher","DOI":"10.1007\/BFb0052997"},{"key":"IJMHCI.2020010102-68","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466142"},{"key":"IJMHCI.2020010102-69","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1978969"},{"key":"IJMHCI.2020010102-70","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2010.08.005"},{"key":"IJMHCI.2020010102-71","first-page":"409","article-title":"A procedure for developing intuitive and ergonomic gesture interfaces for HCI.","author":"M.Nielsen","year":"2003","journal-title":"International gesture workshop"},{"key":"IJMHCI.2020010102-72","doi-asserted-by":"publisher","DOI":"10.1145\/1744161.1744163"},{"key":"IJMHCI.2020010102-73","doi-asserted-by":"crossref","unstructured":"O\u2019Hara, K., Harper, R., Mentis, H., Sellen, A., & Taylor, A. (2013). On the Naturalness of Touchless: Putting the \u201cInteraction\u201d Back into NUI. ACM Transactions on Computer-Human Interaction, 20(1), 5:1\u20135:25. doi:10.1145\/2442106.2442111","DOI":"10.1145\/2442106.2442111"},{"key":"IJMHCI.2020010102-74","doi-asserted-by":"publisher","DOI":"10.1145\/1230819.1241668"},{"key":"IJMHCI.2020010102-75","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2003.817127"},{"key":"IJMHCI.2020010102-76","doi-asserted-by":"publisher","DOI":"10.1145\/330534.330538"},{"key":"IJMHCI.2020010102-77","doi-asserted-by":"publisher","DOI":"10.1145\/258549.258821"},{"key":"IJMHCI.2020010102-78","doi-asserted-by":"crossref","unstructured":"Oviatt, S., & Olsen, E. (1994). Integration themes in multimodal human-computer interaction. Proceedings of theThird International Conference on Spoken Language Processing. Academic Press.","DOI":"10.21437\/ICSLP.1994-140"},{"key":"IJMHCI.2020010102-79","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2012.305"},{"key":"IJMHCI.2020010102-80","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511499968"},{"key":"IJMHCI.2020010102-81","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35292-8_15"},{"key":"IJMHCI.2020010102-82","author":"K.Pernice","year":"2013","journal-title":"Senior Citizens (Ages 65 and older) on the Web"},{"key":"IJMHCI.2020010102-83","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468527"},{"key":"IJMHCI.2020010102-84","unstructured":"Poddar, I., Sethi, Y., Ozyildiz, E., & Sharma, R. (1998). Toward natural gesture\/speech HCI: A case study of weather narration. Proc. Workshop on Perceptual User Interfaces. Academic Press."},{"key":"IJMHCI.2020010102-85","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-011-0470-5"},{"issue":"1","key":"IJMHCI.2020010102-86","first-page":"99","article-title":"Common causes of vision loss in elderly patients.","volume":"60","author":"D. A.Quillen","year":"1999","journal-title":"American Family Physician"},{"key":"IJMHCI.2020010102-87","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466138"},{"key":"IJMHCI.2020010102-88","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753458"},{"key":"IJMHCI.2020010102-89","doi-asserted-by":"publisher","DOI":"10.1518\/0018720054679452"},{"key":"IJMHCI.2020010102-90","doi-asserted-by":"crossref","unstructured":"Ruiz, J., Li, Y., & Lank, E. (2011). User-defined Motion Gestures for Mobile Interaction. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 197\u2013206). New York: ACM. doi:10.1145\/1978942.1978971","DOI":"10.1145\/1978942.1978971"},{"key":"IJMHCI.2020010102-91","doi-asserted-by":"publisher","DOI":"10.1093\/geronj\/47.2.P96"},{"key":"IJMHCI.2020010102-92","doi-asserted-by":"crossref","unstructured":"Schiavo, G., Mich, O., Ferron, M., & Mana, N. (2017). Mobile Multimodal Interaction for Older and Younger Users: Exploring Differences and Similarities. Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia (pp. 407\u2013414). New York: ACM. doi:10.1145\/3152832.3156615","DOI":"10.1145\/3152832.3156615"},{"key":"IJMHCI.2020010102-93","doi-asserted-by":"publisher","DOI":"10.1037\/0278-6133.20.4.243"},{"key":"IJMHCI.2020010102-94","doi-asserted-by":"publisher","DOI":"10.1152\/japplphysiol.00045.2004"},{"key":"IJMHCI.2020010102-95","doi-asserted-by":"publisher","DOI":"10.1145\/348941.348990"},{"key":"IJMHCI.2020010102-96","doi-asserted-by":"publisher","DOI":"10.1145\/1868914.1869031"},{"key":"IJMHCI.2020010102-97","doi-asserted-by":"publisher","DOI":"10.1145\/2503673.2503694"},{"key":"IJMHCI.2020010102-98","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2017.2768720"},{"key":"IJMHCI.2020010102-99","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X00003447"},{"key":"IJMHCI.2020010102-100","doi-asserted-by":"publisher","DOI":"10.1093\/iwc\/iww040"},{"key":"IJMHCI.2020010102-101","doi-asserted-by":"publisher","DOI":"10.4017\/gt.2012.11.02.262.651"},{"key":"IJMHCI.2020010102-102","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-07230-2_17"},{"key":"IJMHCI.2020010102-103","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-007-0190-z"},{"key":"IJMHCI.2020010102-104","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-5915.2008.00192.x"},{"key":"IJMHCI.2020010102-105","doi-asserted-by":"publisher","DOI":"10.2307\/30036540"},{"key":"IJMHCI.2020010102-106","doi-asserted-by":"crossref","unstructured":"Vipperla, R., Renals, S., & Frankel, J. (2008). Longitudinal study of ASR performance on ageing voices. Proceedings of Interspeech 2008 (pp. 2550-2553). Academic Press.","DOI":"10.21437\/Interspeech.2008-632"},{"key":"IJMHCI.2020010102-107","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466139"},{"key":"IJMHCI.2020010102-108","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-27491-6_13"},{"key":"IJMHCI.2020010102-109","author":"D.Wigdor","year":"2011","journal-title":"Brave NUI world: designing natural user interfaces for touch and gesture"},{"key":"IJMHCI.2020010102-110","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-10308-7_6"},{"key":"IJMHCI.2020010102-111","doi-asserted-by":"crossref","first-page":"1083","DOI":"10.1145\/1518701.1518866","article-title":"User-defined gestures for surface computing.","author":"J. O.Wobbrock","year":"2009","journal-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems"},{"key":"IJMHCI.2020010102-112","unstructured":"World Health Organization. (2018, February 5). Ageing and health (Facts sheet). Retrieved from https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/ageing-and-health"},{"key":"IJMHCI.2020010102-113","doi-asserted-by":"publisher","DOI":"10.1145\/2639189.2639251"},{"key":"IJMHCI.2020010102-114","doi-asserted-by":"publisher","DOI":"10.1145\/958432.958480"},{"key":"IJMHCI.2020010102-115","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557085"},{"key":"IJMHCI.2020010102-116","doi-asserted-by":"crossref","unstructured":"Young, V., & Mihailidis, A. (2010). Difficulties in automatic speech recognition of dysarthric speakers and implications for speech-based applications used by the elderly: a literature review. Assistive Technology: The Official Journal of RESNA, 22(2), 99\u2013112. 10.1080\/10400435.2010.483646","DOI":"10.1080\/10400435.2010.483646"},{"issue":"3","key":"IJMHCI.2020010102-117","first-page":"133","article-title":"Speech and voice characteristics of geriatric speakers: A review of the literature and a call for research and training.","volume":"14","author":"R. I.Zraick","year":"2006","journal-title":"Journal of Medical Speech-Language Pathology"}],"container-title":["International Journal of Mobile Human Computer Interaction"],"original-title":[],"language":"ng","link":[{"URL":"https:\/\/www.igi-global.com\/viewtitle.aspx?TitleId=243730","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,7]],"date-time":"2022-10-07T05:54:36Z","timestamp":1665122076000},"score":1,"resource":{"primary":{"URL":"https:\/\/services.igi-global.com\/resolvedoi\/resolve.aspx?doi=10.4018\/IJMHCI.2020010102"}},"subtitle":[""],"short-title":[],"issued":{"date-parts":[[2020,1,1]]},"references-count":118,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,1]]}},"URL":"https:\/\/doi.org\/10.4018\/ijmhci.2020010102","relation":{},"ISSN":["1942-390X","1942-3918"],"issn-type":[{"value":"1942-390X","type":"print"},{"value":"1942-3918","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,1]]}}}