Interactive sonification to assist children with autism during motor therapeutic interventions | Personal and Ubiquitous Computing
Skip to main content
Springer Nature Link
Log in

Interactive sonification to assist children with autism during motor therapeutic interventions

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Interactive sonification is an effective tool used to guide individuals when practicing movements. Little research has shown the use of interactive sonification in supporting motor therapeutic interventions for children with autism who exhibit motor impairments. The goal of this research is to study if children with autism understand the use of interactive sonification during motor therapeutic interventions, its potential impact of interactive sonification in the development of motor skills in children with autism, and the feasibility of using it in specialized schools for children with autism. We conducted two deployment studies in Mexico using Go-with-the-Flow, a framework to sonify movements previously developed for chronic pain rehabilitation. In the first study, six children with autism were asked to perform the forward reach and lateral upper-limb exercises while listening to three different sound structures (i.e., one discrete and two continuous sounds). Results showed that children with autism exhibit awareness about the sonification of their movements and engage with the sonification. We then adapted the sonifications based on the results of the first study, for motor therapy of children with autism. In the next study, nine children with autism were asked to perform upper-limb lateral, cross-lateral, and push movements while listening to five different sound structures (i.e., three discrete and two continues) designed to sonify the movements. Results showed that discrete sound structures engage the children in the performance of upper-limb movements and increase their ability to perform the movements correctly. We finally propose design considerations that could guide the design of projects related to interactive sonification

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Henceforth, we will use the term anchor point to refer to the target points of the movement. For example, the initial position and the end position of the movement.

  2. Go-with-the-Flow is the name given to a framework proposed in [21] that includes a set of design principles, sonification paradigms, and alterations. The app developed using the framework guidelines (e.g., smartphone/wearable application) is also named Go-with-the-Flow in [21]. In this paper, we used the Go-with-the-Flow mainly to name the app that was built using the framework.

  3. https://youtu.be/UJttpUJcIM4

  4. “Pasitos” is a school clinic located in Mexico specialized in the care of children with autism, where 15 psychologists-teachers attend to close to 60 children with severe and mild autism. Both deployment studies were conducted at this clinic.

  5. Psychologist’s quotes were translated from Spanish to English.

  6. https://www.inegi.org.mx/contenidos/saladeprensa/aproposito/2019/internet2019_Nal.pdf

References

  1. Baio J, Wiggins L, Christensen DL et al (2018) Prevalence of autism spectrum disorder among children aged 8 years — autism and developmental disabilities monitoring network, 11 Sites, United States, 2014. MMWR Surveill Summ 67:1–23. https://doi.org/10.15585/mmwr.ss6706a1

    Article  Google Scholar 

  2. Fombonne E, Marcin C, Manero AC, Bruno R, Diaz C, Villalobos M, Ramsay K, Nealy B (2016) Prevalence of autism spectrum disorders in Guanajuato, Mexico: The Leon survey. J Autism Dev Disord 46:1669–1685. https://doi.org/10.1007/s10803-016-2696-6

    Article  Google Scholar 

  3. American Psychiatric Association (ed) (2013) Diagnostic and statistical manual of mental disorders (DSM-5®), 5th edn. American Psychiatric Association, Washington, DC, London

    Google Scholar 

  4. Green D, Charman T, Pickles A et al (2009) Impairment in movement skills of children with autistic spectrum disorders. Dev Med Child Neurol 51:311–316. https://doi.org/10.1111/j.1469-8749.2008.03242.x

    Article  Google Scholar 

  5. Piochon C, Kloth AD, Grasselli G, Titley HK, Nakayama H, Hashimoto K, Wan V, Simmons DH, Eissa T, Nakatani J, Cherskov A, Miyazaki T, Watanabe M, Takumi T, Kano M, Wang SSH, Hansel C (2014) Cerebellar plasticity and motor learning deficits in a copy-number variation mouse model of autism. Nat Commun 5:5586. https://doi.org/10.1038/ncomms6586

    Article  Google Scholar 

  6. Fournier KA, Hass CJ, Naik SK, Lodha N, Cauraugh JH (2010) Motor coordination in autism spectrum disorders: a synthesis and meta-analysis. J Autism Dev Disord 40:1227–1240. https://doi.org/10.1007/s10803-010-0981-3

    Article  Google Scholar 

  7. Staples KL, Reid G (2010) Fundamental movement skills and autism spectrum disorders. J Autism Dev Disord 40:209–217. https://doi.org/10.1007/s10803-009-0854-9

    Article  Google Scholar 

  8. Anzulewicz A, Sobota K, Delafield-Butt JT (2016) Toward the autism motor signature: gesture patterns during smart tablet gameplay identify children with autism. Sci Rep 6:1–13. https://doi.org/10.1038/srep31107

    Article  Google Scholar 

  9. Cibrian FL, Madrigal M, Avelais M, Tentori M (2020) Supporting coordination of children with ASD using neurological music therapy: A pilot randomized control trial comparing an elastic touch-display with tambourines. Res Dev Disabil 106:103741. https://doi.org/10.1016/j.ridd.2020.103741

  10. Boucenna S, Cohen D, Meltzoff AN, Gaussier P, Chetouani M (2016) Robots learn to recognize individuals from imitative encounters with people and avatars. Sci Rep 6:1–10. https://doi.org/10.1038/srep19908

    Article  Google Scholar 

  11. Boucenna S, Anzalone S, Tilmont E, Cohen D, Chetouani M (2014) Learning of social signatures through imitation game between a robot and a human partner. IEEE Trans Auton Ment Dev 6:213–225. https://doi.org/10.1109/TAMD.2014.2319861

    Article  Google Scholar 

  12. Xavier J, Guedjou H, Anzalone SM, Boucenna S, Guigon E, Chetouani M, Cohen D (2018) Toward a motor signature in autism: Studies from human-machine interaction. Encephale 45:6–11. https://doi.org/10.1016/j.encep.2018.08.002

    Article  Google Scholar 

  13. Sterr A, Freivogel S, Voss A (2002) Exploring a repetitive training regime for upper limb hemiparesis in an in-patient setting: a report on three case studies. Brain Inj 16:1093–1107. https://doi.org/10.1080/02699050210155267

    Article  Google Scholar 

  14. Lederman E, Cramer GD, Donatelli R, Willard F (2005) The science and practice of manual therapy. Second Edition. Elsevier/Churchill Livingstone, United Kingdom

  15. Putnam C, Chong L (2008) Software and technologies designed for people with autism: what do users want? In: Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility - ASSETS’08. ACM, Nova Scotia, Canada, pp 3–10. https://doi.org/10.1145/1414471.1414475

  16. Cook JL, Blakemore SJ, Press C (2013) Atypical basic movement kinematics in autism spectrum conditions. Brain 136:2816–2824. https://doi.org/10.1093/brain/awt208

    Article  Google Scholar 

  17. Hermann T, Hunt A (2005) An introduction to interactive sonification. IEEE Multimed 12(2):20–24. https://doi.org/10.1109/MMUL.2005.26

  18. Sigrist R, Rauter G, Marchal-Crespo L, Riener R, Wolf P (2014) Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233:909–925. https://doi.org/10.1007/s00221-014-4167-7

    Article  Google Scholar 

  19. Vogt K, Pirrò D, Kobenz I, Höldrich R, Eckel G (2010) PhysioSonic - evaluated movement sonification as auditory feedback in physiotherapy. In: Ystad S, Aramaki M, Kronland-Martinet R, Jensen K (eds) Auditory Display: 6th International Symposium, CMMR/ICAD 2009, Copenhagen, Denmark, May 18-22, 2009. Revised Papers. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 103–120

  20. Rosati G, Roda A, Avanzini F, Masiero S (2013) On the role of auditory feedback in robotic-assisted movement training after stroke: review of the literature. Comput Intell Neurosci 2013:586138. https://doi.org/10.1155/2013/586138

  21. Singh A, Piana S, Pollarolo D, Volpe G, Varni G, Tajadura-Jiménez A, Williams ACC, Camurri A, Bianchi-Berthouze N (2016) Go-with-the-Flow : tracking, analysis and sonification of movement and breathing to build confidence in activity despite chronic pain. Hum Comput Interact 31:335–383. https://doi.org/10.1080/07370024.2015.1085310

    Article  Google Scholar 

  22. Ghisio S, Coletta P, Piana S, et al (2015) An open platform for full body interactive sonification exergames. Intell Technol Interact Entertain (INTETAIN), 2015 7th Int Conf 17:168–175. https://doi.org/10.4108/icst.intetain.2015.259584

  23. Marchi E, Schuller B, Baird A et al (2018) The ASC-inclusion perceptual serious gaming platform for autistic children. IEEE Trans Games 11(4):328–339. https://doi.org/10.1109/tg.2018.2864640

  24. Effenberg AO (2005) Movement sonification: effects on perception and action. IEEE Multimed 12(2):53–59. https://doi.org/10.1109/MMUL.2005.31

  25. Cesarini D, Calvaresi D, Farnesi C, Taddei D, Frediani S, Ungerechts BE, Hermann T (2016) MEDIATION: An eMbEddeD system for auditory feedback of hand-water InterAcTION while swimming. Procedia Eng 147:324–329. https://doi.org/10.1016/j.proeng.2016.06.301

  26. Schaffert N, Mattes K (2015) Interactive sonification in rowing: an application of acoustic feedback for on-water training. IEEE Multimed 22:58–67. https://doi.org/10.1109/MMUL.2015.9

  27. Grosshauser T, Bläsing B, Spieth C, Hermann T (2012) Wearable sensor-based real-time sonification of motion and foot pressure in dance teaching and training. J Audio Eng Soc 60:580–589

    Google Scholar 

  28. Landry P, Parés N, Minsky J, Parés R (2012) Participatory design for exertion interfaces for children. In: Proceedings of the 11th International Conference on Interaction Design and Children. ACM Press, New York, NY, p 256. https://doi.org/10.1145/2307096.2307139

  29. Cibrian FL, Tentori M, Martínez-García A (2015) Hunting relics: a persuasive exergame to promote collective exercise in young children. Int J Hum Comput Interact 32:277–294. https://doi.org/10.1080/10447318.2016.1136180

    Article  Google Scholar 

  30. Colombo L, Landoni M (2013) Low-tech and high-tech prototyping for eBook co-design with children. In Proceedings of the 12th international conference on interaction design and children, pp 289–292

  31. Lima JL, Axt G, Teixeira DS, Monteiro D, Cid L, Yamamoto T, Murillo-Rodriguez E, Machado S (2020) Exergames for children and adolescents with autism spectrum disorder : an overview. Clin Pract Epidemiol Ment Health 16:1–6. https://doi.org/10.2174/1745017902016010001

    Article  Google Scholar 

  32. Fang Q, Aiken CA, Fang C, Pan Z (2019) Effects of exergaming on physical and cognitive functions in individuals with autism spectrum disorder: a systematic review. Games Health J 8:74–84. https://doi.org/10.1089/g4h.2018.0032

    Article  Google Scholar 

  33. Cibrian FL, Peña O, Ortega D, Tentori M (2017) BendableSound: an elastic multisensory surface using touch-based interactions to assist children with severe autism during music therapy. Int J Hum Comput Stud 107:22–37. https://doi.org/10.1016/j.ijhcs.2017.05.003

    Article  Google Scholar 

  34. Caro K, Tentori M, Martinez-Garcia AI, Alvelais M (2017) Using the FroggyBobby exergame to support eye-body coordination development of children with severe autism. Int J Hum Comput Stud 105:12–27. https://doi.org/10.1016/j.ijhcs.2017.03.005

    Article  Google Scholar 

  35. Peña O, Cibrian FL, Tentori M (2020) Circus in Motion: a multimodal exergame supporting vestibular therapy for children with autism. J Multimodal User Interfaces, 1–17. https://doi.org/10.1007/s12193-020-00345-9

  36. Bhattacharya A, Gelsomini M, Pérez-Fuster P, et al (2015) Designing motion-based activities to engage students with autism in classroom settings. In: Proceedings of the 14th International Conference on Interaction Design and Children - IDC ’15. ACM, Medford, MA, pp 69–78. https://doi.org/10.1145/2771839.2771847

  37. Piana S, Staglianò A, Odone F, Camurri A (2016) Adaptive body gesture representation for automatic emotion recognition. ACM Trans Interact Intell Syst 6(1):1–31. https://doi.org/10.1145/2818740

  38. Schuller B, Marchi E, Baron-Cohen S, et al (2014) The state of play of ASC-inclusion: an integrated internet-based environment for social inclusion of children with autism spectrum conditions. In: 2nd International Workshop on Digital Games for Empowerment and Inclusion. arXiv preprint arXiv:1403.5912

  39. Morreale F, De Angeli A, O’Modhrain S (2014) Musical interface design: an experience-oriented framework. In NIME 467–472

  40. Boulay M, Benveniste S, Boespflug S, Jouvelot P, Rigaud AS (2011) A pilot usability study of MINWii, a music therapy game for demented patients. Technol Health Care 19:233–246. https://doi.org/10.3233/THC-2011-0628

    Article  Google Scholar 

  41. Gorman M, Lahav A, Saltzman E, Betke M (2007) A camera-based music-making tool for physical rehabilitation. Comput Music J 31:39–53. https://doi.org/10.1162/comj.2007.31.2.39

    Article  Google Scholar 

  42. Oliveros P, Miller L, Heyen J, Siddall G, Hazard S (2011) A musical improvisation interface for people with severe physical disabilities. Music Med 3(3):172–181. https://doi.org/10.1177/1943862111411924

  43. Hobbs D, Worthington-Eyre B (2008) The efficacy of combining augmented reality and music therapy with traditional teaching - preliminary results. i-CREATe 2008 - Int Conv Rehabil Eng Assist Technol 2008 241–244

  44. Schaffert N, Janzen TB, Mattes K, Thaut MH (2019) A review on the relationship between sound and movement in sports and rehabilitation. Front Psychol 10:1–20. https://doi.org/10.3389/fpsyg.2019.00244

    Article  Google Scholar 

  45. Wellner M, Schaufelberger A, Riener R (2007) A study on sound feedback in a virtual environment for gait rehabilitation. In 2007 Virtual Rehabilitation. IEEE, Venice, p 53–56. https://doi.org/10.1109/ICVR.2007.4362130

  46. Bhat AN, Landa RJ, Galloway JC(C) (2011) Current perspectives on motor functioning in infants, children, and adults with autism spectrum disorders. Phys Ther 91:1116–1129. https://doi.org/10.2522/ptj.20100294

    Article  Google Scholar 

  47. Tajadura-Jiménez A, Cuadrado F, Rick P, et al (2018) Designing a gesture-sound wearable system to motivate physical activity by altering body perception. In: Proceedings of the 5th International Conference on Movement and Computing. pp 1–6. https://doi.org/10.1145/3212721.3212877

  48. Tajadura-Jiménez A, Basia M, Deroy O, et al (2015) As light as your footsteps: altering walking sounds to change perceived body weight, emotional state and gait. In: CHI ’15 - Proceedings of the 2015 CHI Conference on Human Factors in Computing Systems, Seoul, pp 2943–2952. https://doi.org/10.1145/2702123.2702374

  49. Danna J, Paz-Villagrán V, Gondre C, et al (2013) Handwriting sonification for the diagnosis of dysgraphia. In: Recent progress in graphonomics: Learn from the past–Proceedings of the 16th conference of the international graphonomics society. Tokyo University of Agriculture and Technology Press Tokyo, Tokyo, pp 123–126

  50. Frid E, Bresin R, Alborno P, Elblaus L (2016) Interactive sonification of spontaneous movement of children—cross-modal mapping and the perception of body movement qualities through sound. Front Neurosci 10. https://doi.org/10.3389/fnins.2016.00521

  51. Tajadura-Jiménez A, Tsakiris M, Marquardt T, Bianchi-Berthouze N (2015) Action sounds update the mental representation of arm dimension : contributions of kinaesthesia and agency. Front Psychol 6:1–18. https://doi.org/10.3389/fpsyg.2015.00689

    Article  Google Scholar 

  52. Tentori M, Escobedo L, Balderas G (2015) A smart environment for children with autism. IEEE Pervasive Comput 14(2):42–50. https://doi.org/10.1109/MPRV.2015.22

  53. Tang ST, McCorkle R (2002) Use of family proxies in quality of life research for cancer patients at the end of life: a literature review. Cancer Investig 20:1086–1104. https://doi.org/10.1081/CNV-120005928

    Article  Google Scholar 

  54. Mintzberg H (1970) Structured observation as a method to study managerial work. J Manag Stud 7:87–104. https://doi.org/10.1111/j.1467-6486.1970.tb00484.x

    Article  Google Scholar 

  55. Escobedo L, Tentori M, Quintana E, Favela J, Garcia-Rosas D (2014) Using augmented reality to help children with autism stay focused. IEEE Pervasive Comput 13(1):38–46. https://doi.org/10.1109/MPRV.2014.19

  56. Kim J, Wigram T, Gold C (2009) Emotional, motivational and interpersonal responsiveness of children with autism in improvisational music therapy. Autism 13:389–409. https://doi.org/10.1177/1362361309105660

    Article  Google Scholar 

  57. Corbin J, Strauss A (2015) Basics of qualitative research: techniques and procedures for developing grounded theory, 2nd edn. United Kingdom Sage publications, Thousand Oaks

  58. Volkmar F, Chawarska K, Klin A (2005) Autism in infancy and early childhood. Annu Rev Psychol 56:315–336. https://doi.org/10.1146/annurev.psych.56.091103.070159

    Article  Google Scholar 

  59. Duffy C, Healy O (2011) Spontaneous communication in autism spectrum disorder: a review of topographies and interventions. Res Autism Spectr Disord 5:977–983. https://doi.org/10.1016/j.rasd.2010.12.005

    Article  Google Scholar 

  60. Singh A, Bianchi-Berthouze N, Williams AC (2017) Supporting everyday function in chronic pain using wearable technology. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17, Denver, Colorado, pp 3903–3915. https://doi.org/10.1145/3025453.3025947

  61. Cibrian FL, Mercado J, Escobedo L, Tentori M (2018) A step towards identifying the sound preferences of children with autism. In: ACM (ed) PervasiveHealth 2018 - 12th EAI International Conference on Pervasive Computing Technologies for Healthcare. ACM, New York, pp 158–167. https://doi.org/10.1145/3240925.3240958

  62. Kern JK, Geier DA, Adams JB, Troutman MR, Davis G, King PG, Young JL, Geier MR (2011) Autism severity and muscle strength: a correlation analysis. Res Autism Spectr Disord 5:1011–1015. https://doi.org/10.1016/j.rasd.2010.11.002

    Article  Google Scholar 

  63. Lazar J, Feng JH, Hochheiser H (2017) Research methods in human-computer interaction, 2nd edn. Morgan Kaufmann, Elsevier Science, Cambridge, MA

  64. Berger DS (2002) Music theapy, sensory integration and autistic child. Jessica Kingsley Publishers, London and New York

    Google Scholar 

  65. Salgado-Montejo A, Marmolejo-Ramos F, Alvarado JA, Arboleda JC, Suarez DR, Spence C (2016) Drawing sounds: representing tones and chords spatially. Exp Brain Res 234:3509–3522. https://doi.org/10.1007/s00221-016-4747-9

    Article  Google Scholar 

  66. Spence C (2011) Crossmodal correspondences: a tutorial review. Attent Percep Psychophysiol 73:971–995. https://doi.org/10.3758/s13414-010-0073-7

    Article  Google Scholar 

  67. Koumpouros Y, Kafazis T (2019) Wearables and mobile technologies in autism spectrum disorder interventions: a systematic literature review. Res Autism Spectr Disord 66:101405. https://doi.org/10.1016/j.rasd.2019.05.005

    Article  Google Scholar 

Download references

Acknowledgments

We thank all the participants enrolled in this study and the researchers who provide helpful comments on previous versions of this document. We also thank CONACYT for students’ fellowships, the Jacob Foundation, Gillian Hayes, Armando Beltran, and the Start Lab team for their feedback.

Funding

This work was partially funded by the Microsoft Faculty Fellowship grant and the EU-FP7 Marie Curie IRSES UBIHEALTH: Exchange of Excellence in Ubiquitous Computing Technologies to Address Healthcare Challenges.

Author information

Authors and Affiliations

  1. Fowler School of Engineering, Chapman University, Orange, CA, USA

    Franceli L. Cibrian

  2. Computer Science and Technology at Universidad Carlos III of Madrid, Madrid, Spain

    Judith Ley-Flores

  3. Northumbria University, Newcastle upon Tyne, UK

    Joseph W. Newbold

  4. UCLIC, University College London, London, UK

    Aneesha Singh & Nadia Bianchi-Berthouze

  5. Department of Computer Science, CICESE Research Center, Ensenada, Mexico

    Monica Tentori

Authors
  1. Franceli L. Cibrian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Judith Ley-Flores
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joseph W. Newbold
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aneesha Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nadia Bianchi-Berthouze
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Monica Tentori
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Franceli L. Cibrian.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cibrian, F.L., Ley-Flores, J., Newbold, J.W. et al. Interactive sonification to assist children with autism during motor therapeutic interventions. Pers Ubiquit Comput 25, 391–410 (2021). https://doi.org/10.1007/s00779-020-01479-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-020-01479-z

Keywords

Search

Navigation