MagiPlay: An Augmented Reality Serious Game Allowing Children to Program Intelligent Environments | SpringerLink
Skip to main content
Springer Nature Link
Log in

MagiPlay: An Augmented Reality Serious Game Allowing Children to Program Intelligent Environments

  • Chapter
  • First Online:
Transactions on Computational Science XXXVII

Part of the book series: Lecture Notes in Computer Science ((TCOMPUTATSCIE,volume 12230))

  • 744 Accesses

Abstract

A basic understanding of problem-solving and computational thinking is undoubtedly a benefit for all ages. At the same time, the proliferation of Intelligent Environments has raised the need for configuring their behaviors to address their users’ needs. This configuration can take the form of programming, and coupled with advances in Augmented Reality and Conversational Agents, can enable users to take control of their intelligent surroundings in an efficient and natural manner. Focusing on children, who can greatly benefit by being immersed in programming from an early age, this paper presents an authoring framework in the form of an Augmented Reality serious game, named MagiPlay, allowing children to manipulate and program their Intelligent Environment. This is achieved through a handheld device, which children can use to capture smart objects via its camera and subsequently create rules dictating their behavior. An intuitive user interface permits players to combine LEGO-like 3D bricks as a part of the rule-based creation process, aiming to make the experience more natural. Additionally, children can communicate with the system via natural language through a Conversational Agent, in order to configure the rules by talking with a human-like agent, while the agent also serves as a guide/helper for the player, providing context-sensitive tips for every part of the rule creation process. Finally, MagiPlay enables networked collaboration, to allow parental and teacher guidance and support. The main objective of this research work is to provide young learners with a fun and engaging way to program their intelligent surroundings. This paper describes the game logic of MagiPlay, its implementation details, and discusses the results of a statistically significant evaluation conducted with end-users, i.e. a group of children of seven to twelve years old.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 5719
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 7149
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.statista.com/statistics/471264/iot-number-of-connected-devices-worldwide.

  2. 2.

    https://ifttt.com/discover.

  3. 3.

    https://zapier.com.

  4. 4.

    https://developer.apple.com/arkit/.

  5. 5.

    https://unity3d.com/.

  6. 6.

    https://www.adobe.com/gr_en/products/fuse.html.

  7. 7.

    https://www.mixamo.com/.

  8. 8.

    https://rasa.com/docs/rasa/nlu/about/.

  9. 9.

    The study and its protocol were approved by the Ethics Committee of FORTH-ICS (Reference Number: 57/18-11-2019).

References

  1. Bransford, J., Sherwood, R., Vye, N., Rieser, J.: Teaching thinking and problem solving: research foundations. Am. Psychol. 41, 1078 (1986)

    Article  Google Scholar 

  2. Myth-Busting Reasons to Start Coding Even at an Older Age. https://www.makeuseof.com/tag/3-myth-busting-reasons-start-coding-even-older-age/. Accessed 20 Feb 2018

  3. What’s the Right Age for Kids to Learn to Code? https://people.howstuffworks.com/whats-the-right-age-kids-learn-code.htm. Accessed 20 Feb 2018

  4. Brush, A.J., Lee, B., Mahajan, R., Agarwal, S., Saroiu, S., Dixon, C.: Home automation in the wild: challenges and opportunities. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2115–2124. ACM (2011)

    Google Scholar 

  5. Dahl, Y., Svendsen, R.-M.: End-user composition interfaces for smart environments: a preliminary study of usability factors. In: Marcus, A. (ed.) DUXU 2011. LNCS, vol. 6770, pp. 118–127. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21708-1_14

    Chapter  Google Scholar 

  6. Dey, A.K., Sohn, T., Streng, S., Kodama, J.: iCAP: interactive prototyping of context-aware applications. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) Pervasive 2006. LNCS, vol. 3968, pp. 254–271. Springer, Heidelberg (2006). https://doi.org/10.1007/11748625_16

    Chapter  Google Scholar 

  7. Truong, K.N., Huang, E.M., Abowd, G.D.: CAMP: a magnetic poetry interface for end-user programming of capture applications for the home. In: Davies, N., Mynatt, E.D., Siio, I. (eds.) UbiComp 2004. LNCS, vol. 3205, pp. 143–160. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30119-6_9

    Chapter  Google Scholar 

  8. IFTTT. https://ifttt.com/. Accessed 05 Oct 2017

  9. Newman, M.W., Elliott, A., Smith, T.F.: Providing an integrated user experience of networked media, devices, and services through end-user composition. In: Indulska, J., Patterson, D.J., Rodden, T., Ott, M. (eds.) Pervasive 2008. LNCS, vol. 5013, pp. 213–227. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-79576-6_13

    Chapter  Google Scholar 

  10. Walch, M., Rietzler, M., Greim, J., Schaub, F., Wiedersheim, B., Weber, M.: homeBLOX: making home automation usable. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 295–298. ACM (2013)

    Google Scholar 

  11. De Russis, L., Corno, F.: HomeRules: A Tangible end-user programming interface for smart homes. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 2109–2114. ACM, New York (2015). https://doi.org/10.1145/2702613.2732795

  12. Zipato. https://www.zipato.com/. Accessed 07 Oct 2017

  13. Supermechanical: Twine. Listen to your world. Talk to the web. http://supermechanical.com/twine/technical.html. Accessed 06 Oct 2017

  14. Zapier. https://zapier.com/. Accessed 07 Oct 2017

  15. Stojkoska, B.L.R., Trivodaliev, K.V.: A review of internet of things for smart home: challenges and solutions. J. Clean. Prod. 140, 1454–1464 (2017)

    Article  Google Scholar 

  16. Ur, B., McManus, E., Pak Yong Ho, M., Littman, M.L.: Practical trigger-action programming in the smart home. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 803–812. ACM, New York (2014). https://doi.org/10.1145/2556288.2557420

  17. McNerney, T.S.: Tangible programming bricks: an approach to making programming accessible to everyone (1999)

    Google Scholar 

  18. Brosterman, N.: Inventing Kindergarten. Harry N. Abrams. Inc., Publishers, New York (1997)

    Google Scholar 

  19. Bruner, J.S.: Toward a Theory of Instruction. Harvard University Press, Cambridge (1966)

    Google Scholar 

  20. Papert, S.: Mindstorms: Children, computers, and Powerful Ideas. Basic Books, Inc., New York (1980)

    Google Scholar 

  21. Montemayor, J., Druin, A., Farber, A., Simms, S., Churaman, W., D’Amour, A.: Physical programming: designing tools for children to create physical interactive environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 299–306. ACM (2002)

    Google Scholar 

  22. Wilson, K.A., et al.: Relationships between game attributes and learning outcomes: review and research proposals. Simul. Gaming 40, 217–266 (2009)

    Article  Google Scholar 

  23. Oliveira, I., Tinoca, L., Pereira, A.: Online group work patterns: how to promote a successful collaboration. Comput. Educ. 57, 1348–1357 (2011)

    Article  Google Scholar 

  24. Vygotsky, L.S.: Mind in Society. Harvard University Press, Cambridge (1978)

    Google Scholar 

  25. Stefanidi, E., Arampatzis, D., Leonidis, A., Papagiannakis, G.: BricklAyeR: a platform for building rules for AmI environments in AR. In: Gavrilova, M., Chang, J., Thalmann, N.M., Hitzer, E., Ishikawa, H. (eds.) CGI 2019. LNCS, vol. 11542, pp. 417–423. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22514-8_39

    Chapter  Google Scholar 

  26. Vacchetti, L., et al.: A stable real-time AR framework for training and planning in industrial environments. In: Ong, S.K., Nee, A.Y.C. (eds.) Virtual and Augmented Reality Applications in Manufacturing, pp. 129–145. Springer, London (2004). https://doi.org/10.1007/978-1-4471-3873-0_8

    Chapter  Google Scholar 

  27. Kateros, S., Georgiou, S., Papaefthymiou, M., Papagiannakis, G., Tsioumas, M.: A comparison of gamified, immersive VR curation methods for enhanced presence and human-computer interaction in digital humanities. Int. J. Herit. Digit. Era 4, 221–233 (2015)

    Article  Google Scholar 

  28. Wang, D., Wang, T., Liu, Z.: A tangible programming tool for children to cultivate computational thinking. Sci. World J. 2014, 428080 (2014). https://doi.org/10.1155/2014/428080

    Article  Google Scholar 

  29. Davidoff, S., Lee, M.K., Yiu, C., Zimmerman, J., Dey, Anind K.: Principles of smart home control. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 19–34. Springer, Heidelberg (2006). https://doi.org/10.1007/11853565_2

    Chapter  Google Scholar 

  30. Stefanidi, E., Foukarakis, M., Arampatzis, D., Korozi, M., Leonidis, A., Antona, M.: ParlAm I: a multimodal approach for programming intelligent environments. Technologies 7, 11 (2019)

    Article  Google Scholar 

  31. Korozi, M., Leonidis, A., Antona, M., Stephanidis, C.: LECTOR: towards reengaging students in the educational process inside smart classrooms. In: Horain, P., Achard, C., Mallem, M. (eds.) IHCI 2017. LNCS, vol. 10688, pp. 137–149. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-72038-8_11

    Chapter  Google Scholar 

  32. Neumann, U., Majoros, A.: Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. In: Proceedings of the Virtual Reality Annual International Symposium, IEEE 1998, pp. 4–11. IEEE (1998)

    Google Scholar 

  33. Kasahara, S., Niiyama, R., Heun, V., Ishii, H.: exTouch: spatially-aware embodied manipulation of actuated objects mediated by augmented reality. In: Proceedings of the 7th International Conference on Tangible, Embedded and Embodied Interaction, pp. 223–228. ACM (2013)

    Google Scholar 

  34. Zaeh, M.F., Vogl, W.: Interactive laser-projection for programming industrial robots. In: IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR 2006, pp. 125–128. IEEE (2006)

    Google Scholar 

  35. Heun, V., Hobin, J., Maes, P.: Reality editor: programming smarter objects. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 307–310. ACM, New York (2013). https://doi.org/10.1145/2494091.2494185

  36. Heun, V., Kasahara, S., Maes, P.: Smarter objects: using AR technology to program physical objects and their interactions. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 961–966. ACM (2013)

    Google Scholar 

  37. Mateo, C., Brunete, A., Gambao, E., Hernando, M.: Hammer: an android based application for end-user industrial robot programming. In: 2014 IEEE/ASME 10th International Conference on Mechatronic and Embedded Systems and Applications (MESA), pp. 1–6. IEEE (2014)

    Google Scholar 

  38. Chong, J.W.S., Ong, S.K., Nee, A.Y.C., Youcef-Youmi, K.: Robot programming using augmented reality: an interactive method for planning collision-free paths. Robot. Comput.-Integr. Manuf. 25, 689–701 (2009). https://doi.org/10.1016/j.rcim.2008.05.002

    Article  Google Scholar 

  39. Brown, B., MacColl, I., Chalmers, M., Galani, A., Randell, C., Steed, A.: Lessons from the lighthouse: collaboration in a shared mixed reality system. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 577–584. ACM, New York (2003). https://doi.org/10.1145/642611.642711

  40. Kelleher, C., Pausch, R., Kiesler, S.: Storytelling alice motivates middle school girls to learn computer programming. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1455–1464. ACM (2007)

    Google Scholar 

  41. Resnick, M., et al.: Scratch: programming for all. Commun. ACM 52, 60–67 (2009)

    Article  Google Scholar 

  42. Vahldick, A., Mendes, A.J., Marcelino, M.J.: A review of games designed to improve introductory computer programming competencies. In: 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pp. 1–7 (2014). https://doi.org/10.1109/FIE.2014.7044114

  43. Aldrich, C.: Learning Online with Games, Simulations, and Virtual Worlds: Strategies for Online Instruction. Wiley, Hoboken (2009)

    Google Scholar 

  44. McTear, M., Callejas, Z., Griol, D.: Introducing the conversational interface. In: McTear, M., Callejas, Z., Griol, D. (eds.) the conversational interface, pp. 1–7. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-32967-3_1

    Chapter  Google Scholar 

  45. Van Merriënboer, J.J., Kirschner, P.A.: Ten Steps to Complex Learning: A Systematic Approach to Four-Component Instructional Design. Routledge, London (2017)

    Google Scholar 

  46. Alborzi, H., et al.: Designing StoryRooms: interactive storytelling spaces for children. In: Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, pp. 95–104. ACM (2000)

    Google Scholar 

  47. Seraj, M., Große, C.S., Autexier, S., Drechsler, R.: Smart homes programming: development and evaluation of an educational programming application for young learners. In: Proceedings of the 18th ACM International Conference on Interaction Design and Children, pp. 146–152 (2019)

    Google Scholar 

  48. Blockly. https://developers.google.com/blockly. Accessed 16 Mar 2020

  49. Blannin, J., Symons, D.: Algorithmic thinking in primary schools (2019). https://doi.org/10.1007/978-3-319-60013-0_128-1

  50. Tucker, A., Deek, F., Jones, J., McCowan, D., Stephenson, C., Verno, A.: A model curriculum for K-12 computer science. Final Report of the ACM K-12 Task Force Curriculum Committee, CSTA (2003)

    Google Scholar 

  51. Leonidis, A., et al.: Ambient intelligence in the living room. Sensors 19, 5011 (2019)

    Article  Google Scholar 

  52. Newman, M.W.: Now we’re cooking: Recipes for end-user service composition in the digital home (2006)

    Google Scholar 

  53. Weintrop, D., Wilensky, U.: To block or not to block, that is the question: students’ perceptions of blocks-based programming. In: Proceedings of the 14th International Conference on Interaction Design and Children, pp. 199–208. ACM (2015)

    Google Scholar 

  54. Stefanidi, E., Partarakis, N., Zabulis, X., Zikas, P., Papagiannakis, G., Thalmann, N.M.: TooltY: An approach for the combination of motion capture and 3D reconstruction to present tool usage in 3D environments. In: Thalmann, N.M., Zheng, J. (eds.) Intelligent Scene Modelling and Human Computer Interaction. Springer, Basel (2020)

    Google Scholar 

  55. Manca, M., Paternò, F., Santoro, C., Corcella, L.: Supporting end-user debugging of trigger-action rules for IoT applications. Int. J. Hum Comput Stud. 123, 56–69 (2019)

    Article  Google Scholar 

  56. Corno, F., De Russis, L., Monge Roffarello, A.: My IoT Puzzle: debugging IF-THEN rules through the jigsaw metaphor. In: Malizia, A., Valtolina, S., Morch, A., Serrano, A., Stratton, A. (eds.) IS-EUD 2019. LNCS, vol. 11553, pp. 18–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24781-2_2

    Chapter  Google Scholar 

  57. Laporte, L., Zaman, B.: A comparative analysis of programming games, looking through the lens of an instructional design model and a game attributes taxonomy. Entertain. Comput. 25, 48–61 (2017)

    Article  Google Scholar 

  58. Schmid, R.F., Miodrag, N., Francesco, N.D.: A human-computer partnership: The tutor/child/computer triangle promoting the acquisition of early literacy skills. J. Res. Technol. Educ. 41, 63–84 (2008)

    Article  Google Scholar 

  59. Greenlee, B.J., Karanxha, Z.: A study of group dynamics in educational leadership cohort and non-cohort groups. J. Res. Leadersh. Educ. 5, 357–382 (2010)

    Article  Google Scholar 

  60. Remagnino, P., Foresti, G.L.: Ambient intelligence: a new multidisciplinary paradigm. IEEE Trans. Syst. Man Cybern. - Part A: Syst. Hum. 35, 1–6 (2005). https://doi.org/10.1109/TSMCA.2004.838456

    Article  Google Scholar 

  61. Leonidis, A., Arampatzis, D., Louloudakis, N., Stephanidis, C.: The AmI-solertis system: creating user experiences in smart environments. In: Proceedings of the 13th IEEE International Conference on Wireless and Mobile Computing, Networking and Communications (2017)

    Google Scholar 

  62. Preuveneers, D., et al.: Towards an extensible context ontology for ambient intelligence. In: Markopoulos, P., Eggen, B., Aarts, E., Crowley, James L. (eds.) EUSAI 2004. LNCS, vol. 3295, pp. 148–159. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30473-9_15

    Chapter  Google Scholar 

  63. Core ML | Apple Developer Documentation. https://developer.apple.com/documentation/coreml. Accessed 31 Oct 2018

  64. Louloudakis, N.: AmITest : a framework for semi - automated testing of Ambient Intelligence environments (2017). https://elocus.lib.uoc.gr/dlib/e/f/b/metadata-dlib-1499846119-176002-29371.tkl

  65. Stefanidi, Z., Leonidis, A., Antona, M.: A multi-stage approach to facilitate interaction with intelligent environments via natural language. In: Stephanidis, C., Antona, M. (eds.) HCII 2019. CCIS, vol. 1088, pp. 67–77. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30712-7_9

    Chapter  Google Scholar 

  66. Norman, D.: The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York (2013)

    Google Scholar 

Download references

Acknowledgments

This work is supported by the FORTH-ICS internal RTD Programme ‘Ambient Intelligence Environments’.

Author information

Authors and Affiliations

  1. Institute of Computer Science (ICS), Foundation for Research and Technology – Hellas (FORTH), Heraklion, Greece

    Evropi Stefanidi, Dimitrios Arampatzis, Asterios Leonidis, Maria Korozi, Margherita Antona & George Papagiannakis

  2. Department of Computer Science, University of Crete, Heraklion, Greece

    George Papagiannakis

Authors
  1. Evropi Stefanidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dimitrios Arampatzis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Asterios Leonidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maria Korozi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Margherita Antona
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. George Papagiannakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: [ES, DA, AL, MK, GP]; Implementation: [ES, DA]; Evaluation experiment preparation and execution: [ES, DA, AL, MK]; Writing - original draft preparation: [ES, DA, AL, MK]; Writing - review and editing: [ΜΑ, GP]; Supervision: [AL, ΜΑ, GP].

Corresponding author

Correspondence to Asterios Leonidis .

Editor information

Editors and Affiliations

  1. University of Calgary, Calgary, AB, Canada

    Marina L. Gavrilova

  2. Sardina Systems OÜ, Tallinn, Estonia

    C. J. Kenneth Tan

  3. Bournemouth University, Poole, UK

    Jian Chang

  4. MiraLab, University of Geneva, Geneva, Switzerland

    Nadia Magnenat Thalmann

Ethics declarations

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer-Verlag GmbH Germany, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Stefanidi, E., Arampatzis, D., Leonidis, A., Korozi, M., Antona, M., Papagiannakis, G. (2020). MagiPlay: An Augmented Reality Serious Game Allowing Children to Program Intelligent Environments. In: Gavrilova, M., Tan, C., Chang, J., Thalmann, N. (eds) Transactions on Computational Science XXXVII. Lecture Notes in Computer Science(), vol 12230. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-61983-4_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-61983-4_9

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-61982-7

  • Online ISBN: 978-3-662-61983-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics