Abstract
Despite the momentum that Artificial Intelligence (AI) is gaining in education, its role and impact on teachers’ learning design practices are still underexplored. This paper reports an experimental study (N = 38) taking place in a teacher training where an AI-driven feedback system aided teachers in the creation of learning designs. The study analyses the impact that using the AI feedback had on the quality of designs that teachers created, and the usability evaluation of the system. We noticed statistically significant differences between the designs created by the randomly assigned teachers in the experimental (using AI) and control group (without AI), suggesting that AI algorithms specialized to perform specific tasks related to the learning design could help teachers to better meet their design goals. While teachers graded the usability of the feedback system as above average, they also found it easy to use and its functions well integrated. In open-ended questions, teachers expressed doubts about their trust in AI systems and the impact that they may have in school communities, suggesting that future work should explore not only the long-term impact that using AI can have on teachers’ design practices, but also on their perceptions and understanding of the technology.
This research has been partially funded by the European Union in the context of Twinning (EuropeAid/160712/ID/ACT/ACT/DZ), and the Estonian Research Council’s Personal Research Grant (PRG1634).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Brooke, J., et al.: SUS - a quick and dirty usability scale. Usabil. Eval. Ind. 189(194), 4–7 (1996)
Cavalcanti, A.P., et al.: Automatic feedback in online learning environments: a systematic literature review. Comput. Educ. Artif. Intell. 2, 100027 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Di Mitri, D., Schneider, J., Specht, M., Drachsler, H.: From signals to knowledge: a conceptual model for multimodal learning analytics. J. Comput. Assist. Learn. 34(4), 338–349 (2018)
Hernández-Leo, D., et al.: An integrated environment for learning design. Front. ICT 5, 9 (2018)
Holstein, K., Aleven, V., Rummel, N.: A conceptual framework for human–AI hybrid adaptivity in education. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) AIED 2020. LNCS (LNAI), vol. 12163, pp. 240–254. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52237-7_20
Krathwohl, D.R.: A revision of bloom’s taxonomy: an overview. Theory Pract. 41(4), 212–218 (2002)
Kratochwill, T.R., et al.: Single-case intervention research design standards. Remedial Spec. Educ. 34(1), 26–38 (2013)
Lockyer, L., Heathcote, E., Dawson, S.: Informing pedagogical action: aligning learning analytics with learning design. Am. Behav. Sci. 57(10), 1439–1459 (2013)
Maguire, M., Delahunt, B.: Doing a thematic analysis: a practical, step-by-step guide for learning and teaching scholars. All Ireland J. High. Educ. 9(3) (2017)
Mavrikis, M., Cukurova, M., Di Mitri, D., Schneider, J., Drachsler, H.: A short history, emerging challenges and co-operation structures for artificial intelligence in education. Bildung und Erziehung 74(3), 249–263 (2021)
Molenaar, I.: Towards hybrid human-AI learning technologies. Eur. J. Educ. 57(4), 632–645 (2022)
Moreno-Marcos, P.M., Muñoz-Merino, P.J., Maldonado-Mahauad, J., Perez-Sanagustin, M., Alario-Hoyos, C., Kloos, C.D.: Temporal analysis for dropout prediction using self-regulated learning strategies in self-paced MOOCs. Comput. Educ. 145, 103728 (2020)
Pedaste, M., et al.: Phases of inquiry-based learning: definitions and the inquiry cycle. Educ. Res. Rev. 14, 47–61 (2015)
Perez-Alvarez, R., Jivet, I., Pérez-Sanagustin, M., Scheffel, M., Verbert, K.: Tools designed to support self-regulated learning in online learning environments: a systematic review. IEEE Trans. Learn. Technol. (2022)
Persico, D., Pozzi, F.: Informing learning design with learning analytics to improve teacher inquiry. Br. J. Edu. Technol. 46(2), 230–248 (2015)
Pishtari, G., Prieto, L.P., Rodríguez-Triana, M.J., Martinez-Maldonado, R.: Design analytics for mobile learning: scaling up the classification of learning designs based on cognitive and contextual elements. J. Learn. Anal. 9(2), 236–252 (2022)
Pishtari, G., et al.: Learning design and learning analytics in mobile and ubiquitous learning: a systematic review. Br. J. Educ. Technol. 51(4), 1078–1100 (2020)
Pishtari, G., Rodríguez-Triana, M.J.: An analysis of mobile learning tools in terms of pedagogical affordances and support to the learning activity life cycle. In: Gil, E., Mor, Y., Dimitriadis, Y., Köppe, C. (eds.) Hybrid Learning Spaces. UTLP, pp. 167–183. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-88520-5_10
Pishtari, G., Rodríguez-Triana, M.J., Prieto, L.P., Ruiz-Calleja, A., Väljataga, T.: What kind of learning designs do practitioners create for mobile learning? Lessons learnt from two in-the-wild case studies. J. Comput. Assist. Learn. (2022)
Pishtari, G., Rodríguez-Triana, M.J., Väljataga, T.: A multi-stakeholder perspective of analytics for learning design in location-based learning. Int. J. Mob. Blend. Learn. (IJMBL) 13(1), 1–17 (2021)
Rodríguez-Triana, M.J., Prieto, L.P., Pishtari, G.: What do learning designs show about pedagogical adoption? An analysis approach and a case study on inquiry-based learning. In: De Laet, T., Klemke, R., Alario-Hoyos, C., Hilliger, I., Ortega-Arranz, A. (eds.) EC-TEL 2021. LNCS, vol. 12884, pp. 275–288. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86436-1_21
Sharples, M.: Making sense of context for mobile learning. In: Mobile Learning, pp. 140–153. Routledge, London (2015)
Sie, R.L., et al.: Artificial intelligence to enhance learning design in UOW online, a unified approach to fully online learning. In: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 761–767. IEEE (2018)
Spikol, D., Ruffaldi, E., Dabisias, G., Cukurova, M.: Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. J. Comput. Assist. Learn. 34(4), 366–377 (2018)
Tlili, A., et al.: What if the devil is my guardian angel: Chatgpt as a case study of using chatbots in education. Smart Learn. Environ. 10(1), 15 (2023)
Vanlommel, K., Van Gasse, R., Vanhoof, J., Van Petegem, P.: Teachers’ decision-making: data based or intuition driven? Int. J. Educ. Res. 83, 75–83 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pishtari, G., Sarmiento-Márquez, E.M., Rodríguez-Triana, M.J., Wagner, M., Ley, T. (2023). Evaluating the Impact and Usability of an AI-Driven Feedback System for Learning Design. In: Viberg, O., Jivet, I., Muñoz-Merino, P., Perifanou, M., Papathoma, T. (eds) Responsive and Sustainable Educational Futures. EC-TEL 2023. Lecture Notes in Computer Science, vol 14200. Springer, Cham. https://doi.org/10.1007/978-3-031-42682-7_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-42682-7_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42681-0
Online ISBN: 978-3-031-42682-7
eBook Packages: Computer ScienceComputer Science (R0)