Following the Impact Chain of the LA Cockpit

An Intervention Study Investigating a Teacher Dashboard’s Effect on Student Learning

Authors

DOI:

https://doi.org/10.18608/jla.2024.8399

Keywords:

teacher dashboards, K-12, intervention study, feedback systems, knowledge gain, authentic educational settings, research paper

Abstract

This paper presents a teacher dashboard intervention study in secondary school practice involving teachers (n = 16) with their classes (n = 22) and students (n = 403). A quasi-experimental treatment-control group design was implemented to compare student learning outcomes between classrooms where teachers did not have access to the dashboard and classrooms where teachers had access to the dashboard. We examined different points in the impact chain of the “LA Cockpit,” a teacher dashboard with a feedback system through which teachers can send feedback to their students on student learning. To investigate this impact chain from teacher use of dashboards to student learning, we analyzed 1) teachers’ perceived technology acceptance of the LA Cockpit, 2) teacher feedback practices using the LA Cockpit, and 3) student knowledge gains as measured by pre- and post-tests. The analysis of n = 355 feedback messages sent by teachers through the LA Cockpit revealed that the dashboard assists teachers in identifying students facing difficulties and that teachers mostly provided process feedback, which is known to be effective for student learning. For student learning, significantly higher knowledge gains were found in the teacher dashboard condition compared to the control condition.

References

Ahmad, A., Schneider, J., Weidlich, J., Di Mitri, D., Yau, J. Y.-K., Schiffner, D., & Drachsler, H. (2022). What indicators can I serve you with? An evaluation of a research-driven learning analytics indicator repository. Proceedings of the 14th International Conference on Computer Supported Education (CSEDU 2022), 22–24 April 2022, Online (Vol. 1, pp. 58–68). ScitePress. https://doi.org/10.5220/0010995800003182

Al-Emran, M., Mezhuyev, V., & Kamaludin, A. (2018). Technology acceptance model in M-learning context: A systematic review. Computers & Education, 125, 389–412. https://doi.org/10.1016/j.compedu.2018.06.008

Banihashem, S. K., Noroozi, O., van Ginkel, S., Macfadyen, L. P., & Biemans, H. J. A. (2022). A systematic review of the role of learning analytics in enhancing feedback practices in higher education. Educational Research Review, 37, 100489. https://doi.org/10.1016/j.edurev.2022.100489

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge. https://doi.org/10.4324/9781315814698

Broadfoot, P., Daugherty, R., Gardner, J., Harlen, W., James, M., & Stobart, G. (2002). Assessment for learning: 10 principles: Research-based principles to guide classroom practice. Assessment Reform Group.

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245

Campos, F. C., Ahn, J., DiGiacomo, D. K., Nguyen, H., & Hays, M. (2021). Making sense of sensemaking: Understanding how K–12 teachers and coaches react to visual analytics. Journal of Learning Analytics, 8(3), 60–80. https://doi.org/10.18608/jla.2021.7113

Carless, D., & Winstone, N. (2023). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education, 28(1), 150–163. https://doi.org/10.1080/13562517.2020.1782372

Cohen, J. (1988). Set correlation and contingency tables. Applied Psychological Measurement, 12(4), 425–434. https://doi.org/10.1177/014662168801200410

Davis, F. D. (1985). A technology acceptance model for empirically testing new end-user information systems: Theory and results [Unpublished doctoral dissertation]. Massachusetts Institute of Technology. http://hdl.handle.net/1721.1/15192

DeMars, C. (2010). Item response theory. Oxford Academic. https://doi.org/10.1093/acprof:oso/9780195377033.001.0001

Gombert, S., Di Mitri, D., Karademir, O., Kubsch, M., Kolbe, H., Tautz, S., Grimm, A., Bohm, I., Neumann, K., & Drachsler, H. (2023). Coding energy knowledge in constructed responses with explainable NLP models. Journal of Computer Assisted Learning, 39(3), 767–786. https://doi.org/10.1111/jcal.12767

Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology & Society, 15(3), 42–57.

Hattie, J. (2012). Visible learning for teachers: Maximizing impact on learning. Routledge.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Hoth, J., Döhrmann, M., Kaiser, G., Busse, A., König, J., & Blömeke, S. (2016). Diagnostic competence of primary school mathematics teachers during classroom situations. ZDM, 48(1–2), 41–53. https://doi.org/10.1007/s11858-016-0759-y

Iraj, H., Fudge, A., Faulkner, M., Pardo, A., & Kovanović, V. (2020). Understanding students’ engagement with personalised feedback messages. Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 438–447). https://doi.org/10.1145/3375462.3375527

Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 82–96). Lecture Notes in Computer Science, Springer. https://doi.org/10.1007/978-3-319-66610-5_7

Kaliisa, R., Jivet, I., & Prinsloo, P. (2023). A checklist to guide the planning, designing, implementation, and evaluation of learning analytics dashboards. International Journal of Educational Technology in Higher Education, 20(1), 28. https://doi.org/10.1186/s41239-023-00394-6

Karademir, O., Di Mitri, D., Schneider, J., Jivet, I., Allmang, J., Gombert, S., Kubsch, M., Neumann, K., & Drachsler, H. (2024). I don’t have time! But keep me in the loop: Co-designing requirements for a learning analytics cockpit with teachers. Journal of Computer Assisted Learning. https://doi.org/10.1111/jcal.12997

Knoop-van Campen, C. A. N., Wise, A., & Molenaar, I. (2023). The equalizing effect of teacher dashboards on feedback in K–12 classrooms. Interactive Learning Environments, 31(6), 3447–3463. https://doi.org/10.1080/10494820.2021.1931346

Knoop-van Campen, C., & Molenaar, I. (2020). How teachers integrate dashboards into their feedback practices. Frontline Learning Research, 8(4), 37–51. https://doi.org/10.14786/flr.v8i4.641

Krajcik, J. S., & Shin, N. (2014). Project-based learning. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 275–297). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.018

Kubsch, M., Czinczel, B., Lossjew, J., Wyrwich, T., Bednorz, D., Bernholt, S., Fiedler, D., Strauß, S., Cress, U., Drachsler, H., Neumann, K., & Rummel, N. (2022). Toward learning progression analytics: Developing learning environments for the automated analysis of learning using evidence centered design. Frontiers in Education, 7, 981910. https://www.frontiersin.org/articles/10.3389/feduc.2022.981910

Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33(2), 363–374.

Li, S. (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60(2), 309–365. https://doi.org/10.1111/j.1467-9922.2010.00561.x

Lipnevich, A. A., & Panadero, E. (2021). A review of feedback models and theories: Descriptions, definitions, and conclusions. Frontiers in Education, 6, 720195. https://www.frontiersin.org/article/10.3389/feduc.2021.720195

Lodge, J. M., Kennedy, G., Lockyer, L., Arguel, A., & Pachman, M. (2018). Understanding difficulties and resulting confusion in learning: An integrative review. Frontiers in Education, 3, 49. https://doi.org/10.3389/feduc.2018.00049

Magill, R. A. (1994). The influence of augmented feedback on skill learning depends on characteristics of the skill and the learner. Quest, 46(3), 314–327. https://doi.org/10.1080/00336297.1994.10484129

Martinez-Maldonado, R., Clayphan, A., Yacef, K., & Kay, J. (2015). MTFeedback: Providing notifications to enhance teacher awareness of small group work in the classroom. IEEE Transactions on Learning Technologies, 8(2), 187–200. https://doi.org/10.1109/TLT.2014.2365027

Molenaar, I., & Knoop-van Campen, C. (2017). Teacher dashboards in practice: Usage and impact. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 125–138). Springer Cham. https://doi.org/10.1007/978-3-319-66610-5_10

Molenaar, I., & Knoop-van Campen, C. A. N. (2019). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355. https://doi.org/10.1109/TLT.2018.2851585

Motz, B. A., Bergner, Y., Brooks, C. A., Gladden, A., Gray, G., Lang, C., Li, W., Marmolejo-Ramos, F., & Quick, J. D. (2023). A LAK of direction: Misalignment between the goals of learning analytics and its research scholarship. Journal of Learning Analytics, 10(2), 1–13. https://doi.org/10.18608/jla.2023.7913

Neumann, K., Viering, T., Boone, W. J., & Fischer, H. E. (2013). Towards a learning progression of energy. Journal of Research in Science Teaching, 50(2), 162–188. https://doi.org/10.1002/tea.21061

Ngoc Anh, B., Tung Son, N., Truong Lam, P., Phuong Chi, L., Huu Tuan, N., Cong Dat, N., Huu Trung, N., Umar Aftab, M., & Van Dinh, T. (2019). A computer-vision based application for student behavior monitoring in classroom. Applied Sciences, 9(22), 4729. https://doi.org/10.3390/app9224729

Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. The International Review of Research in Open and Distributed Learning, 19(5). https://doi.org/10.19173/irrodl.v19i5.3493

Schrader, F.-W., & Helmke, A. (2001). Alltägliche leistungsbeurteilung durch lehrer [Everyday performance assessment by teachers]. In F. E. Weinert (Ed.), Leistungsmessungen in schulen [Performance measurements in schools] (pp. 45–58). Beltz.

Sergis, S., & Sampson, D. G. (2017). Teaching and learning analytics to support teacher inquiry: A systematic literature review. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends: A view of the current state of the art to enhance e-learning (pp. 25–63). Springer International Publishing. https://doi.org/10.1007/978-3-319-52977-6_2

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 31–40.

van Leeuwen, A. (2019). Teachers’ perceptions of the usability of learning analytics reports in a flipped university course: When and how does information become actionable knowledge? Educational Technology Research and Development, 67(5), 1043–1064. https://doi.org/10.1007/s11423-018-09639-y

van Leeuwen, A., Rummel, N., & van Gog, T. (2019). What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? International Journal of Computer-Supported Collaborative Learning, 14(3), 261–289. https://doi.org/10.1007/s11412-019-09299-x

van Leeuwen, A., Teasley, S. D., & Wise, A. F. (2022). Teacher and student facing learning analytics. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics (2nd ed., pp. 130–140). Society for Learning Analytics Research. https://doi.org/10.18608/hla22.013

Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning analytics dashboard applications. American Behavioral Scientist, 57(10), 1500–1509. https://doi.org/10.1177/0002764213479363

Viberg, O., & Grönlund, Å. (2023). Practicable learning analytics. Springer.

Watts, S. A. (2007). Evaluative feedback: Perspectives on media effects. Journal of Computer-Mediated Communication, 12(2), 384–411. https://doi.org/10.1111/j.1083-6101.2007.00330.x

Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://doi.org/10.18608/jla.2019.62.4

Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087. https://doi.org/10.3389/fpsyg.2019.03087

Xhakaj, F., Aleven, V., & McLaren, B. M. (2017). Effects of a teacher dashboard for an intelligent tutoring system on teacher knowledge, lesson planning, lessons and student learning. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 315–329). Springer International Publishing. https://doi.org/10.1007/978-3-319-66610-5_23

Downloads

Published

2024-07-25

How to Cite

Karademir, O., Borgards, L., Di Mitri, D., Strauß, S., Kubsch, M., Brobeil, M., Grimm, A., Gombert, S., Rummel, N., Neumann, K., & Drachsler, H. (2024). Following the Impact Chain of the LA Cockpit: An Intervention Study Investigating a Teacher Dashboard’s Effect on Student Learning. Journal of Learning Analytics, 11(2), 215-228. https://doi.org/10.18608/jla.2024.8399

Issue

Section

Research Papers

Most read articles by the same author(s)

1 2 > >>