{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T18:47:42Z","timestamp":1735584462488},"reference-count":44,"publisher":"Wiley","issue":"5","license":[{"start":{"date-parts":[[2020,8,4]],"date-time":"2020-08-04T00:00:00Z","timestamp":1596499200000},"content-version":"am","delay-in-days":366,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#am"},{"start":{"date-parts":[[2019,8,4]],"date-time":"2019-08-04T00:00:00Z","timestamp":1564876800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/100006754","name":"Army Research Laboratory","doi-asserted-by":"publisher","award":["W911NF-18-2-0039"],"id":[{"id":"10.13039\/100006754","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["bera-journals.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Brit J Educational Tech"],"published-print":{"date-parts":[[2019,9]]},"abstract":"Abstract<\/jats:title>Learning\u2010in\u2010action<\/jats:italic> depends on interactions with learning content, peers and real world problems. However, effective learning\u2010in\u2010action also depends on the extent to which students are active\u2010in\u2010thinking<\/jats:italic>, making meaning of their learning experience. A critical component of any technology to support active thinking is the ability to ascertain whether (or to what extent) students have succeeded in internalizing the disciplinary strategies, norms of thinking, discourse practices and habits of mind that characterize deep understanding in a domain. This presents what we call a dilemma of modeling\u2010in\u2010context<\/jats:italic>: teachers routinely analyze this kind of thinking for small numbers of students in activities they create or customize for the needs of their students; however, doing so at scale and in real\u2010time requires some automated processes for modeling student work. Current techniques for developing models that reflect specific pedagogical activities and learning objectives that a teacher might create require either more expertise or more time than teachers have. In this paper, we examine a theoretical approach to addressing the problem of modeling active thinking in its pedagogical context that uses teacher\u2010created rubrics to generate models of student work. The results of this examination show how appropriately constructed learning technologies can enable teachers to develop custom automated rubrics for modeling active thinking and meaning\u2010making from the records of students' dialogic work.<\/jats:p><\/jats:sec>\nPractitioner Notes<\/jats:title><\/jats:caption>What is already known about this topic\n\nMany immersive educational technologies, such as digital games and simulations, enable students to take consequential action in a realistic context and to interact with peers, mentors and pedagogical agents. Such technologies help students to be active\u2010in\u2010thinking<\/jats:italic>: engaging deeply with, reflecting on and otherwise making meaning of their learning experience.<\/jats:p><\/jats:list-item>\nThere are now many immersive educational technologies with integrated authoring tools that enable teachers to customize the learning experience with relative ease, reducing barriers to adoption and improving student learning.<\/jats:p><\/jats:list-item>\nEducational technologies that support learning\u2010in\u2010action typically contain student models that operate in real\u2010time to control the behavior of pedagogical agents, deliver just\u2010in\u2010time interventions, select an appropriate content or otherwise measure and promote active thinking, but these student models may not work appropriately if teachers customize the learning experience.<\/jats:p><\/jats:list-item>\nMuch as there are authoring tools that allow teachers to customize the curriculum<\/jats:italic> of a given learning technology, there is a need for authoring tools that allow teachers to customize the associated student models<\/jats:italic> as well.<\/jats:p><\/jats:list-item>\n<\/jats:list><\/jats:p>What this paper adds\n\nThis paper presents a novel, rubric\u2010based approach to develop automated student models for new activities that teachers develop in digital learning environments that promote active thinking.<\/jats:p><\/jats:list-item>\nOur approach combines machine learning techniques with teacher expertise, allowing teachers to participate in the design of automated student models of active thinking that with further development could be scaled by leveraging their skills in rubric development.<\/jats:p><\/jats:list-item>\nOur results show that a rubric\u2010based approach can outperform a machine learning approach in this context. More importantly, in some cases, the rubric\u2010based approach can produce reliable automated models based on the information that a teacher can easily provide.<\/jats:p><\/jats:list-item>\n<\/jats:list><\/jats:p>Implications for practice and\/or policy\n\nIf integrated into authoring tools, the rubric\u2010based approach could allow teachers to participate in the design of automated models for educational technologies customized to their instructional needs.<\/jats:p><\/jats:list-item>\nThrough this design process, teachers could develop a better understanding of how the automated modeling system works, which in turn could increase the adoption of educational technologies that promote active thinking.<\/jats:p><\/jats:list-item>\nBecause the rubric\u2010based approach enables teachers to identify key connections among concepts relevant to the pedagogical context, rather than general concepts or linguistic features, it is more likely to facilitate targeted feedback to help promote the development of active thinking.<\/jats:p><\/jats:list-item>\n<\/jats:list><\/jats:p><\/jats:boxed-text>\n<\/jats:p><\/jats:sec>","DOI":"10.1111\/bjet.12869","type":"journal-article","created":{"date-parts":[[2019,8,5]],"date-time":"2019-08-05T00:13:07Z","timestamp":1564963987000},"page":"2346-2364","update-policy":"http:\/\/dx.doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Understanding when students are active\u2010in\u2010thinking through modeling\u2010in\u2010context"],"prefix":"10.1111","volume":"50","author":[{"given":"Zachari","family":"Swiecki","sequence":"first","affiliation":[]},{"ORCID":"http:\/\/orcid.org\/0000-0003-1382-4677","authenticated-orcid":false,"given":"Andrew R.","family":"Ruis","sequence":"additional","affiliation":[]},{"given":"Dipesh","family":"Gautam","sequence":"additional","affiliation":[]},{"given":"Vasile","family":"Rus","sequence":"additional","affiliation":[]},{"given":"David","family":"Williamson Shaffer","sequence":"additional","affiliation":[]}],"member":"311","published-online":{"date-parts":[[2019,8,4]]},"reference":[{"issue":"2","key":"e_1_2_8_2_1","first-page":"105","article-title":"A new paradigm for intelligent tutoring systems: Example\u2010tracing tutors","volume":"19","author":"Aleven V.","year":"2009","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.4018\/jgcms.2009010103"},{"key":"e_1_2_8_4_1","first-page":"199","volume-title":"Design recommendations for intelligent tutoring systems: Authoring tools","author":"Cai Z.","year":"2015"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/cae.21650"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.sbspro.2014.01.316"},{"key":"e_1_2_8_7_1","first-page":"47","volume-title":"Constructivism in the computer age","author":"DiSessa A. A.","year":"1988"},{"key":"e_1_2_8_8_1","first-page":"363","volume-title":"Proceedings of the 10th International Conference of the Learning Sciences","author":"Dyke G.","year":"2012"},{"key":"e_1_2_8_9_1","volume-title":"rhoR: Rho for inter rater reliability (Version 1.1.0)","author":"Eagan B. R.","year":"2016"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15326977ea0504_1"},{"key":"e_1_2_8_11_1","first-page":"278","volume-title":"Proceedings of the 10th International Conference on Educational Data Mining","author":"Gautam D.","year":"2017"},{"key":"e_1_2_8_12_1","volume-title":"Curriculum design for writing instruction: Creating standards\u2010based lesson plans and rubrics","author":"Glass K. T.","year":"2004"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-015-0086-4"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2005.856149"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1108\/S1534-085620180000019012"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1080\/10508406.2013.799475"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1098\/rspb.2001.1800"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/01638539809545028"},{"key":"e_1_2_8_19_1","first-page":"29","volume-title":"Internet environments for science education","author":"Linn M. C.","year":"2004"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03204766"},{"key":"e_1_2_8_21_1","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0000000000001962"},{"key":"e_1_2_8_22_1","doi-asserted-by":"publisher","DOI":"10.2190\/1RU5-HDTJ-A5C8-JVWE"},{"key":"e_1_2_8_23_1","first-page":"113","volume-title":"Proceedings of the 1st International Open Workshop on Intelligent Personalization and Adaptation in Digital Educational Games","author":"Mehm F.","year":"2009"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1080\/00098650009599436"},{"key":"e_1_2_8_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2729.2010.00385.x"},{"key":"e_1_2_8_26_1","volume-title":"A framework for K\u201012 science education: Practices, crosscutting concepts, and core ideas","author":"National Research Council","year":"2012"},{"key":"e_1_2_8_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-014-0029-5"},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1177\/1541931213601320"},{"key":"e_1_2_8_29_1","doi-asserted-by":"publisher","DOI":"10.1080\/02602930902862859"},{"issue":"1","key":"e_1_2_8_30_1","first-page":"115","article-title":"Integrating learning, problem solving, and engagement in narrative-centered learning environments","volume":"21","author":"Rowe J. P.","year":"2011","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"e_1_2_8_31_1","first-page":"136","volume-title":"Proceedings of the 13th International Conference on Computer\u2010Supported Collaborative Learning","author":"Ruis A. R.","year":"2019"},{"key":"e_1_2_8_32_1","first-page":"496","volume-title":"Proceedings of the 9th International Conference on Educational Data Mining","author":"Rus V.","year":"2016"},{"key":"e_1_2_8_33_1","first-page":"163","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Rus V.","year":"2013"},{"key":"e_1_2_8_34_1","volume-title":"Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence","author":"Rus V.","year":"2015"},{"key":"e_1_2_8_35_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2017.42.15"},{"key":"e_1_2_8_36_1","volume-title":"How computer games help children learn","author":"Shaffer D. W.","year":"2007"},{"key":"e_1_2_8_37_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139031127.028"},{"key":"e_1_2_8_38_1","volume-title":"Quantitative ethnography","author":"Shaffer D. W.","year":"2017"},{"key":"e_1_2_8_39_1","doi-asserted-by":"publisher","DOI":"10.4324\/9780203122761"},{"key":"e_1_2_8_40_1","doi-asserted-by":"publisher","DOI":"10.18608\/jla.2017.43.7"},{"issue":"2","key":"e_1_2_8_41_1","first-page":"119","article-title":"Automated domain model creation for adaptive social educational environments","volume":"3","author":"\u0160imko M.","year":"2011","journal-title":"Information Sciences and Technologies Bulletin of the ACM Slovakia"},{"key":"e_1_2_8_42_1","volume-title":"Design recommendations for intelligent tutoring systems: Authoring tools and expert modeling techniques","author":"Sottilare R.","year":"2015"},{"key":"e_1_2_8_43_1","volume-title":"Design recommendations for intelligent tutoring systems: Learner modeling","author":"Sottilare R.","year":"2013"},{"key":"e_1_2_8_44_1","first-page":"167","volume-title":"Game\u2010based learning: Theory, strategies and performance outcomes","author":"Swiecki Z.","year":"2017"},{"key":"e_1_2_8_45_1","first-page":"169","article-title":"Authoring conversation\u2010based assessment scenarios","volume":"3","author":"Zapata\u2010Rivera D.","year":"2015","journal-title":"Design Recommendations for Intelligent Tutoring Systems"}],"container-title":["British Journal of Educational Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/bjet.12869","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/bjet.12869","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/bera-journals.onlinelibrary.wiley.com\/doi\/am-pdf\/10.1111\/bjet.12869","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/bjet.12869","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,10]],"date-time":"2023-09-10T04:13:30Z","timestamp":1694319210000},"score":1,"resource":{"primary":{"URL":"https:\/\/bera-journals.onlinelibrary.wiley.com\/doi\/10.1111\/bjet.12869"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,4]]},"references-count":44,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,9]]}},"alternative-id":["10.1111\/bjet.12869"],"URL":"https:\/\/doi.org\/10.1111\/bjet.12869","archive":["Portico"],"relation":{},"ISSN":["0007-1013","1467-8535"],"issn-type":[{"value":"0007-1013","type":"print"},{"value":"1467-8535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,8,4]]},"assertion":[{"value":"2019-08-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}