PapagAI: Automated Feedback for Reflective Essays | SpringerLink
Skip to main content
Springer Nature Link
Log in

PapagAI: Automated Feedback for Reflective Essays

  • Conference paper
  • First Online:
KI 2023: Advances in Artificial Intelligence (KI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14236))

Included in the following conference series:

Abstract

Written reflective practice is a regular exercise pre-service teachers perform during their higher education. Usually, their lecturers are expected to provide individual feedback, which can be a challenging task to perform on a regular basis. In this paper, we present the first open-source automated feedback tool based on didactic theory and implemented as a hybrid AI system. We describe the components and discuss the advantages and disadvantages of our system compared to the state-of-art generative large language models. The main objective of our work is to enable better learning outcomes for students and to complement the teaching activities of lecturers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 6634
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 8293
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    All ML models are available in our OSF depository (https://osf.io/ytesn/), while linguistic processing code can be shared upon request.

  2. 2.

    This still non-published data can be obtained upon request.

  3. 3.

    https://spacy.io.

  4. 4.

    We use Connective-Lex list for German: https://doi.org/10.4000/discours.10098.

  5. 5.

    https://pypi.org/project/deep-translator/.

References

  1. Batbaatar, E., Li, M., Ryu, K.H.: Semantic-emotion neural network for emotion recognition from text. IEEE Access 7, 111866–111878 (2019). https://doi.org/10.1109/ACCESS.2019.2934529

    Article  Google Scholar 

  2. Becker, A.: 83 Prozent der Studenten brechen Lehramts-Studium ab. Nordkurier (2021)

    Google Scholar 

  3. Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc., Sebastopol (2009)

    Google Scholar 

  4. Brown, T.B., et al.: Language models are few-shot learners (2020)

    Google Scholar 

  5. Cevher, D., Zepf, S., Klinger, R.: Towards multimodal emotion recognition in German speech events in cars using transfer learning (2019)

    Google Scholar 

  6. Chen, Y., Yu, B., Zhang, X., Yu, Y.: Topic modeling for evaluating students’ reflective writing: a case study of pre-service teachers’ journals. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, LAK 2016, pp. 1–5. Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2883851.2883951

  7. Chiorrini, A., Diamantini, C., Mircoli, A., Potena, D.: Emotion and sentiment analysis of tweets using bert. In: EDBT/ICDT Workshops (2021)

    Google Scholar 

  8. De Lin, O., Gottipati, S., Ling, L.S., Shankararaman, V.: Mining informal & short student self-reflections for detecting challenging topics - a learning outcomes insight dashboard. In: 2021 IEEE Frontiers in Education Conference (FIE), pp. 1–9 (2021). https://doi.org/10.1109/FIE49875.2021.9637181

  9. Ekman, P.: Basic emotions. and book of cognition and emotion 98, 16 (2023)

    Google Scholar 

  10. Elands, P., Huizing, A., Kester, J., Peeters, M.M.M., Oggero, S.: Governing ethical and effective behaviour of intelligent systems: a novel framework for meaningful human control in a military context. Militaire Spectator 188(6), 302–313 (2019)

    Google Scholar 

  11. Fleck, R., Fitzpatrick, G.: Reflecting on reflection: framing a design landscape. In: Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction, OZCHI 2010, pp. 216–223. Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1952222.1952269

  12. Geden, M., Emerson, A., Carpenter, D., Rowe, J.P., Azevedo, R., Lester, J.C.: Predictive student modeling in game-based learning environments with word embedding representations of reflection. Int. J. Artif. Intell. Educ. 31, 1–23 (2021)

    Article  Google Scholar 

  13. Gibbs, G., Unit, G.B.F.E.: Learning by Doing: A Guide to Teaching and Learning Methods. FEU. Oxford Brookes University, Oxford (1988)

    Google Scholar 

  14. Grootendorst, M.R.: Bertopic: neural topic modeling with a class-based tf-idf procedure. ArXiv (2022)

    Google Scholar 

  15. Guhr, O., Schumann, A.K., Bahrmann, F., Böhme, H.J.: Training a broad-coverage German sentiment classification model for dialog systems. In: Proceedings of the 12th Language Resources and Evaluation Conference, pp. 1627–1632. European Language Resources Association, Marseille, France, May 2020. https://aclanthology.org/2020.lrec-1.202

  16. Jena, R.K.: Sentiment mining in a collaborative learning environment: capitalising on big data. Behav. Inf. Technol. 38(9), 986–1001 (2019). https://doi.org/10.1080/0144929X.2019.1625440

  17. Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12), 1–38 (2023). https://doi.org/10.1145/3571730

  18. Jung, Y., Wise, A.F.: How and how well do students reflect?: multi-dimensional automated reflection assessment in health professions education. In: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (2020)

    Google Scholar 

  19. Klamm, C., Rehbein, I., Ponzetto, S.: Frameast: a framework for second-level agenda setting in parliamentary debates through the lense of comparative agenda topics. ParlaCLARIN III at LREC2022 (2022)

    Google Scholar 

  20. Klemm, K., Zorn, D.: Steigende Schülerzahlen im Primarbereich: Lehrkräftemangel deutlich stärker als von der KMK erwartet. Bertelsmann Stiftung, September 2019

    Google Scholar 

  21. Knight, S., et al.: Acawriter: a learning analytics tool for formative feedback on academic writing. J. Writing Res. 12(1), 141–186 (2020). https://doi.org/10.17239/jowr-2020.12.01.06

  22. Kovanović, V., et al.: Understand students’ self-reflections through learning analytics. In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge, LAK 2018, pp. 389–398. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3170358.3170374

  23. Liu, M., Kitto, K., Buckingham Shum, S.: Combining factor analysis with writing analytics for the formative assessment of written reflection. Comput. Hum. Behav. 120, 106733 (2021). https://doi.org/10.1016/j.chb.2021.106733

    Article  Google Scholar 

  24. Liu, M., Shum, S.B., Mantzourani, E., Lucas, C.: Evaluating machine learning approaches to classify pharmacy students’ reflective statements. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019, Part I. LNCS (LNAI), vol. 11625, pp. 220–230. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23204-7_19

    Chapter  Google Scholar 

  25. Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach. ArXiv (2019)

    Google Scholar 

  26. Lui, M., Baldwin, T.: langid.py: an off-the-shelf language identification tool. In: Proceedings of the ACL 2012 System Demonstrations. pp. 25–30. Association for Computational Linguistics, Jeju Island, Korea (2012). https://aclanthology.org/P12-3005

  27. Manakul, P., Liusie, A., Gales, M.J.F.: Selfcheckgpt: zero-resource black-box hallucination detection for generative large language models (2023)

    Google Scholar 

  28. McCallum, A.K.: Mallet: a machine learning for language toolkit. https://mallet.cs.umass.edu (2002)

  29. Napanoy, J., Gayagay, G., Tuazon, J.: Difficulties encountered by pre-service teachers: basis of a pre-service training program. Univ. J. Educ. Res. 9, 342–349 (2021). https://doi.org/10.13189/ujer.2021.090210

  30. OpenAI: Gpt-4 technical report (2023)

    Google Scholar 

  31. Plutchik, R.: A psychoevolutionary theory of emotions. Soc. Sci. Inf. 21(4–5), 529–553 (1982). https://doi.org/10.1177/053901882021004003

    Article  Google Scholar 

  32. Schmid, H., Laws, F.: Estimation of conditional probabilities with decision trees and an application to fine-grained POS tagging. In: Proceedings of the 22nd International Conference on Computational Linguistics - COLING 2008. Association for Computational Linguistics, Morristown, NJ, USA (2008)

    Google Scholar 

  33. Shashkov, A., Gold, R., Hemberg, E., Kong, B., Bell, A., O’Reilly, U.M.: Analyzing student reflection sentiments and problem-solving procedures in moocs. In: Proceedings of the Eighth ACM Conference on Learning @ Scale, L@S 2021, pp. 247–250. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3430895.3460150

  34. Sidarenka, U.: PotTS: the potsdam twitter sentiment corpus. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 1133–1141. European Language Resources Association (ELRA), Portorož, Slovenia, May 2016. https://aclanthology.org/L16-1181

  35. Solopova, V., Popescu, O.I., Chikobava, M., Romeike, R., Landgraf, T., Benzmüller, C.: A German corpus of reflective sentences. In: Proceedings of the 18th International Conference on Natural Language Processing (ICON), pp. 593–600. NLP Association of India (NLPAI), National Institute of Technology Silchar, Silchar, India, December 2021. https://aclanthology.org/2021.icon-main.72

  36. Ullmann, T.: Automated analysis of reflection in writing: validating machine learning approaches. Int. J. Artif. Intell. Educ. 29 (2019). https://doi.org/10.1007/s40593-019-00174-2

  37. Wojatzki, M., Ruppert, E., Holschneider, S., Zesch, T., Biemann, C.: GermEval 2017: shared task on aspect-based sentiment in social media customer feedback. In: Proceedings of the GermEval 2017 - Shared Task on Aspect-based Sentiment in Social Media Customer Feedback, pp. 1–12. Berlin, Germany (2017)

    Google Scholar 

  38. Wulff, D., et al.: Computer-based classification of preservice physics teachers’ written reflections. J. Sci. Educ. Technol. 30, 1–15 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Freie Universität Berlin, Berlin, Germany

    Veronika Solopova, Eiad Rostom, Fritz Cremer, Adrian Gruszczynski, Sascha Witte, Fernando Ramos López, Ralf Romeike & Tim Landgraf

  2. Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

    Chengming Zhang, Lea Plößl, Florian Hofmann, Michaela Gläser-Zikuda & Christoph Benzmüller

  3. Otto-Friedrich-Universität Bamberg, Bamberg, Germany

    Christoph Benzmüller

Authors
  1. Veronika Solopova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eiad Rostom
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fritz Cremer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adrian Gruszczynski
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sascha Witte
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chengming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Fernando Ramos López
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lea Plößl
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Florian Hofmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Ralf Romeike
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Michaela Gläser-Zikuda
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Christoph Benzmüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Tim Landgraf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Veronika Solopova .

Editor information

Editors and Affiliations

  1. Universität Würzburg, Würzburg, Germany

    Dietmar Seipel

  2. University of Greifswald, Greifswald, Germany

    Alexander Steen

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Solopova, V. et al. (2023). PapagAI: Automated Feedback for Reflective Essays. In: Seipel, D., Steen, A. (eds) KI 2023: Advances in Artificial Intelligence. KI 2023. Lecture Notes in Computer Science(), vol 14236. Springer, Cham. https://doi.org/10.1007/978-3-031-42608-7_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42608-7_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42607-0

  • Online ISBN: 978-3-031-42608-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation