A Study of LLM Generated Line-by-Line Explanations in the Context of Conversational Program Comprehension Tutoring Systems | SpringerLink
Skip to main content
Springer Nature Link
Log in

A Study of LLM Generated Line-by-Line Explanations in the Context of Conversational Program Comprehension Tutoring Systems

  • Conference paper
  • First Online:
Technology Enhanced Learning for Inclusive and Equitable Quality Education (EC-TEL 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15159))

Included in the following conference series:

  • 633 Accesses

Abstract

This research paper explores to what extent large language models (LLMs) can generate line-by-line explanations of code examples used in intro-to-programming courses such as CS1 (Computer Science) and CS2. While it is known that LLMs can generate code explanations, a systematic analysis of those explanations and their appropriateness for instructional and learning purposes is needed, which is the goal of this paper. Specifically, the paper explores how different types of prompts impact the nature and quality of line-by-line explanations relative to human expert explanations. We report a quantitative and qualitative analysis that compares AI-generated explanations with explanations produced by human experts. Furthermore, we investigate to what degree LLM can generate explanations for learners of various levels of mastery.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9151
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11439
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdi, S., Khosravi, H., Sadiq, S., Demartini, G.: Evaluating the quality of learning resources: a learner sourcing approach. IEEE Trans. Learn. Technol. 14(1), 81–92 (2021)

    Article  Google Scholar 

  2. Aleven, V.A., Koedinger, K.R.: An effective metacognitive strategy: learning by doing and explaining with a computer-based cognitive tutor. Cogn. Sci. 26(2), 147–179 (2002)

    Google Scholar 

  3. Barke, S., James, M.B., Polikarpova, N.: Grounded copilot: how programmers interact with code-generating models. Proc. ACM Programm. Lang. 7(OOPSLA1), 85–111 (2023)

    Article  Google Scholar 

  4. Chi, M.T.: Self-explaining expository texts: the dual processes of generating inferences and repairing mental models. Adv. Instr. Psychol. 5, 161–238 (2000)

    Google Scholar 

  5. Denny, P., Kumar, V., Giacaman, N.: Conversing with copilot: exploring prompt engineering for solving CS1 problems using natural language. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education, vol. 1, pp. 1136–1142 (2023)

    Google Scholar 

  6. Denny, P., Luxton-Reilly, A., Simon, B.: Quality of student contributed questions using PeerWise. In: Proceedings of the Eleventh Australasian Conference on Computing Education-Volume 95, pp. 55–63 (2009)

    Google Scholar 

  7. Denny, P., Prather, J., Becker, B.A.: Error message readability and novice debugging performance. In: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2020, pp. 480–486. Association for Computing Machinery, New York, NY, USA (2020)

    Google Scholar 

  8. Denny, P., et al.: On designing programming error messages for novices: readability and its constituent factors. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021)

    Google Scholar 

  9. Finnie-Ansley, J., Denny, P., Becker, B., Luxton-Reilly, A., Prather, J.: The robots are coming: exploring the implications of OpenAI codex on introductory programming. In: Proceedings of the 24th Australasian Computing Education Conference, pp. 10–19 (2022)

    Google Scholar 

  10. Golchin, S., Surdeanu, M.: Time travel in LLMs: tracing data contamination in large language models. arXiv e-prints (2023)

    Google Scholar 

  11. Graesser, A.C., Chipman, P., Haynes, B.C., Olney, A.: AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans. Educ. 48(4), 612–618 (2005)

    Article  Google Scholar 

  12. Hicks, A., Akhuseyinoglu, K., Shaffer, C., Brusilovsky, P.: Live catalog of smart learning objects for computer science education. In: Sixth SPLICE Workshop “Building an Infrastructure for Computer Science Education Research and Practice at Scale” (2020)

    Google Scholar 

  13. Johansson, V.: Lexical diversity and lexical density in speech and writing: a developmental perspective. In: Working papers/Lund University, Department of Linguistics and Phonetics, vol. 53, pp. 61–79 (2008)

    Google Scholar 

  14. Leinonen, J., et al.: Comparing code explanations created by students and large language models (2023)

    Google Scholar 

  15. Leinonen, J., Pirttinen, N., Hellas, A.: Crowdsourcing content creation for SQL practice. In: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education, pp. 349–355 (2020)

    Google Scholar 

  16. MacNeil, S., et al.: Experiences from using code explanations generated by large language models in a web software development e-book. In: Proceedings of the 54th ACM Technical Symposium on Computer Science Education, vol. 1, pp. 931–937 (2023)

    Google Scholar 

  17. MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., Huang, Z.: Generating diverse code explanations using the GPT-3 large language model. In: Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2, pp. 37–39 (2022)

    Google Scholar 

  18. Murphy, L., McCauley, R., Fitzgerald, S.: ‘Explain in plain English’ questions: implications for teaching. In: Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, pp. 385–390 (2012)

    Google Scholar 

  19. Oli, P., Banjade, R., Chapagain, J., Rus, V.: The behavior of large language models when prompted to generate code explanations. arXiv preprint arXiv:2311.01490 (2023)

  20. Peng, H., Li, G., Zhao, Y., Jin, Z.: Rethinking positional encoding in tree transformer for code representation. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, December 2022

    Google Scholar 

  21. Pirttinen, N., Kangas, V., Nikkarinen, I., Nygren, H., Leinonen, J., Hellas, A.: Crowdsourcing programming assignments with CrowdSorcerer. In: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, pp. 326–331 (2018)

    Google Scholar 

  22. Roy, M., Chi, M.T.: The self-explanation principle in multimedia learning. In: The Cambridge Handbook of Multimedia Learning, pp. 271–286 (2005)

    Google Scholar 

  23. Rugaber, S.: The use of domain knowledge in program understanding. Ann. Softw. Eng. 9(1), 143–192 (2000)

    Article  Google Scholar 

  24. Rus, V., et al.: An intelligent tutoring system for source code comprehension. In: The 20th International Conference on Artificial Intelligence in Education, 25–29 June, Chicago, IL, USA (2019)

    Google Scholar 

  25. Rus, V., Niraula, N., Banjade, R.: DeepTutor: an effective, online intelligent tutoring system that promotes deep learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29 (2015)

    Google Scholar 

  26. Sarsa, S., Denny, P., Hellas, A., Leinonen, J.: Automatic generation of programming exercises and code explanations using large language models. In: Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1, pp. 27–43 (2022)

    Google Scholar 

  27. Sridhara, G., Hill, E., Muppaneni, D., Pollock, L., Vijay-Shanker, K.: Towards automatically generating summary comments for Java methods. In: Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineering, pp. 43–52 (2010)

    Google Scholar 

  28. Tack, A., Piech, C.: The AI teacher test: measuring the pedagogical ability of blender and GPT-3 in educational dialogues. arXiv preprint arXiv:2205.07540 (2022)

  29. Tian, H., et al.: Is ChatGPT the ultimate programming assistant – how far is it? (2023). arXiv:2304.11938

  30. Vaithilingam, P., Zhang, T., Glassman, E.L.: Expectation vs. experience: evaluating the usability of code generation tools powered by large language models. In: CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–7 (2022)

    Google Scholar 

  31. VanLehn, K., Jones, R.M., Chi, M.T.: A model of the self-explanation effect. J. Learn. Sci. 2(1), 1–59 (1992)

    Article  Google Scholar 

  32. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with BERT. arXiv preprint arXiv:1904.09675 (2019)

Download references

Acknowledgements

This work has been supported by the following grants awarded to Dr. Vasile Rus: the Learner Data Institute (NSF award 1934745); CSEdPad (NSF award 1822816); and iCODE (IES award R305A220385). The opinions, findings, and results are solely those of the authors and do not reflect those of NSF or IES.

Author information

Authors and Affiliations

  1. University of Memphis, Memphis, TN, 38152, USA

    Jeevan Chapagain, Mahmudul Islam Sajib & Vasile Rus

  2. University of Klagenfurt, Klagenfurt, Austria

    Radu Prodan

Authors
  1. Jeevan Chapagain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahmudul Islam Sajib
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Radu Prodan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vasile Rus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Radu Prodan .

Editor information

Editors and Affiliations

  1. Universidade Federal Rural de Pernambuco, Recife, Pernambuco, Brazil

    Rafael Ferreira Mello

  2. Ruhr University Bochum, Bochum, Germany

    Nikol Rummel

  3. FernUniversität in Hagen, Hagen, Germany

    Ioana Jivet

  4. University for Continuing Education Krems, Krems an der Donau, Austria

    Gerti Pishtari

  5. University of Murcia, Murcia, Spain

    José A. Ruipérez Valiente

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chapagain, J., Sajib, M.I., Prodan, R., Rus, V. (2024). A Study of LLM Generated Line-by-Line Explanations in the Context of Conversational Program Comprehension Tutoring Systems. In: Ferreira Mello, R., Rummel, N., Jivet, I., Pishtari, G., Ruipérez Valiente, J.A. (eds) Technology Enhanced Learning for Inclusive and Equitable Quality Education. EC-TEL 2024. Lecture Notes in Computer Science, vol 15159. Springer, Cham. https://doi.org/10.1007/978-3-031-72315-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72315-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72314-8

  • Online ISBN: 978-3-031-72315-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation