{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,12]],"date-time":"2024-09-12T15:56:34Z","timestamp":1726156594054},"publisher-location":"New York, NY, USA","reference-count":38,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,6,29]]},"DOI":"10.1145\/3587102.3588792","type":"proceedings-article","created":{"date-parts":[[2023,6,30]],"date-time":"2023-06-30T22:17:59Z","timestamp":1688163479000},"update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":43,"title":["Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?"],"prefix":"10.1145","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-3674-5456","authenticated-orcid":false,"given":"Jaromir","family":"Savelka","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-9848-1663","authenticated-orcid":false,"given":"Arav","family":"Agarwal","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-8581-115X","authenticated-orcid":false,"given":"Christopher","family":"Bogart","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"ORCID":"http:\/\/orcid.org\/0009-0004-6205-2655","authenticated-orcid":false,"given":"Yifan","family":"Song","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"ORCID":"http:\/\/orcid.org\/0000-0001-5150-8259","authenticated-orcid":false,"given":"Majd","family":"Sakr","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]}],"member":"320","published-online":{"date-parts":[[2023,6,30]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Issue","author":"Ankur Desai","year":"2022","unstructured":"Desai Ankur and Deo Atul. 2022. Introducing Amazon CodeWhisperer, the ML-powered Coding Companion. AWS Machine Learning Blog (2022). Issue June 24, 2022. https:\/\/aws.amazon.com\/blogs\/machine-learning\/introducing-amazon-codewhisperer-the-ml-powered-coding-companion\/"},{"key":"e_1_3_2_1_2_1","volume-title":"Grounded Copilot: How Programmers Interact with Code-Generating Models. ArXiv abs\/2206.15000","author":"Barke Shraddha","year":"2022","unstructured":"Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2022. Grounded Copilot: How Programmers Interact with Code-Generating Models. ArXiv abs\/2206.15000 (2022)."},{"key":"e_1_3_2_1_3_1","volume-title":"Programming Is Hard--Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation. arXiv preprint arXiv:2212.01020","author":"Becker Brett A","year":"2022","unstructured":"Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2022. Programming Is Hard--Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation. arXiv preprint arXiv:2212.01020 (2022)."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557079"},{"key":"e_1_3_2_1_5_1","volume-title":"Daniel Martin Katz, and Jessica Katz","author":"Bommarito Jillian","year":"2023","unstructured":"Jillian Bommarito, Michael Bommarito, Daniel Martin Katz, and Jessica Katz. 2023. GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI) CPA Capabilities. arXiv preprint arXiv:2301.04408 (2023)."},{"key":"e_1_3_2_1_6_1","volume-title":"GPT Takes the Bar Exam. arXiv preprint arXiv:2212.14402","author":"Daniel Martin Katz Michael","year":"2022","unstructured":"Michael Bommarito II and Daniel Martin Katz. 2022. GPT Takes the Bar Exam. arXiv preprint arXiv:2212.14402 (2022)."},{"key":"e_1_3_2_1_7_1","volume-title":"A college student created an app that can tell whether AI wrote an essay. NPR Technology","author":"Bowman Emma","year":"2023","unstructured":"Emma Bowman. 2023. A college student created an app that can tell whether AI wrote an essay. NPR Technology (2023). Issue January 9, 2023. https:\/\/www.npr.org\/2023\/01\/09\/1147549845\/gptzero-ai-chatgpt-edward-tian-plagiarism"},{"key":"e_1_3_2_1_8_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877--1901."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. https:\/\/doi.org\/10.48550\/ARXIV.2107.03374","DOI":"10.48550\/ARXIV.2107.03374"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","unstructured":"Paul Denny Viraj Kumar and Nasser Giacaman. 2022. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. https:\/\/doi.org\/10.48550\/ARXIV.2210.15157","DOI":"10.48550\/ARXIV.2210.15157"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","unstructured":"Iddo Drori and Nakul Verma. 2021. Solving Linear Algebra by Program Synthesis. https:\/\/doi.org\/10.48550\/ARXIV.2111.08171","DOI":"10.48550\/ARXIV.2111.08171"},{"key":"e_1_3_2_1_12_1","volume-title":"NYC education department blocks ChatGPT on school devices, networks. Chalkbeat New York","author":"Elsen-Rooney Michael","year":"2023","unstructured":"Michael Elsen-Rooney. 2023. NYC education department blocks ChatGPT on school devices, networks. Chalkbeat New York (2023). Issue January 3, 2023. https:\/\/ny.chalkbeat.org\/2023\/1\/3\/23537987\/nyc-schools-ban-chatgpt-writing-artificial-intelligence"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511861.3511863"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3576123.3576134"},{"key":"e_1_3_2_1_15_1","volume-title":"Roderick A. Taylor, and David Chartash.","author":"Gilson Amelia","year":"2022","unstructured":"Amelia Gilson, Conrad W. Safranek, Tao Huang, Vimig Socrates, Lim Sze Chi, Roderick A. Taylor, and David Chartash. 2022. How Well Does ChatGPT Do When Taking the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. In medRxiv."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","unstructured":"Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2020. Measuring Massive Multitask Language Understanding. https:\/\/doi.org\/10.48550\/ARXIV.2009.03300","DOI":"10.48550\/ARXIV.2009.03300"},{"key":"e_1_3_2_1_17_1","volume-title":"Universities Start Revamping How They Teach.","author":"Huang Kalley","year":"2023","unstructured":"Kalley Huang. 2023. Alarmed by A.I. Chatbots, Universities Start Revamping How They Teach. New York Times (2023). Issue January 16, 2023. https:\/\/www.nytimes.com\/2023\/01\/16\/technology\/chatgpt-artificial-intelligence-universities.html"},{"key":"e_1_3_2_1_18_1","volume-title":"ChatGPT and Software Testing Education: Promises & Perils. arXiv preprint arXiv:2302.03287","author":"Jalil Sajed","year":"2023","unstructured":"Sajed Jalil, Suzzana Rafi, Thomas D LaToza, Kevin Moran, and Wing Lam. 2023. ChatGPT and Software Testing Education: Promises & Perils. arXiv preprint arXiv:2302.03287 (2023)."},{"key":"e_1_3_2_1_19_1","volume-title":"Marco D'Ambros, and Romain Robbes.","author":"Karmakar Anjan","year":"2022","unstructured":"Anjan Karmakar, Julian Aron Prenner, Marco D'Ambros, and Romain Robbes. 2022. Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation. ArXiv abs\/2212.02684 (2022)."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.15388\/infedu.2019.15"},{"key":"e_1_3_2_1_21_1","volume-title":"Project-based learning: A review of the literature. Improving schools 19, 3","author":"Kokotsaki Dimitra","year":"2016","unstructured":"Dimitra Kokotsaki, Victoria Menzies, and Andy Wiggins. 2016. Project-based learning: A review of the literature. Improving schools 19, 3 (2016), 267--277."},{"key":"e_1_3_2_1_22_1","volume-title":"Camille Elepano, Marie Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al.","author":"Kung Tiffany H","year":"2022","unstructured":"Tiffany H Kung, Morgan Cheatham, Arielle Medinilla, Czarina Sillos, Lorie De Leon, Camille Elepano, Marie Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2022. Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. medRxiv (2022)."},{"key":"e_1_3_2_1_23_1","volume-title":"Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683","author":"Lai Guokun","year":"2017","unstructured":"Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 (2017)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","unstructured":"Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser R\u00e9mi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d'Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu and Oriol Vinyals. 2022. Competition-level code generation with AlphaCode. Science 378 6624 (2022) 1092--1097. https:\/\/doi.org\/10.1126\/science.abq1158 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/science.abq1158","DOI":"10.1126\/science.abq1158"},{"key":"e_1_3_2_1_25_1","volume-title":"Christoffer Egeberg Hother, and Ole Winther","author":"Li'evin Valentin","year":"2022","unstructured":"Valentin Li'evin, Christoffer Egeberg Hother, and Ole Winther. 2022. Can large language models reason about medical questions? ArXiv abs\/2207.08143 (2022)."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","unstructured":"Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark and Ashwin Kalyan. 2022. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. https:\/\/doi.org\/10.48550\/ARXIV.2209.09513","DOI":"10.48550\/ARXIV.2209.09513"},{"key":"e_1_3_2_1_27_1","volume-title":"Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789","author":"Mihaylov Todor","year":"2018","unstructured":"Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 (2018)."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1098"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528470"},{"key":"e_1_3_2_1_30_1","unstructured":"Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022)."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP46214.2022.9833571"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2210.12353"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5220\/0011996900003470"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM55253.2022.00014"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_2_1_36_1","volume-title":"Attention is all you need. Advances in neural information processing systems 30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017)."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Michel Wermelinger. 2023. Using GitHub Copilot to Solve Simple Programming Problems. (2023).","DOI":"10.1145\/3545945.3569830"},{"key":"e_1_3_2_1_38_1","volume-title":"Proceedings of the Thirteenth AAAI Symposium on Educational Advances in Artificial Intelligence.","author":"Zong Mingyu","year":"2022","unstructured":"Mingyu Zong and Bhaskar Krishnamachari. 2022. Solving math word problems concerning systems of equations with gpt-3. In Proceedings of the Thirteenth AAAI Symposium on Educational Advances in Artificial Intelligence."}],"event":{"name":"ITiCSE 2023: Innovation and Technology in Computer Science Education","location":"Turku Finland","acronym":"ITiCSE 2023","sponsor":["SIGCSE ACM Special Interest Group on Computer Science Education"]},"container-title":["Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3587102.3588792","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,29]],"date-time":"2024-06-29T10:37:44Z","timestamp":1719657464000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587102.3588792"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,29]]},"references-count":38,"alternative-id":["10.1145\/3587102.3588792","10.1145\/3587102"],"URL":"https:\/\/doi.org\/10.1145\/3587102.3588792","relation":{},"subject":[],"published":{"date-parts":[[2023,6,29]]},"assertion":[{"value":"2023-06-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}