Abstract
In role-playing games a Game Master (GM) is the player in charge of the game, who must design the challenges the players face and narrate the outcomes of their actions. In this work we discuss some challenges to model GMs from an Interactive Storytelling and Natural Language Processing perspective. Following those challenges we propose three test categories to evaluate such dialogue systems, and we use them to test ChatGPT, Bard and OpenAssistant as out-of-the-box GMs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
For example “Call of Cthulhu: The Official Video Game”, an adaptation of the RPG.
- 2.
An interesting example of this is “The Stanley Parable”, a novel videogame that makes the players think about free will and the impact of their actions.
- 3.
- 4.
- 5.
- 6.
- 7.
“ChatGPT Mar 14 Version. Free Research Preview”.
- 8.
Bard Experimental. Accessed on the 17th of July, 2023.
- 9.
“Model: OpenAssistant/oasst-sft-6-llama-30b”. Accessed on the 14th of July, 2023.
- 10.
- 11.
This is evidenced by the massive amount of adventures published for RPGs with this theme, such as Dungeons & Dragons or Pathfinder.
References
Acharya, D., Mateas, M., Wardrip-Fruin, N.: Story improvisation in tabletop roleplaying games: towards a computational assistant for game masters. In: 2021 IEEE Conference on Games (CoG), pp. 01–08 (2021). https://doi.org/10.1109/CoG52621.2021.9619006
Arenas, D.L., Viduani, A., Araujo, R.B.: Therapeutic use of role-playing game (RPG) in mental health: a scoping review. Simul. Gaming 53(3), 285–311 (2022)
Behnke, H., Fomicheva, M., Specia, L.: Bias mitigation in machine translation quality estimation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1475–1487. Association for Computational Linguistics, Dublin (2022). https://doi.org/10.18653/v1/2022.acl-long.104
Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2021, pp. 610–623. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3442188.3445922
Bender, E.M., Koller, A.: Climbing towards NLU: on meaning, form, and understanding in the age of data. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185–5198. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.463
Benotti, L.: Implicature as an interactive process. L’implicature comme un Processus Interactif. Ph.D. thesis, Henri Poincaré University, Nancy, France (2010)
Botelho, L.M.: A guided journey through non-interactive automatic story generation (2021)
Callison-Burch, C., Tomar, G.S., Martin, L., Ippolito, D., Bailis, S., Reitter, D.: Dungeons and dragons as a dialog challenge for artificial intelligence. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9379–9393. Association for Computational Linguistics, Abu Dhabi (2022)
Chen, H., Liu, X., Yin, D., Tang, J.: A survey on dialogue systems: recent advances and new frontiers. SIGKDD Explor. Newsl. 19(2), 25–35 (2017). https://doi.org/10.1145/3166054.3166058
Colton, S., Wiggins, G.A.: Computational creativity: the final frontier? In: European Conference on Artificial Intelligence (2012)
Davis, E., Marcus, G.: Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58(9), 92–103 (2015)
Durall, J., Perrin, S.: Basic Roleplaying: Universal Game Engine. Chaosium Inc. (2023)
Gatsakou, C., Bardis, N., Drigas, A.: Role playing vs RPGs as teaching strategies in educational procedure. Technium Soc. Sci. J. 26, 186 (2021)
Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12), 1–38 (2023). https://doi.org/10.1145/3571730
Jordanous, A.: A standardised procedure for evaluating creative systems: computational creativity evaluation based on what it is to be creative. Cogn. Comput. 4(3), 246–279 (2012). https://doi.org/10.1007/s12559-012-9156-1
Köpf, A., et al.: OpenAssistant conversations - democratizing large language model alignment (2023)
Louis, A., Sutton, C.: Deep dungeons and dragons: learning character-action interactions from role-playing game transcripts. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 708–713. Association for Computational Linguistics, New Orleans (2018). https://doi.org/10.18653/v1/N18-2111
Martin, L.J., Harrison, B., Riedl, M.O.: Improvisational computational storytelling in open worlds. In: Nack, F., Gordon, A.S. (eds.) ICIDS 2016. LNCS, vol. 10045, pp. 73–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48279-8_7
Martin, L.J., Sood, S., Riedl, M.O.: Dungeons and DQNs: toward reinforcement learning agents that play tabletop roleplaying games. In: INT/WICED@AIIDE (2018)
Oñate, A., Méndez, G., Gervás, P.: Emolift: elevator conversations based on emotions. In: International Conference on Innovative Computing and Cloud Computing (2019)
Peinado, F., Gervás, P.: Transferring game mastering laws to interactive digital storytelling. In: Göbel, S., et al. (eds.) TIDSE 2004. LNCS, vol. 3105, pp. 48–54. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27797-2_7
Peinado, F., Gervás, P.: Automatic direction of interactive storytelling: formalizing the game master paradigm. In: Cavazza, M., Donikian, S. (eds.) ICVS 2007. LNCS, vol. 4871, pp. 196–201. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-77039-8_18
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D.: Is ChatGPT a general-purpose natural language processing task solver? (2023). https://doi.org/10.48550/ARXIV.2302.06476
Rameshkumar, R., Bailey, P.: Storytelling with dialogue: a critical Role Dungeons and Dragons Dataset. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5121–5134. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.459
Riedl, M.O., Bulitko, V.: Interactive narrative: an intelligent systems approach. AI Mag. 34(1), 67 (2012). https://doi.org/10.1609/aimag.v34i1.2449
Sap, M., Shwartz, V., Bosselut, A., Choi, Y., Roth, D.: Commonsense reasoning for natural language processing. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 27–33. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-tutorials.7
Tychsen, A., Hitchens, M.: The many faces of role-playing games. Int. J. Role-Play. 1(1), 3–21 (2009)
Zhou, P., et al.: An AI dungeon master’s guide: learning to converse and guide with intents and theory-of-mind in dungeons and dragons (2022). https://doi.org/10.48550/ARXIV.2212.10060
Zhu, A., Aggarwal, K., Feng, A., Martin, L., Callison-Burch, C.: FIREBALL: a dataset of dungeons and dragons actual-play with structured game state information. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4171–4193. Association for Computational Linguistics, Toronto (2023)
Zhu, A., Martin, L.J., Head, A., Callison-Burch, C.: CALYPSO: LLMs as dungeon masters’ assistants. UMBC Faculty Collection (2023)
Acknowledgements
This paper has been partially funded by ANII (Uruguayan Innovation and Research National Agency), Grant No. \(POS\_NAC\_2022\_1\_173659\) and by the project CANTOR: Automated Composition of Personal Narratives as an aid for Occupational Therapy based on Reminescence, Grant No. \(PID2019-108927RB-I00\) (Spanish Ministry of Science and Innovation).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Góngora, S., Chiruzzo, L., Méndez, G., Gervás, P. (2024). Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-Playing Games. In: Dondio, P., et al. Games and Learning Alliance. GALA 2023. Lecture Notes in Computer Science, vol 14475. Springer, Cham. https://doi.org/10.1007/978-3-031-49065-1_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-49065-1_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49064-4
Online ISBN: 978-3-031-49065-1
eBook Packages: Computer ScienceComputer Science (R0)