Abstract
In this paper, we report an experiment to investigate the performance of ChatGPT in the task of classifying domain terms according to the categories of upper-level ontologies. The experiment consisted of (1) starting a conversation in ChatGPT with a contextual prompt listing the categories of an upper-level ontology along with their definitions, (2) submitting a follow-up prompt with a list of terms from a domain along with informal definitions, (3) asking ChatGPT to classify the terms according to the categories of the chosen upper-level ontology and explain its decision, and (4) comparing the answers of ChatGPT with the classification proposed by experts in the chosen ontology. Given the results, we evaluated the success rate of ChatGPT in performing the task and analyzed the cases of misclassification to understand the possible reasons underlying them. Based on that, we made some considerations about the extent to which we can employ ChatGPT as an assistant tool for the task of classifying domain terms into upper-level ontologies. For our experiment, we selected a set of 19 terms from the manufacturing domain that were gathered by the Industrial Ontologies Foundry (IOF) and for which there are informal textual definitions reflecting a community view of them. Also, as a baseline for comparison, we resorted to publicly available classifications of such terms according to DOLCE and BFO upper-level ontologies, which resulted from a thorough ontological analysis of those terms and informal definitions by experts in each of the ontologies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
“Act as an expert in Formal Ontology and Upper Ontologies. Here are the definitions of categories of the <DOLCE|BFO> upper ontology. Make a sketch of the taxonomy of these categories. Present the sketch using bullet points.”.
- 3.
We used the definitions/descriptions of BFO categories as provided in https://standards.iso.org/iso-iec/21838/-2/ed-1/en/ and definitions of DOLCE categories as provided in https://github.com/gruninger/colore/blob/master/ontologies/dolce/DOLCE-Terms.docx.
- 4.
“Now, consider the following list of terms with their respective definitions. Classify each of these terms according to the <DOLCE|BFO> taxonomy you sketched and provide a short explanation of why you classified each term in the given <DOLCE|BFO> category. Present the classification in a bullet list.”.
References
Abdullah, M., Madain, A., Jararweh, Y.: ChatGPT: fundamentals, applications and social impacts. In: 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS), pp. 1–8. IEEE (2022)
Abel, M., Perrin, M., Carbonera, J.L.: Ontological analysis for information integration in geomodeling. Earth Sci. Inf. 8, 21–36 (2015)
Arp, R., Smith, B., Spear, A.D.: Building Ontologies with Basic Formal Ontology. MIT Press, Cambridge (2015)
Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)
Garcia, L.F., Rodrigues, F.H., Lopes, A., Kuchle, R.d.S.A., Perrin, M., Abel, M.: What geologists talk about: towards a frequency-based ontological analysis of petroleum domain terms. In: ONTOBRAS, pp. 190–203 (2020)
Guarino, N.: Formal ontology and information systems. In: International Conference on Formal Ontology and Information Systems (FOIS 1998), pp. 3–15 (1998)
Guarino, N., Sanfilippo, E.: Characterizing IOF terms with the DOLCE and UFO ontologies. In: 10th International Workshop on Formal Ontologies Meet Industry (2019)
Guarino, N., Welty, C.: Evaluating ontological decisions with ontoclean. Commun. ACM 45(2), 61–65 (2002). https://doi.org/10.1145/503124.503150
Kulvatunyou, B.S., Wallace, E., Kiritsis, D., Smith, B., Will, C.: The industrial ontologies foundry proof-of-concept project. In: Moon, I., Lee, G.M., Park, J., Kiritsis, D., von Cieminski, G. (eds.) APMS 2018. IAICT, vol. 536, pp. 402–409. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99707-0_50
Masolo, C., Borgo, S., Gangemi, A., Guarino, N., Oltramari, A.: WonderWeb deliverable D18: ontology library. Laboratory for Applied Ontology, ISTC-CNR (2003)
OpenAI: GPT-4 technical report (2023)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Ray, P.P.: ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. IoT Cyber-Phys. Syst. (2023)
Smith, B., et al.: A first-order logic formalization of the industrial ontologies foundry signature using basic formal ontology. In: JOWO (2019)
Wei, J., et al.: Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)
Acknowledgements
This study was financed by Petwin Project (PeTWIN.org), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001, CNPq, FINEP, and LIBRA Consortium.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rodrigues, F.H., Lopes, A.G., dos Santos, N.O., Garcia, L.F., Carbonera, J.L., Abel, M. (2023). On the Use of ChatGPT for Classifying Domain Terms According to Upper Ontologies. In: Sales, T.P., Araújo, J., Borbinha, J., Guizzardi, G. (eds) Advances in Conceptual Modeling. ER 2023. Lecture Notes in Computer Science, vol 14319. Springer, Cham. https://doi.org/10.1007/978-3-031-47112-4_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-47112-4_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47111-7
Online ISBN: 978-3-031-47112-4
eBook Packages: Computer ScienceComputer Science (R0)