Abstract
In this paper, we present Russian language datasets in the digital humanities domain for the evaluation of word embedding techniques or similar language modeling and feature learning algorithms. The datasets are split into two task types, word intrusion and word analogy, and contain 31362 task units in total. The characteristics of the tasks and datasets are that they build upon small, domain-specific corpora, and that the datasets contain a high number of named entities. The datasets were created manually for two fantasy novel book series (“A Song of Ice and Fire” and “Harry Potter”). We provide baseline evaluations with popular word embedding models trained on the book corpora for the given tasks, both for the Russian and English language versions of the datasets. Finally, we compare and analyze the results and discuss specifics of Russian language with regards to the problem setting.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
Using this toolkit: tech.yandex.ru/mystem.
- 6.
size=300, -negative=0, sg=1, hs=1, iter=15.
- 7.
size=300, -negative=0, sg=1, hs=1, iter=15, -window=12.
- 8.
References
Harris, Z.: Distributional structure. Word 10(2–3), 146–162 (1954)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Herbelot, A.: Mr. Darcy and Mr. Toad, gentlemen: distributional names and their kinds. In: Proceedings of the 11th International Conference on Computational Semantics, pp. 151–161 (2015)
Boleda, G., Padó, S., Gupta, A.: Instances and concepts in distributional space. In: 2017 EACL, Valencia, Spain, 3–7 April 2017, Volume 2: Short Papers, pp. 79–85 (2017)
Wohlgenannt, G., Chernyak, E., Ilvovsky, D., Barinova, A., Mouromtsev, D.: Relation extraction datasets in the digital humanities domain and their evaluation with word embeddings. In: CICLING 2018. Volume upcoming of Lecture Notes in Computer Science., Hanoi, Vietnam, Springer (2018). upcoming
Řehůřek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, Valletta, Malta, ELRA, pp. 45–50 (2010)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
Ghannay, S., Favre, B., Estève, Y., Camelin, N.: Word embedding evaluation and combination. In: Proceedings of the LREC 2016, Paris, France, ELRA (2016)
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)
Kutuzov, A., Kuzmenko, E.: WebVectors: a toolkit for building web interfaces for vector semantic models. In: Ignatov, D.I., et al. (eds.) AIST 2016. CCIS, vol. 661, pp. 155–161. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52920-2_15
Panchenko, A., Loukachevitch, N.V., Ustalov, D., Paperno, D., Meyer, C., Konstantinova, N.: RUSSE: the first workshop on Russian semantic similarity. CoRR abs/1803.05820 (2015)
Finkelstein, L., et al.: Placing search in context: the concept revisited. In: Proceedings of the 10th International Conference of the World Wide Web, pp. 406–414. ACM (2001)
Bruni, E., Tran, N.K., Baroni, M.: Multimodal distributional semantics. J. Artif. Int. Res. 49(1), 1–47 (2014)
Gladkova, A., Drozd, A., Matsuoka, S.: Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In: SRW@HLT-NAACL, pp. 8–15. ACL (2016)
Sahlgren, M., Lenci, A.: The effects of data size and frequency range on distributional semantic models. In: EMNLP, pp. 975–980. ACL (2016)
Boleda, G., Padó, S., Pham, N.T., Baroni, M.: Living a discrete life in a continuous world: reference in cross-modal entity tracking. In: IWCS 2017–12th International Conference on Computational Semantics – Short papers (2017)
Elson, D.K., Dames, N., McKeown, K.R.: Extracting social networks from literary fiction. In: Proceedings of the 48th Annual Meeting of the ACL (ACL 2010), pp. 138–147. Association for Computational Linguistics, Stroudsburg, PA, USA (2010)
Jayannavar, P., Agarwal, A., Ju, M., Rambow, O.: Validating literary theories using automatic social network extraction. In: CLFL@ NAACL-HLT, pp. 32–41 (2015)
Vala, H., Jurgens, D., Piper, A., Ruths, D.: Mr. Bennet, his coachman, and the archbishop walk into a bar but only one of them gets recognized: on the difficulty of detecting characters in literary texts. In: Màrquez, L.E.A., ed.: EMNLP, pp. 769–774. ACL (2015)
Acknowledgments
This work was supported by the Government of the Russian Federation (Grant 074-U01) through the ITMO Fellowship and Professorship Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Wohlgenannt, G., Babushkin, A., Romashov, D., Ukrainets, I., Maskaykin, A., Shutov, I. (2023). Russian Language Datasets in the Digital Humanities Domain and Their Evaluation with Word Embeddings. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2019. Lecture Notes in Computer Science, vol 13451. Springer, Cham. https://doi.org/10.1007/978-3-031-24337-0_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-24337-0_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-24336-3
Online ISBN: 978-3-031-24337-0
eBook Packages: Computer ScienceComputer Science (R0)