Abstract
There is a huge imbalance between languages currently spoken and corresponding resources to study them. Most of the attention naturally goes to the “big” languages—those which have the largest presence in terms of media and number of speakers. Other less represented languages sometimes do not even have a good quality corpus to study them. In this paper, we tackle this imbalance by presenting a new set of evaluation resources for Tatar, a language of the Turkic language family which is mainly spoken in Tatarstan Republic, Russia.
We present three datasets: Similarity and Relatedness datasets that consist of human scored word pairs and can be used to evaluate semantic models; and Analogies dataset that comprises analogy questions and allows to explore semantic, syntactic, and morphological aspects of language modeling. All three datasets build upon existing datasets for the English language and follow the same structure. However, they are not mere translations. They take into account specifics of the Tatar language and expand beyond the original datasets. We evaluate state-of-the-art word embedding models for two languages using our proposed datasets for Tatar and the original datasets for English and report our findings on performance comparison.
The datasets are available at https://github.com/tat-nlp/SART.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Safiullina, ISBN 5-94113-178-X.
- 2.
- 3.
- 4.
- 5.
References
Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Pasca, M., Soroa, A.: A study on similarity and relatedness using distributional and wordnet-based approaches. In: Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 19–27. Association for Computational Linguistics (2009). http://aclweb.org/anthology/N09-1003
Beltagy, I., Erk, K., Mooney, R.: Semantic parsing using distributional semantics and probabilistic logic. In: Proceedings of the ACL 2014 Workshop on Semantic Parsing, pp. 7–11. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/W14-2402, http://aclweb.org/anthology/W14-2402
Berardi, G., Esuli, A., Marcheggiani, D.: Word embeddings go to italy: a comparison of models and training datasets. In: IIR (2015)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
Bruni, E., Boleda, G., Baroni, M., Tran, N.K.: Distributional semantics in technicolor. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 136–145. Association for Computational Linguistics (2012). http://aclweb.org/anthology/P12-1015
Finkelstein, L., et al.: Placing search in context: the concept revisited. In: Proceedings of the 10th International Conference on World Wide Web, pp. 406–414. WWW 2001, ACM, New York, NY, USA (2001). https://doi.org/10.1145/371920.372094
He, X., Yang, M., Gao, J., Nguyen, P., Moore, R.: Indirect-hmm-based hypothesis alignment for combining outputs from machine translation systems. In: Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pp. 98–107. Association for Computational Linguistics (2008). http://aclweb.org/anthology/D08-1011
Hill, F., Reichart, R., Korhonen, A.: Simlex-999: evaluating semantic models with (genuine) similarity estimation. Comput. Linguist. 41(4), 665–695 (2015). https://doi.org/10.1162/COLI_a_00237, http://aclweb.org/anthology/J15-4004
Leviant, I., Reichart, R.: Judgment language matters: multilingual vector space models for judgment language aware lexical semantics. CoRR, abs/1508.00106 (2015)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, vol. 2, pp. 3111–3119. NIPS 2013, Curran Associates Inc. USA (2013). http://dl.acm.org/citation.cfm?id=2999792.2999959
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162
Rubenstein, H., Goodenough, J.B.: Contextual correlates of synonymy. Commun. ACM 8(10), 627–633 (1965)
Svoboda, L., Brychcín, T.: New word analogy corpus for exploring embeddings of Czech words. In: Gelbukh, A. (ed.) CICLing 2016. LNCS, vol. 9623, pp. 103–114. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75477-2_6
Acknowledgments
We thank Mansur Saykhunov, the main author and maintainer of Corpus of Written Tatar (http://www.corpus.tatar/en), for providing us data; and all respondents of the surveys for constructing Similarity/Relatedness datasets. This research was partially supported by the Brazilian National Council for Scientific and Technological Development (CNPq grant # 307425/2017-7).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this paper
Cite this paper
Khusainova, A., Khan, A., Rivera, A.R. (2023). SART - Similarity, Analogies, and Relatedness for Tatar Language: New Benchmark Datasets for Word Embeddings Evaluation. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2019. Lecture Notes in Computer Science, vol 13451. Springer, Cham. https://doi.org/10.1007/978-3-031-24337-0_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-24337-0_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-24336-3
Online ISBN: 978-3-031-24337-0
eBook Packages: Computer ScienceComputer Science (R0)