Abstract
The use of Deep Neural Network architectures for Language Modelling has recently seen a tremendous increase in interest in the field of NLP with the advent of transfer learning and the shift in focus from rule-based and predictive models (supervised learning) to generative or unsupervised models to solve the long-standing problems in NLP like Information Extraction or Question Answering. While this shift has worked greatly for langauges lacking in inflectional morphology, such as English, challenges still arise when trying to build similar systems for morphologically-rich langauges, since their individual words shift forms in context more often [8]. In this paper we investigate the extent to which these new unsupervised or generative techniques can serve to alleviate the type-token ratio disparity in morphologically rich languages. We apply an off-the-shelf neural language modelling library [20] to the newly introduced [9] task of unsupervised inflection generation in the nominal domain of three morphologically rich languages: Romanian, German, and Finnish. We show that this neural language model architecture can successfully generate the full inflection table of nouns without needing any pre-training on large, wikipedia-sized corpora, as long as the model is shown enough inflection examples. In fact, our experiments show that pre-training hinders the generation performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
airweirdness.com has used it extensively.
References
Aharoni, R., Goldberg, Y.: Morphological inflection generation with hard monotonic attention. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, 30 July–4 August, Long Papers, vol. 1, pp. 2004–2015 (2017). https://doi.org/10.18653/v1/P17-1183
Ahlberg, M., Forsberg, M., Hulden, M.: Paradigm classification in supervised learning of morphology. In: Mihalcea, R., Chai, J.Y., Sarkar, A. (eds.) The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2015, Denver, Colorado, USA, 31 May–5 June, 2015, pp. 1024–1029. The Association for Computational Linguistics (2015). http://aclweb.org/anthology/N/N15/N15-1107.pdf
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
Barbu, A.M.: Conjugarea verbelor româneşti. Dicţionar: 7500 de verbe româneşti grupate pe clase de conjugare, 4th edition, revised, 263 pp. Coresi, Bucharest (2007). (in Romanian)
Barbu, A.M.: Romanian lexical databases: inflected and syllabic forms dictionaries (2008)
Cotterell, R., Kirov, C., Sylak-Glassman, J., Yarowsky, D., Eisner, J., Hulden, M.: The SIGMORPHON 2016 shared task–morphological reinflection. In: Proceedings of the 2016 Meeting of SIGMORPHON. Association for Computational Linguistics, Berlin, Germany, August 2016
Şulea, O.M.: Semi-supervised approach to Romanian noun declension. In: Knowledge-Based and Intelligent Information Engineering Systems: Proceedings of the 20th International Conference KES-2016, York, UK, 5–7 September 2016, pp. 664–671 (2016)
Şulea, O.M., Young, S., Dinu, L.P.: Morphogen: full inflection generation using recurrent neural networks. In: CICLing 2019 (to appear)
Dinu, L.P., Şulea, O.M., Niculae, V.: Sequence tagging for verb conjugation in Romanian. In: RANLP, pp. 215–220 (2013)
Dinu, L.P., Ionescu, E., Niculae, V., Şulea, O.M.: Can alternations be learned? A machine learning approach to verb alternations. In: Recent Advances in Natural Language Processing 2011, pp. 539–544 (2011)
Dinu, L.P., Niculae, V., Şulea, O.M.: Learning how to conjugate the Romanian verb. Rules for regular and partially irregular verbs. In: European Chapter of the Association for Computational Linguistics 2012, pp. 524–528, April 2012
Durrett, G., DeNero, J.: Supervised learning of complete morphological paradigms. In: Vanderwende, L., Daumé III, H., Kirchhoff, K. (eds.) Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, 9–14 June 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pp. 1185–1195. The Association for Computational Linguistics (2013). http://aclweb.org/anthology/N/N13/N13-1138.pdf
Faruqui, M., Tsvetkov, Y., Neubig, G., Dyer, C.: Morphological inflection generation using character sequence to sequence learning. CoRR abs/1512.06110 (2015). http://arxiv.org/abs/1512.06110
Felbo, B., Mislove, A., Søgaard, A., Rahwan, I., Lehmann, S.: Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2017)
Guţu-Romalo, V.: Morfologie Structurală a limbii române. Editura Academiei Republicii Socialiste România (1968). (in Romanian)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: ACL. Association for Computational Linguistics (2018). http://arxiv.org/abs/1801.06146
Koehn, P.: Europarl: a parallel corpus for statistical machine translation (2005)
Şulea, O.M., Nisioi, S., Dinu, L.P.: Using word embeddings to translate named entities. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation, LREC 2016, Portorož, Slovenia, 23–28 May 2016 (2016). http://www.lrec-conf.org/proceedings/lrec2016/summaries/1167.html
Woolf, M.: textgenrnn. http://github.com/minimaxir/textgenrnn
Zhou, C., Neubig, G.: Morphological inflection generation with multi-space variational encoder-decoders. In: Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, Vancouver, BC, Canada, 3–4 August 2017, pp. 58–65 (2017). https://doi.org/10.18653/v1/K17-2005
Acknowledgements
SY would like to thank Noisebridge Hackerspace in San Francisco for use of their computing facilities.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Şulea, OM., Young, S. (2019). Unsupervised Inflection Generation Using Neural Language Modelling. In: Rojas, I., Joya, G., Catala, A. (eds) Advances in Computational Intelligence. IWANN 2019. Lecture Notes in Computer Science(), vol 11506. Springer, Cham. https://doi.org/10.1007/978-3-030-20521-8_55
Download citation
DOI: https://doi.org/10.1007/978-3-030-20521-8_55
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20520-1
Online ISBN: 978-3-030-20521-8
eBook Packages: Computer ScienceComputer Science (R0)