Abstract
In this digital era language recognition of text plays an important role in the fields like information retrieval systems. Language recognition makes such systems capable of handling multilingual queries for which relevant documents are fetched according to their respective language. It also helps in retrieving information from multilingual sites such as Twitter. Existing work in language identification mainly focuses on large text. This works addresses the problem of language recognition of short text. The work employs two machine learning approaches based on n-gram representation of text - Random Forest and Weighted Ensemble learning. The study performed over 4 popular languages (English, Spanish, French, and German) reveals that Random Forest Algorithm outperforms Naive Bayes, Logistic Classifier and Weighted Ensemble approaches by up to 33.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Botha, G.R., Barnard, E.: Factors that affect the accuracy of text-based language identification. Comput. Speech Lang. 26(5), 307–320 (2012)
Bergsma, S., McNamee, P., Bagdouri, M., Fink, C., Wilson, T.: Language identification for creating language-specific twitter collections. In: Proceedings of the Second Workshop on Language in Social Media, pp. 65–74 (2012)
Krıž, V., Holub, M., Pecina, P.: Feature extraction for native language identification using language modeling. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, pp. 298–306 (2015)
Tejedor, J.E.A.: Comparison of methods for language dependent and language independent query by example spoken term detection. ACM Trans. Inf. Syst. 30(3), 1–34 (2012)
Patel, S., Desai, V.: LIGA and syllabification approach for language identification and back transliteration: a shared task report by DA-IICT. In: Proceedings of the Forum for Information Retrieval Evaluation, pp. 43–47 (2014)
Carter, S., Weerkamp, W., Tsagkias, M.: Microblog language identification: overcoming the limitations of short, unedited and idiomatic text. Lang. Res. Eval. 47(1), 195–215 (2013)
Hinguruduwa, L., Marx, E., Soru, T., Riechert, T.: Assessing language identification over DBpedia. In: 2021 IEEE 15th International Conference on Semantic Computing (ICSC), pp. 296–297 (2021). https://doi.org/10.1109/ICSC50631.2021.00084
Lui, M., Lau, J.H., Baldwin, T.: Automatic detection and language identification of multilingual documents. Trans. Assoc. Comput. Ling. 2, 27–40 (2014)
Jauhiainen, T., Lui, M., Zampieri, M., Baldwin, T., Lindén, K.: Automatic language identification in texts: a survey. J. Artif. Intell. Res. 65, 675–782 (2019)
Gaurav, D., Rodriguez, F.O., Tiwari, S., Jabbar, M.: Review of machine learning approach for drug development process. In: Deep Learning in Biomedical and Health Informatics. CRC Press, pp. 53–77 (2021)
Gaurav, D., Shandilya, S., Tiwari, S., Goyal, A.: A machine learning method for recognizing invasive content in memes. In: Villazón-Terrazas, B., Ortiz-Rodríguez, F., Tiwari, S.M., Shandilya, S.K. (eds.) KGSWC 2020. CCIS, vol. 1232, pp. 195–213. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65384-2_15
Graham, M., Hale, S.A., Gaffney, D.: Where in the world are you? Geolocation and language identification in twitter. Prof. Geogr. 66(4), 568–578 (2014)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Mehta, S., Jain, T., Aggarwal, N. (2021). Multilingual Short Text Analysis of Twitter Using Random Forest Approach. In: Villazón-Terrazas, B., Ortiz-Rodríguez, F., Tiwari, S., Goyal, A., Jabbar, M. (eds) Knowledge Graphs and Semantic Web. KGSWC 2021. Communications in Computer and Information Science, vol 1459. Springer, Cham. https://doi.org/10.1007/978-3-030-91305-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-91305-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91304-5
Online ISBN: 978-3-030-91305-2
eBook Packages: Computer ScienceComputer Science (R0)