{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:29:21Z","timestamp":1721867361919},"reference-count":53,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2024,7,23]],"date-time":"2024-07-23T00:00:00Z","timestamp":1721692800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"OPERAS"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"This study assesses the usability of machine-translated texts in scholarly communication, using self-paced reading experiments with texts from three scientific disciplines, translated from French into English and vice versa. Thirty-two participants, proficient in the target language, participated. This study uses three machine translation engines (DeepL, ModernMT, OpenNMT), which vary in translation quality. The experiments aim to determine the relationship between translation quality and readers\u2019 reception effort, measured by reading times. The results show that for two disciplines, manual and automatic translation quality measures are significant predictors of reading time. For the most technical discipline, this study could not build models that outperformed the baseline models, which only included participant and text ID as random factors. This study acknowledges the need to include reader-specific features, such as prior knowledge, in future research.<\/jats:p>","DOI":"10.3390\/info15080427","type":"journal-article","created":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T14:46:20Z","timestamp":1721832380000},"page":"427","source":"Crossref","is-referenced-by-count":0,"title":["Machine Translation for Open Scholarly Communication: Examining the Relationship between Translation Quality and Reading Effort"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-7516-7487","authenticated-orcid":false,"given":"Lieve","family":"Macken","sequence":"first","affiliation":[{"name":"LT3<\/sup> Language and Translation Technology Team, Faculty of Arts and Philosophy, Ghent University, Groot-Brittanni\u00eblaan 45, B-9000 Ghent, Belgium"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-3005-9620","authenticated-orcid":false,"given":"Vanessa","family":"De Wilde","sequence":"additional","affiliation":[{"name":"MULTIPLES \u2014Research Centre for Multilingual Practices and Language Learning in Society, Faculty of Arts and Philosophy, Ghent University, Groot-Brittanni\u00eblaan 45, B-9000 Ghent, Belgium"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-8707-6176","authenticated-orcid":false,"given":"Arda","family":"Tezcan","sequence":"additional","affiliation":[{"name":"LT3<\/sup> Language and Translation Technology Team, Faculty of Arts and Philosophy, Ghent University, Groot-Brittanni\u00eblaan 45, B-9000 Ghent, Belgium"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Amano, T., Ram\u00edrez-Casta\u00f1eda, V., Berdejo-Espinola, V., Borokini, I., Chowdhury, S., Golivets, M., Gonz\u00e1lez-Trujillo, J.D., Monta\u00f1o-Centellas, F., Paudel, K., and White, R.L. (2023). The manifold costs of being a non-native English speaker in science. PLoS Biol., 21.","DOI":"10.1371\/journal.pbio.3002184"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1002\/jrsm.1699","article-title":"Language inclusion in ecological systematic reviews and maps: Barriers and perspectives","volume":"15","author":"Hannah","year":"2024","journal-title":"Res. Synth. Methods"},{"key":"ref_3","unstructured":"Bowker, L., Ayeni, P., and Kulczycki, E. (2023). Linguistic Privilege and Marginalization in Scholarly Communication: Understanding the Role of New Language Technologies for Shifting Language Dynamics, Social Sciences and Humanities Research Council of Canada. Technical Report."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Moorkens, J., Castilho, S., Gaspari, F., and Doherty, S. (2018). Machine translation and self-post-editing for academic writing support: Quality explorations. Translation Quality Assessment: From Principles to Practice, Springer.","DOI":"10.1007\/978-3-319-91241-7"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhivotova, A., Berdonosov, V., and Redkolis, E. (2020, January 6\u20137). Improving the quality of scientific articles machine translation while writing original text. Proceedings of the 2020 International Multi-Conference on Industrial Engineering and Modern Technologies (FarEastCon), Vladivostok, Russia.","DOI":"10.1109\/FarEastCon50210.2020.9271442"},{"key":"ref_6","unstructured":"Simonova, V., and Patiniotaki, E. (2022, January 4\u20136). Pre-editing for the Translation of Life-Science Texts from Russian into English via Google Translate. Proceedings of the New Trends in Translation and Technology 2022, Rhodes Island, Greece."},{"key":"ref_7","unstructured":"Roussis, D., Sofianopoulos, S., and Piperidis, S. (2024, January 24\u201327). Enhancing Scientific Discourse: Machine Translation for the Scientific Domain. Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), Sheffield, UK."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"482","DOI":"10.1075\/cogls.00087.wal","article-title":"Investigating how we read translations: A call to action for experimental studies of translation reception","volume":"8","author":"Walker","year":"2021","journal-title":"Cogn. Linguist. Stud."},{"key":"ref_9","first-page":"105","article-title":"Conducting experimental research in audiovisual translation (AVT): A position paper","volume":"30","author":"Orero","year":"2018","journal-title":"JosTrans J. Spec. Transl."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"100149","DOI":"10.1016\/j.amper.2023.100149","article-title":"The proof of the translation process is in the reading of the target text: An eyetracking reception study","volume":"11","author":"Whyatt","year":"2023","journal-title":"Ampersand"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10590-010-9070-9","article-title":"Eye tracking as an MT evaluation technique","volume":"24","author":"Doherty","year":"2010","journal-title":"Mach. Transl."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"271","DOI":"10.35699\/1983-3652.2020.24399","article-title":"Quality assessment of machine translation output: Cognitive evaluation approach in an eye tracking experiment","volume":"13","year":"2020","journal-title":"Texto Livre Ling. Tecnol."},{"key":"ref_13","unstructured":"Colman, T., Fonteyne, M., Daems, J., Dirix, N., and Macken, L. (2022, January 20\u201325). GECO-MT: The Ghent Eye-tracking Corpus of Machine Translation. Proceedings of the Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kasper\u0117, R., Motiej\u016bnien\u0117, J., Patasien\u0117, I., Pata\u0161ius, M., and Horba\u010dauskien\u0117, J. (2023). Is machine translation a dim technology for its users? An eye tracking study. Front. Psychol., 14.","DOI":"10.3389\/fpsyg.2023.1076379"},{"key":"ref_15","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_16","unstructured":"Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv."},{"key":"ref_17","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"Volume 35","author":"Koyejo","year":"2022","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_18","unstructured":"Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (2023, January 6\u20137). Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet. Proceedings of the Eighth Conference on Machine Translation, Singapore."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A. (August, January 30). OpenNMT: Open-Source Toolkit for Neural Machine Translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Vancouver, BC, Canada.","DOI":"10.18653\/v1\/P17-4012"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Germann, U., Fikri Aji, A., and Bogoychev, N. (2018, January 15\u201320). Marian: Fast Neural Machine Translation in C++. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), Melbourne, Australia.","DOI":"10.18653\/v1\/P18-4020"},{"key":"ref_21","unstructured":"Martins, A., Moniz, H., Fumega, S., Martins, B., Batista, F., Coheur, L., Parra, C., Trancoso, I., Turchi, M., and Bisazza, A. (2020). OPUS-MT \u2013 Building open translation services for the World. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, European Association for Machine Translation."},{"key":"ref_22","unstructured":"Team, N., Costa-juss\u00e0, M.R., Cross, J., \u00c7elebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., and Licht, D. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"455","DOI":"10.5565\/rev\/tradumatica.77","article-title":"Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics","volume":"12","author":"Lommel","year":"2014","journal-title":"Tradum\u00e0tica"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1163\/9789004351790_012","article-title":"SCATE taxonomy and corpus of machine translation errors","volume":"Volume 45","author":"Pastor","year":"2017","journal-title":"Trends in E-Tools and Resources for Translators and Interpreters"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6\u201312). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1007\/s10590-009-9059-4","article-title":"The METEOR metric for automatic evaluation of machine translation","volume":"23","author":"Lavie","year":"2009","journal-title":"Mach. Transl."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Hokamp, C., Huck, M., Logacheva, V., and Pecina, P. (2015, January 17\u201318). chrF: Character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal.","DOI":"10.18653\/v1\/W15-3001"},{"key":"ref_28","unstructured":"Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006, January 8\u201312). A study of translation edit rate with targeted human annotation. Proceedings of the 7th Conference of the Association for Machine Translation in the Americas, Cambridge, MA, USA."},{"key":"ref_29","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Sellam, T., Das, D., and Parikh, A.P. (2020). BLEURT: Learning robust metrics for text generation. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.704"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Rei, R., Stewart, C., Farinha, A.C., and Lavie, A. (2020, January 16\u201320). COMET: A Neural Framework for MT Evaluation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.213"},{"key":"ref_32","unstructured":"Koehn, P., Barrault, L., Bojar, O., Bougares, F., Chatterjee, R., Costa-juss\u00e0, M.R., Federmann, C., Fishel, M., Fraser, A., and Freitag, M. (2022, January 7\u20138). CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. Proceedings of the Seventh Conference on Machine Translation (WMT), Abu Dhabi, United Arab Emirates."},{"key":"ref_33","unstructured":"Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (2023, January 7\u20138). Quality Estimation Using Minimum Bayes Risk. Proceedings of the Eighth Conference on Machine Translation, Singapore."},{"key":"ref_34","unstructured":"Barrault, L., Bojar, O., Bougares, F., Chatterjee, R., Costa-Jussa, M.R., Federmann, C., Fishel, M., Fraser, A., Freitag, M., and Graham, Y. (2021, January 10\u201311). To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation. Proceedings of the Sixth Conference on Machine Translation, Online."},{"key":"ref_35","unstructured":"Freitag, M., Rei, R., Mathur, N., Lo, C.k., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., Lavie, A., and Martins, A.F. (2022, January 7\u20138). Results of WMT22 metrics shared task: Stop using BLEU\u2013neural metrics are better and more robust. Proceedings of the Seventh Conference on Machine Translation (WMT), Abu Dhabi, United Arab Emirates."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Freitag, M., Mathur, N., Lo, C.k., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., and Stewart, C. (2023, January 6\u20137). Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. Proceedings of the Eighth Conference on Machine Translation, Singapore.","DOI":"10.18653\/v1\/2023.wmt-1.51"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Lee, S., Lee, J., Moon, H., Park, C., Seo, J., Eo, S., Koo, S., and Lim, H. (2023). A Survey on Evaluation Metrics for Machine Translation. Mathematics, 11.","DOI":"10.3390\/math11041006"},{"key":"ref_38","unstructured":"Orasan, C., Mitkov, R., Pastor, G.C., and Monti, J. (2023, January 7\u20139). Translations and Open Science: Exploring how translation technologies can support multilingualism in scholarly communication. Proceedings of the International Conference on Human-Informed Translation and Interpreting Technology (HiT-IT 2023), Naples, Italy."},{"key":"ref_39","unstructured":"Calzolari, N., Choukri, K., Declerck, T., Do\u011fan, M.U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., and Piperidis, S. (2012, January 21\u201327). Parallel Data, Tools and Interfaces in OPUS. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC\u201912), Istanbul, Turkey."},{"key":"ref_40","unstructured":"Calzolari, N., B\u00e9chet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., and Mariani, J. (2022, January 20\u201325). SciPar: A Collection of Parallel Corpora from Scientific Abstracts. Proceedings of the Thirteenth Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jegerski, J. (2013). Self-paced reading. Research Methods in Second Language Psycholinguistics, Routledge.","DOI":"10.4324\/9780203123430"},{"key":"ref_42","unstructured":"Mary, N., Brenner, J., Koponen, M., Latomaa, S., Mikhailov, M., Schierl, F., Ranasinghe, T., Vanmassenhove, E., Alvarez Vidal, S., and Aranberri, N. (2023, January 12\u201315). MATEO: MAchine Translation Evaluation Online. Proceedings of the 24th Annual Conference of the European Association for Machine Translation, Tampere, Finland."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"100505","DOI":"10.1016\/j.asw.2020.100505","article-title":"Investigating minimum text lengths for lexical diversity indices","volume":"47","author":"Zenker","year":"2021","journal-title":"Assess. Writ."},{"key":"ref_44","first-page":"103396","article-title":"Vocabulary of reading materials in English and French L2 textbooks: A cross-lingual corpus study","volume":"124","author":"Macken","year":"2024","journal-title":"System"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1176","DOI":"10.1080\/17470218.2013.850521","article-title":"SUBTLEX-UK: A new and improved word frequency database for British English","volume":"67","author":"Mandera","year":"2014","journal-title":"Q. J. Exp. Psychol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"516","DOI":"10.3758\/BF03195598","article-title":"Lexique 2: A new French lexical database","volume":"36","author":"New","year":"2004","journal-title":"Behav. Res. Methods Instrum. Comput."},{"key":"ref_47","unstructured":"R Core Team (2024). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v067.i01","article-title":"Fitting Linear Mixed-Effects Models Using lme4","volume":"67","author":"Bates","year":"2015","journal-title":"J. Stat. Softw."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v082.i13","article-title":"lmerTest Package: Tests in Linear Mixed Effects Models","volume":"82","author":"Kuznetsova","year":"2017","journal-title":"J. Stat. Softw."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"3139","DOI":"10.21105\/joss.03139","article-title":"performance: An R Package for Assessment, Comparison and Testing of Statistical Models","volume":"6","author":"Patil","year":"2021","journal-title":"J. Open Source Softw."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1111\/j.1467-9892.1993.tb00144.x","article-title":"A corrected Akaike information criterion for vector autoregressive model selection","volume":"14","author":"Hurvich","year":"1993","journal-title":"J. Time Ser. Anal."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.jml.2019.104047","article-title":"How many words do we read per minute? A review and meta-analysis of reading rate","volume":"109","author":"Brysbaert","year":"2019","journal-title":"J. Mem. Lang."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"1209","DOI":"10.1080\/09500693.2022.2072013","article-title":"Using an eye tracker to examine the effect of prior knowledge on reading processes while reading a printed scientific text with multiple representations","volume":"44","author":"Jian","year":"2022","journal-title":"Int. J. Sci. Educ."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/8\/427\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T16:12:59Z","timestamp":1721837579000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/8\/427"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,23]]},"references-count":53,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["info15080427"],"URL":"https:\/\/doi.org\/10.3390\/info15080427","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,23]]}}}