{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,4,3]],"date-time":"2024-04-03T07:10:53Z","timestamp":1712128253309},"reference-count":70,"publisher":"Cambridge University Press (CUP)","issue":"1","license":[{"start":{"date-parts":[[2019,3,27]],"date-time":"2019-03-27T00:00:00Z","timestamp":1553644800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2020,1]]},"abstract":"Abstract<\/jats:title>Various studies show that statistical machine translation (SMT) systems suffer from fluency errors, especially in the form of grammatical errors and errors related to idiomatic word choices. In this study, we investigate the effectiveness of using monolingual information contained in the machine-translated text to estimate word-level quality of SMT output. We propose a recurrent neural network architecture which uses morpho-syntactic features and word embeddings as word representations within surface and syntactic n<\/jats:italic>-grams. We test the proposed method on two language pairs and for two tasks, namely detecting fluency errors and predicting overall post-editing effort. Our results show that this method is effective for capturing all types of fluency errors at once. Moreover, on the task of predicting post-editing effort, while solely relying on monolingual information, it achieves on-par results with the state-of-the-art quality estimation systems which use both bilingual and monolingual information.<\/jats:p>","DOI":"10.1017\/s1351324919000111","type":"journal-article","created":{"date-parts":[[2019,3,27]],"date-time":"2019-03-27T16:49:28Z","timestamp":1553705368000},"page":"73-94","source":"Crossref","is-referenced-by-count":8,"title":["Estimating word-level quality of statistical machine translation output using monolingual information alone"],"prefix":"10.1017","volume":"26","author":[{"given":"Arda","family":"Tezcan","sequence":"first","affiliation":[]},{"given":"V\u00e9ronique","family":"Hoste","sequence":"additional","affiliation":[]},{"given":"Lieve","family":"Macken","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2019,3,27]]},"reference":[{"key":"S1351324919000111_ref70","unstructured":"Wolk, K. and Marasek, K. (2015). Building subject-aligned comparable corpora and mining it for truly parallel sentence pairs. In CoRR, abs\/1509.08881. Retrieved from http:\/\/arxiv.org\/abs\/1509.08881"},{"key":"S1351324919000111_ref69","author":"White","year":"1995"},{"key":"S1351324919000111_ref67","unstructured":"Van Noord, G. (2006). At last parsing is now operational. In TALN06. Verbum ex machina. Actes de la 13e conference sur le traitement automatique des langues naturelles, pp. 20\u201342."},{"key":"S1351324919000111_ref63","volume-title":"Lecture 6.5\u2013RmsProp: Divide the gradient by a running average of its recent magnitude","author":"Tieleman","year":"2012"},{"key":"S1351324919000111_ref61","doi-asserted-by":"publisher","DOI":"10.1515\/pralin-2017-0015"},{"key":"S1351324919000111_ref60","first-page":"203","article-title":"Detecting grammatical errors in machine translation output using dependency parsing and treebank querying","volume":"4","author":"Tezcan","year":"2016","journal-title":"Baltic Journal of Modern Computing"},{"key":"S1351324919000111_ref59","unstructured":"Stymne, S. and Ahrenberg, L. (2010). Using a grammar checker for evaluation and postprocessing of statistical machine translation. In Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC\u201910). European Language Resources Association (ELRA)."},{"key":"S1351324919000111_ref58","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324919000111_ref57","unstructured":"Specia, L. , Logacheva, V. and Scarton, C. (2016). WMT16 quality estimation shared task training and development data. Retrieved from http:\/\/hdl.handle.net\/11372\/LRT-1646 (LINDAT\/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University)"},{"key":"S1351324919000111_ref56","unstructured":"Specia, L. , Shah, K. , De Souza, J.G.C. , Cohn, T. and Kessler, F.B. (2013). QuEst - A translation quality estimation framework. In Proceedings of the 51th Conference of the Association for Computational Linguistics (ACL), Demo Session."},{"key":"S1351324919000111_ref55","first-page":"28","volume-title":"13th Annual Conference of the European Association for Machine Translation","author":"Specia","year":"2009"},{"key":"S1351324919000111_ref54","first-page":"151","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Socher","year":"2011b"},{"key":"S1351324919000111_ref52","first-page":"223","volume-title":"Proceedings of Association for Machine Translation in the Americas","author":"Snover","year":"2006"},{"key":"S1351324919000111_ref51","first-page":"831","volume-title":"Proceedings of the First Conference onMachine Translation","author":"Scarton","year":"2016"},{"key":"S1351324919000111_ref68","volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC-2006)","author":"Vilar","year":"2006"},{"key":"S1351324919000111_ref48","doi-asserted-by":"publisher","DOI":"10.3115\/1626355.1626369"},{"key":"S1351324919000111_ref47","unstructured":"Oostdijk, N. , Reynaert, M. , Monachesi, P. , Noord, G.V. , Ordelman, R. and Schuurman, I . (2008). From DCoi to SoNaR: A reference corpus for dutch. In Proceedings of the Sixth International Conference on Language Resources and Evaluation."},{"key":"S1351324919000111_ref46","unstructured":"Mikolov, T. , Chen, K. , Corrado, G. and Dean, J. (2013). Efficient estimation of word representations in vector space. In CoRR, abs\/1301.3781."},{"key":"S1351324919000111_ref44","first-page":"806","volume-title":"Proceedings of the First Conference on Machine Translation","author":"Martins","year":"2016"},{"key":"S1351324919000111_ref42","first-page":"1","article-title":"Detecting and correcting syntactic errors in machine translation using feature-based lexicalized tree adjoining grammars","volume":"17","author":"Ma","year":"2012","journal-title":"IJCLCLP"},{"key":"S1351324919000111_ref40","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-2095"},{"key":"S1351324919000111_ref39","unstructured":"Logacheva, V. , Hokamp, C. and Specia, L. (2016a). Marmot: A toolkit for translation quality estimation at the word level. In Proceedings of the 10th Edition of the Language Resources and Evaluation Conference (LREC)."},{"key":"S1351324919000111_ref37","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2016.05.045"},{"key":"S1351324919000111_ref36","unstructured":"Kusner, M. , Sun, Y. , Kolkin, N. and Weinberger, K.Q. (2015). From word embeddings to document distances. In Blei, D. and & Bach, F. (eds), Proceedings of the 32nd International Conference on Machine Learning (ICML-15). JMLR Workshop and Conference Proceedings, pp. 957\u2013966."},{"key":"S1351324919000111_ref35","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3037"},{"key":"S1351324919000111_ref33","doi-asserted-by":"publisher","DOI":"10.1515\/pralin-2017-0014"},{"key":"S1351324919000111_ref32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4763"},{"key":"S1351324919000111_ref28","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3341"},{"key":"S1351324919000111_ref26","unstructured":"Graham, Y. , Baldwin, T. , Moffat, A. and Zobel, J. (2014). Is machine translation getting better over time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 443\u2013451."},{"key":"S1351324919000111_ref24","doi-asserted-by":"publisher","DOI":"10.3115\/1119176.1119189"},{"key":"S1351324919000111_ref23","volume-title":"Translating the post-editor: An investigation of post-editing changes and correlations with professional experience across two romance languages","author":"de Almeida","year":"2013"},{"key":"S1351324919000111_ref22","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2017.01282"},{"key":"S1351324919000111_ref19","unstructured":"Chung, J. , G\u00fcl\u00e7ehre, \u00c7. , Cho, K. and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. In CoRR, abs\/1412.3555. Retrieved from http:\/\/arxiv.org\/abs\/1412.3555"},{"key":"S1351324919000111_ref18","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"S1351324919000111_ref17","first-page":"109","volume":"108","author":"Castilho","year":"2017","journal-title":"Is neural machine translation the new state of the art? The Prague Bulletin of Mathematical Linguistics"},{"key":"S1351324919000111_ref14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-30"},{"key":"S1351324919000111_ref11","volume-title":"Proceedings of the 20th International Conference on Computational Linguistics","author":"Blatz","year":"2004"},{"key":"S1351324919000111_ref10","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4760"},{"key":"S1351324919000111_ref9","doi-asserted-by":"publisher","DOI":"10.3115\/1626431.1626468"},{"key":"S1351324919000111_ref7","unstructured":"Bahdanau, D. , Cho, K. and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. In CoRR, abs\/1409.0473. Retrieved from http:\/\/arxiv.org\/abs\/1409.0473"},{"key":"S1351324919000111_ref6","first-page":"355","volume-title":"Proceedings of the conference on empirical methods in natural language processing","author":"Axelrod","year":"2011"},{"key":"S1351324919000111_ref5","doi-asserted-by":"publisher","DOI":"10.1515\/pralin-2017-0029"},{"key":"S1351324919000111_ref4","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-2067"},{"key":"S1351324919000111_ref3","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2014.6854200"},{"key":"S1351324919000111_ref2","first-page":"764","volume-title":"Proceedings of the First Conference on Machine Translation","author":"Abdelsalam","year":"2016"},{"key":"S1351324919000111_ref1","unstructured":"Abadi, M. , et al. (2016). Tensorflow: Large-Scale machine learning on heterogeneous distributed systems. In CoRR, abs\/1603.04467."},{"key":"S1351324919000111_ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2384"},{"key":"S1351324919000111_ref66","unstructured":"Ueffing, N. and Ney, H. (2005). Application of word-level confidence measures in interactive statistical machine translation. In Proceedings of EAMT 2005 10th Annual Conference of the European Association for Machine Translation, pp. 262\u2013270."},{"key":"S1351324919000111_ref25","unstructured":"Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS\u201910). Society for Artificial Intelligence and Statistics."},{"key":"S1351324919000111_ref29","volume-title":"Evaluating Natural Language Processing Systems: An Analysis and Review","volume":"1083","author":"Jones","year":"1995"},{"key":"S1351324919000111_ref49","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2389"},{"key":"S1351324919000111_ref27","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4775"},{"key":"S1351324919000111_ref71","author":"Xu","year":"2007"},{"key":"S1351324919000111_ref15","first-page":"131","volume-title":"Proceedings of the Frst Conference on Machine Translation","author":"Bojar","year":"2016"},{"key":"S1351324919000111_ref8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1025"},{"key":"S1351324919000111_ref13","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3302"},{"key":"S1351324919000111_ref21","unstructured":"Daems, J. , Macken, L. and Vandepitte, S. (2014). On the origin of errors: A finegrained analysis of mt and pe errors and their relationship. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). European Language Resources Association (ELRA), pp. 62\u201366."},{"key":"S1351324919000111_ref50","first-page":"45","volume-title":"Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks","author":"\u0158eh\u016f\u0159ek","year":"2010"},{"key":"S1351324919000111_ref30","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2378"},{"key":"S1351324919000111_ref20","doi-asserted-by":"publisher","DOI":"10.1007\/s10590-015-9169-0"},{"key":"S1351324919000111_ref34","first-page":"11","volume-title":"AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012)","author":"Koponen","year":"2012"},{"key":"S1351324919000111_ref45","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4764"},{"key":"S1351324919000111_ref53","unstructured":"Socher, R. , Lin, C.C. , Ng, A.Y. and Manning, C.D. (2011a). Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 26th International Conference on Machine Learning (ICML)."},{"key":"S1351324919000111_ref62","first-page":"219","volume-title":"Trends in e-Tools and Resources for Translators and Interpreters","volume":"45","author":"Tezcan","year":"2017b"},{"key":"S1351324919000111_ref65","unstructured":"Turian, J. , Ratinov, L. and Bengio, Y. (2010). Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 384\u2013394."},{"key":"S1351324919000111_ref41","doi-asserted-by":"publisher","DOI":"10.5565\/rev\/tradumatica.77"},{"key":"S1351324919000111_ref12","unstructured":"Bohnet, B. and Nivre, J. (2012). A transition-based system for joint part-of speech tagging and labeled non-projective dependency parsing. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, pp. 1455\u20131465."},{"key":"S1351324919000111_ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-4717"},{"key":"S1351324919000111_ref64","first-page":"198","article-title":"The nature and role of norms in translation","volume":"2","author":"Toury","year":"2000","journal-title":"The Translation Studies Reader"},{"key":"S1351324919000111_ref43","doi-asserted-by":"publisher","DOI":"10.7202\/1006182ar"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324919000111","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,4,21]],"date-time":"2020-04-21T03:58:05Z","timestamp":1587441485000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324919000111\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,27]]},"references-count":70,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,1]]}},"alternative-id":["S1351324919000111"],"URL":"https:\/\/doi.org\/10.1017\/s1351324919000111","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,3,27]]}}}