{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,12,31]],"date-time":"2022-12-31T05:23:46Z","timestamp":1672464226766},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2014,8,1]],"date-time":"2014-08-01T00:00:00Z","timestamp":1406851200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2014,8]]},"abstract":"An enormous potential exists for solving certain classes of computational problems through rich collaboration among crowds of humans supported by computers. Solutions to these problems used to involve human professionals, who are expensive to hire or difficult to find. Despite significant advances, fully automatic systems still have much room for improvement. Recent research has involved recruiting large crowds of skilled humans (\u201ccrowdsourcing\u201d), but crowdsourcing solutions are still restricted by the availability of those skilled human participants. With translation, for example, professional translators incur a high cost and are not always available; machine translation systems have been greatly improved recently but still can only provide passable translation; and crowdsourced translation is limited by the availability of bilingual humans.<\/jats:p>\n This article describes crowdsourced monolingual translation, where monolingual translation is translation performed by monolingual people. Crowdsourced monolingual translation is a collaborative form of translation performed by two crowds of people who speak the source or the target language, respectively, with machine translation as the mediating device.<\/jats:p>\n This article describes a general protocol to handle crowdsourced monolingual translation and analyzes three systems that implemented the protocol. These systems were studied in various settings and were found to supply significant improvement in quality over both machine translation and monolingual editing of machine translation output (\u201cpostediting\u201d).<\/jats:p>","DOI":"10.1145\/2627751","type":"journal-article","created":{"date-parts":[[2014,8,1]],"date-time":"2014-08-01T20:13:24Z","timestamp":1406924004000},"page":"1-35","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Crowdsourced Monolingual Translation"],"prefix":"10.1145","volume":"21","author":[{"given":"Chang","family":"Hu","sequence":"first","affiliation":[{"name":"University of Maryland, College Park, MD"}]},{"given":"Philip","family":"Resnik","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}]},{"given":"Benjamin B.","family":"Bederson","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD"}]}],"member":"320","published-online":{"date-parts":[[2014,8]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Benjamins Translation Library","volume":"35","author":"Allen J.","year":"2003","unstructured":"J. Allen . 2003 . Post-editing. Edited by Harold Somers . Benjamins Translation Library , Vol. 35 . Amsterdam : John Benjamins., 297--317. J. Allen. 2003. Post-editing. Edited by Harold Somers. Benjamins Translation Library, Vol. 35. Amsterdam: John Benjamins., 297--317."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866080"},{"key":"e_1_2_1_4_1","volume-title":"NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk.","author":"Buzek O.","unstructured":"O. Buzek , P. Resnik , and B. B. Bederson . 2010. Error driven paraphrase annotation using mechanical turk . In NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk. O. Buzek, P. Resnik, and B. B. Bederson. 2010. Error driven paraphrase annotation using mechanical turk. In NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk."},{"key":"e_1_2_1_5_1","unstructured":"C. Callison-Burch. 2005. Linear B system description for the 2005 NIST MT evaluation exercise. Available at http:\/\/www.cs.jhu.edu\/ccb\/publications\/linear-b-system-description-for-nist-mt-eval-2005.pdf. C. Callison-Burch. 2005. Linear B system description for the 2005 NIST MT evaluation exercise. Available at http:\/\/www.cs.jhu.edu\/ccb\/publications\/linear-b-system-description-for-nist-mt-eval-2005.pdf."},{"key":"e_1_2_1_6_1","first-page":"10","volume-title":"Cambridge: Cambridge University Press.","author":"Clark H. H.","year":"1981","unstructured":"H. H. Clark and C. R. Marshall . 1981 . Definite reference and mutual knowledge. In A. K. Joshi, B. Webber, & I. Sag (Eds.), Elements of discourse understanding (pp. 10 -- 63 ). Cambridge: Cambridge University Press. H. H. Clark and C. R. Marshall. 1981. Definite reference and mutual knowledge. In A. K. Joshi, B. Webber, & I. Sag (Eds.), Elements of discourse understanding (pp. 10--63). Cambridge: Cambridge University Press."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 2002 Language Resources and Evaluation Conference (LREC\u201902)","author":"Dabbadie M.","year":"2002","unstructured":"M. Dabbadie , A. Hartley , M. King , K. J. Miller , W. M. El Hadi , A. Popescu-Belis , F. Reeder , and M. Vanni . 2002. A hands-on study of the reliability and coherence of evaluation metrics . In Proceedings of the 2002 Language Resources and Evaluation Conference (LREC\u201902) . Available at http:\/\/mt-archive.info\/LREC- 2002 -Dabbadie-2.pdf. M. Dabbadie, A. Hartley, M. King, K. J. Miller, W. M. El Hadi, A. Popescu-Belis, F. Reeder, and M. Vanni. 2002. A hands-on study of the reliability and coherence of evaluation metrics. In Proceedings of the 2002 Language Resources and Evaluation Conference (LREC\u201902). Available at http:\/\/mt-archive.info\/LREC-2002-Dabbadie-2.pdf."},{"key":"e_1_2_1_8_1","unstructured":"D. Danet and S. C. Herring (Eds.). 2007. The Multilingual Internet: Language Culture and Communication Online. Oxford University Press New York. D. Danet and S. C. Herring (Eds.). 2007. The Multilingual Internet: Language Culture and Communication Online. Oxford University Press New York."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the Annual Meeting of the Association of Computational Linguistics. 7--12","author":"Dyer C.","unstructured":"C. Dyer , A. Lopez , J. Ganitkevitch , J. Weese , F. Ture , P. Blunsom , H. Setiawan , V. Eidelman , and P. Resnik . 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models . In Proceedings of the Annual Meeting of the Association of Computational Linguistics. 7--12 . C. Dyer, A. Lopez, J. Ganitkevitch, J. Weese, F. Ture, P. Blunsom, H. Setiawan, V. Eidelman, and P. Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In Proceedings of the Annual Meeting of the Association of Computational Linguistics. 7--12."},{"key":"e_1_2_1_10_1","unstructured":"J. Edwards. 1994. Multilingualism. Routledge. J. Edwards. 1994. Multilingualism. Routledge."},{"key":"e_1_2_1_11_1","first-page":"70","article-title":"Evaluating lemmatic communication","volume":"3","author":"Everitt K.","year":"2010","unstructured":"K. Everitt , C. Lim , O. Etzioni , J. Pool , S. Colowick , and S. Soderland . 2010 . Evaluating lemmatic communication . Journal of Translation and Technical Communication Research 3 (2010), 70 -- 84 . K. Everitt, C. Lim, O. Etzioni, J. Pool, S. Colowick, and S. Soderland. 2010. Evaluating lemmatic communication. Journal of Translation and Technical Communication Research 3 (2010), 70--84.","journal-title":"Journal of Translation and Technical Communication Research"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue.","author":"Gabsdil M.","year":"2003","unstructured":"M. Gabsdil . 2003 . Clarification in spoken dialogue systems . In Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue. Available at https:\/\/www.aaai.org\/Papers\/Symposia\/Spring\/2003\/SS-03-06\/SS03-06-006.pdf. M. Gabsdil. 2003. Clarification in spoken dialogue systems. In Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue. Available at https:\/\/www.aaai.org\/Papers\/Symposia\/Spring\/2003\/SS-03-06\/SS03-06-006.pdf."},{"key":"e_1_2_1_13_1","volume-title":"Traslation and the Internet: Evaluating the quality of free online machine translators. Quaderns: Revista de traducci\u00f3 17","author":"Hampshire S.","year":"2010","unstructured":"S. Hampshire and C. Porta Salvia . 2010. Traslation and the Internet: Evaluating the quality of free online machine translators. Quaderns: Revista de traducci\u00f3 17 ( 2010 ), 197--209. S. Hampshire and C. Porta Salvia. 2010. Traslation and the Internet: Evaluating the quality of free online machine translators. Quaderns: Revista de traducci\u00f3 17 (2010), 197--209."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of Graphics Interface","author":"Hu C.","year":"2010","unstructured":"C. Hu , B. B. Bederson , and P. Resnik . 2010. Translation by iterative collaboration between monolingual users . In Proceedings of Graphics Interface 2010 . Canadian Information Processing Society, Ottawa, Ontario, Canada, 39--46. C. Hu, B. B. Bederson, and P. Resnik. 2010. Translation by iterative collaboration between monolingual users. In Proceedings of Graphics Interface 2010. Canadian Information Processing Society, Ottawa, Ontario, Canada, 39--46."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979111"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208700"},{"key":"e_1_2_1_17_1","unstructured":"C. Hu P. Resnik Y. Kronrod V. Eidelman O. Buzek and B. B. Bederson. 2011. The value of monolingual crowdsourcing in a real-world translation scenario: Simulation using Haitian Creole emergency SMS messages. Association for Computational Linguistics 399--404. Available at http:\/\/www.aclweb.org\/anthology\/W11-2148http:\/\/aclweb.org\/anthology-new\/W \/W11\/W11-2148.pdf. C. Hu P. Resnik Y. Kronrod V. Eidelman O. Buzek and B. B. Bederson. 2011. The value of monolingual crowdsourcing in a real-world translation scenario: Simulation using Haitian Creole emergency SMS messages. Association for Computational Linguistics 399--404. Available at http:\/\/www.aclweb.org\/anthology\/W11-2148http:\/\/aclweb.org\/anthology-new\/W \/W11\/W11-2148.pdf."},{"key":"e_1_2_1_18_1","first-page":"1","article-title":"Current commercial machine translation systems and computer-based translation tools: system types and their uses","volume":"17","author":"Hutchins J.","year":"2005","unstructured":"J. Hutchins . 2005 . Current commercial machine translation systems and computer-based translation tools: system types and their uses . International Journal of Translation 17 , 1 -- 2 (2005), 5--38. J. Hutchins. 2005. Current commercial machine translation systems and computer-based translation tools: system types and their uses. International Journal of Translation 17, 1--2 (2005), 5--38.","journal-title":"International Journal of Translation"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073149"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SAINT.2006.40"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007911416676"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869096"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of Human Language Technologies the 2010 Annual Conference of the North American Chapter of the ACL. 88","author":"Koehn P.","year":"2010","unstructured":"P. Koehn . 2010 . Enabling monolingual translators: Post-editing vs. options . In Proceedings of Human Language Technologies the 2010 Annual Conference of the North American Chapter of the ACL. 88 , 537--545. P. Koehn. 2010. Enabling monolingual translators: Post-editing vs. options. In Proceedings of Human Language Technologies the 2010 Annual Conference of the North American Chapter of the ACL. 88, 537--545."},{"key":"e_1_2_1_24_1","volume-title":"EAMT 2010: Proceedings of the 14th Annual Conference of the European Association for Machine Translation (EAMT\u201910)","author":"Lewis W.","year":"2010","unstructured":"W. Lewis . 2010 . Haitian Creole: How to build and ship an MT engine from scratch in 4 days, 17 hours, & 30 minutes . In EAMT 2010: Proceedings of the 14th Annual Conference of the European Association for Machine Translation (EAMT\u201910) . Available at http:\/\/research.microsoft.com\/pubs\/145627\/eamt-05.pdf. W. Lewis. 2010. Haitian Creole: How to build and ship an MT engine from scratch in 4 days, 17 hours, & 30 minutes. In EAMT 2010: Proceedings of the 14th Annual Conference of the European Association for Machine Translation (EAMT\u201910). Available at http:\/\/research.microsoft.com\/pubs\/145627\/eamt-05.pdf."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1754145"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1380584.1380586"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1502650.1502701"},{"key":"#cr-split#-e_1_2_1_28_1.1","doi-asserted-by":"crossref","unstructured":"D. Morita and T. Ishida. 2011. Collaborative translation protocols. In The Language Grid T. Ishida (Ed.). Springer Berlin 215--230. DOI: http:\/\/dx.doi.org\/10.1007\/978-3-642-21178-2\\_14. 10.1007\/978-3-642-21178-2","DOI":"10.1007\/978-3-642-21178-2_14"},{"key":"#cr-split#-e_1_2_1_28_1.2","doi-asserted-by":"crossref","unstructured":"D. Morita and T. Ishida. 2011. Collaborative translation protocols. In The Language Grid T. Ishida (Ed.). Springer Berlin 215--230. DOI: http:\/\/dx.doi.org\/10.1007\/978-3-642-21178-2\\_14.","DOI":"10.1007\/978-3-642-21178-2_14"},{"key":"e_1_2_1_29_1","volume-title":"Control Systems Engineering (2 ed.)","author":"Nise N. S.","unstructured":"N. S. Nise . 1995. Control Systems Engineering (2 ed.) . Vol. 34 . Addison-Wesley , 11--13. N. S. Nise. 1995. Control Systems Engineering (2 ed.). Vol. 34. Addison-Wesley, 11--13."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/974740.974741"},{"key":"e_1_2_1_31_1","unstructured":"A. J. Quinn and B. B. Bederson. 2009. A Taxonomy of Distributed Human Computation. Human-Computer Interaction Lab Technical Report University of Maryland. Available at http:\/\/hcil.cs.umd.edu\/trs\/2009-23\/2009-23.pdf. A. J. Quinn and B. B. Bederson. 2009. A Taxonomy of Distributed Human Computation. Human-Computer Interaction Lab Technical Report University of Maryland. Available at http:\/\/hcil.cs.umd.edu\/trs\/2009-23\/2009-23.pdf."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb00917.x"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1951.tb01366.x"},{"key":"e_1_2_1_34_1","volume-title":"Proceedinigs of the 7th Conference on Cybernetics, Circular Causal and Feedback Mechanisms in Biological and Social Systems. 248--272","author":"Shannon C. E.","unstructured":"C. E. Shannon , Bell Laboratories , and M. Hill . 1950. The redundancy of English . In Proceedinigs of the 7th Conference on Cybernetics, Circular Causal and Feedback Mechanisms in Biological and Social Systems. 248--272 . C. E. Shannon, Bell Laboratories, and M. Hill. 1950. The redundancy of English. In Proceedinigs of the 7th Conference on Cybernetics, Circular Causal and Feedback Mechanisms in Biological and Social Systems. 248--272."},{"key":"e_1_2_1_35_1","first-page":"223","article-title":"A study of translation edit rate with targeted human annotation","volume":"231","author":"Snover M.","year":"2006","unstructured":"M. Snover , B. Dorr , R. Schwartz , L. Micciulla , and J. Makhoul . 2006 . A study of translation edit rate with targeted human annotation . In Proceedings of the Conference of the Association for Machine Translation in the Americas. 231 , 223 . M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Conference of the Association for Machine Translation in the Americas. 231, 223.","journal-title":"Proceedings of the Conference of the Association for Machine Translation in the Americas."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 12th Machine Translation Summit (MT Summit XII). 128--135","author":"Soderland S.","unstructured":"S. Soderland , C. Lim , B. Q Mausam , O. Etzioni , and J. Pool . 2009. Lemmatic Machine Translation . In Proceedings of the 12th Machine Translation Summit (MT Summit XII). 128--135 . S. Soderland, C. Lim, B. Q Mausam, O. Etzioni, and J. Pool. 2009. Lemmatic Machine Translation. In Proceedings of the 12th Machine Translation Summit (MT Summit XII). 128--135."},{"key":"e_1_2_1_37_1","volume-title":"The Wisdom of Crowds","author":"Surowiecki J.","unstructured":"J. Surowiecki . 2005. The Wisdom of Crowds . Vol. 75 . Anchor . DOI: http:\/\/dx.doi.org\/10.1038\/climate.2009.73 10.1038\/climate.2009.73 J. Surowiecki. 2005. The Wisdom of Crowds. Vol. 75. Anchor. DOI: http:\/\/dx.doi.org\/10.1038\/climate.2009.73"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the Language Resources and Evaluation Conference (LREC\u201906)","author":"Vilar D.","unstructured":"D. Vilar , J. Xu , L. F. D\u2019Haro , and H. Ney . 2006. Error analysis of statistical machine translation output . In Proceedings of the Language Resources and Evaluation Conference (LREC\u201906) . 697--702. D. Vilar, J. Xu, L. F. D\u2019Haro, and H. Ney. 2006. Error analysis of statistical machine translation output. In Proceedings of the Language Resources and Evaluation Conference (LREC\u201906). 697--702."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.3115\/1072133.1072187"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 1220--1229","author":"Zaidan O.","unstructured":"O. Zaidan and C. Callison-Burch . 2011. Crowdsourcing translation: Professional quality from non-professionals . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 1220--1229 . O. Zaidan and C. Callison-Burch. 2011. Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 1220--1229."}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2627751","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,30]],"date-time":"2022-12-30T22:11:46Z","timestamp":1672438306000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2627751"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,8]]},"references-count":42,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2014,8]]}},"alternative-id":["10.1145\/2627751"],"URL":"https:\/\/doi.org\/10.1145\/2627751","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,8]]},"assertion":[{"value":"2013-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2014-08-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}