{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,5]],"date-time":"2024-08-05T18:42:02Z","timestamp":1722883322553},"reference-count":33,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,12,6]],"date-time":"2018-12-06T00:00:00Z","timestamp":1544054400000},"content-version":"vor","delay-in-days":365,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1162114"],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2017,12,6]]},"abstract":"Social computing researchers are beginning to apply machine learning tools to classify and analyze social media data. Our interest in understanding politeness in an online community focused our attention on tools that would help automate politeness analysis. This paper highlights one popular classification tool designed to score the politeness of text. Our application of this tool to Wikipedia data yielded some unexpected results. Those unexpected results led us to question how the tool worked and its effectiveness relative to human judgment and classification. We designed a user study to revalidate the tool with crowdworkers labeling samples of content from Wikipedia talk pages, imitating the original efforts to validate the tool. This revalidation points to challenges for automatic labeling. Based on our results, this paper reconsiders politeness in online communities as well as broader trends in the use of machine classifiers in social computing research.<\/jats:p>","DOI":"10.1145\/3134687","type":"journal-article","created":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T21:23:15Z","timestamp":1512595395000},"page":"1-14","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Evaluating a Computational Approach to Labeling Politeness"],"prefix":"10.1145","volume":"1","author":[{"given":"Erin R.","family":"Hoffman","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"David W.","family":"McDonald","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Mark","family":"Zachry","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2017,12,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Eighth International AAAI Conference on Weblogs and Social Media.","author":"Althoff Tim","year":"2014","unstructured":"Tim Althoff , Cristian Danescu-Niculescu-Mizil , and Dan Jurafsky . 2014 . How to Ask for a Favor: A Case Study on the Success of Altruistic Requests . In Eighth International AAAI Conference on Weblogs and Social Media. Tim Althoff, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2014. How to Ask for a Favor: A Case Study on the Success of Altruistic Requests. In Eighth International AAAI Conference on Weblogs and Social Media."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557403"},{"key":"e_1_2_1_3_1","volume-title":"Inter Annotator Agreement. GitHub repository. Retrieved","author":"Becker Lee","year":"2017","unstructured":"Lee Becker . 2013. Inter Annotator Agreement. GitHub repository. Retrieved July 2017 from: https:\/\/github.com\/leebecker\/InterAnnotatorAgreement Lee Becker. 2013. Inter Annotator Agreement. GitHub repository. Retrieved July 2017 from: https:\/\/github.com\/leebecker\/InterAnnotatorAgreement"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the Workshop on Languages in Social Media, Association for Computational Linguistics, 48--57","author":"Bender Emily M.","year":"2011","unstructured":"Emily M. Bender , Jonathan T. Morgan , Meghan Oxley , Mark Zachry , Brian Hutchinson , Alex Marin , Bin Zhang , and Mari Ostendorf . 2011 . Annotating social acts: Authority claims and alignment moves in Wikipedia talk pages . In Proceedings of the Workshop on Languages in Social Media, Association for Computational Linguistics, 48--57 . Emily M. Bender, Jonathan T. Morgan, Meghan Oxley, Mark Zachry, Brian Hutchinson, Alex Marin, Bin Zhang, and Mari Ostendorf. 2011. Annotating social acts: Authority claims and alignment moves in Wikipedia talk pages. In Proceedings of the Workshop on Languages in Social Media, Association for Computational Linguistics, 48--57."},{"key":"e_1_2_1_5_1","volume-title":"Levinson","author":"Brown Penelope","year":"1987","unstructured":"Penelope Brown and Stephen C . Levinson . 1987 . Politeness : Some universals in language usage. Cambridge University Press . Penelope Brown and Stephen C. Levinson. 1987. Politeness: Some universals in language usage. Cambridge University Press."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1460563.1460609"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2014.05.029"},{"issue":"3295","key":"e_1_2_1_8_1","first-page":"13","article-title":"Discovering Response-Eliciting Factors in Social Question Answering: A Reddit Inspired Study","volume":"24196","author":"Dahiya Yogesh","year":"2016","unstructured":"Yogesh Dahiya and Partha Talukdar . 2016 . Discovering Response-Eliciting Factors in Social Question Answering: A Reddit Inspired Study . Director 24196 ( 3295 ), 13 -- 61 . Yogesh Dahiya and Partha Talukdar. 2016. Discovering Response-Eliciting Factors in Social Question Answering: A Reddit Inspired Study. Director 24196(3295), 13--61.","journal-title":"Director"},{"key":"e_1_2_1_9_1","volume-title":"51st Annual Meeting of the Association for Computational Linguistics, 250--259","author":"Danescu-Niculescu-Mizil Christian","year":"2013","unstructured":"Christian Danescu-Niculescu-Mizil , Moritz Sudhof , Dan Jurafsky , Jure Leskovec , and Christopher Potts . 2013 . A Computational Approach to Politeness with Application to Social Factors . In 51st Annual Meeting of the Association for Computational Linguistics, 250--259 . Christian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A Computational Approach to Politeness with Application to Social Factors. In 51st Annual Meeting of the Association for Computational Linguistics, 250--259."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958844"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Giuseppe Destefanis Marco Ortu Steve Counsell Michele Marchesi and Roberto Tonelli. 2015. Software development: do good manners matter? (No. e1892). PeerJ PrePrints. Giuseppe Destefanis Marco Ortu Steve Counsell Michele Marchesi and Roberto Tonelli. 2015. Software development: do good manners matter? (No. e1892). PeerJ PrePrints.","DOI":"10.7287\/peerj.preprints.1515v1"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1214\/088342304000000396"},{"issue":"1","key":"e_1_2_1_13_1","first-page":"10","article-title":"ReCal OIR: Ordinal, interval, and ratio intercoder reliability as a web service","volume":"8","author":"Freelon Deen","year":"2013","unstructured":"Deen Freelon . 2013 . ReCal OIR: Ordinal, interval, and ratio intercoder reliability as a web service . International Journal of Internet Science , 8 ( 1 ), 10 -- 16 . Deen Freelon. 2013. ReCal OIR: Ordinal, interval, and ratio intercoder reliability as a web service. International Journal of Internet Science, 8(1), 10--16.","journal-title":"International Journal of Internet Science"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/10572252.2011.551507"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481336"},{"key":"e_1_2_1_16_1","volume-title":"PhD. diss., School of Interactive Computing","author":"Grevet Catherine","unstructured":"Catherine Grevet . 2016. Being Nice on the Internet: Designing for the Coexistence of Diverse Opinions Oneline , PhD. diss., School of Interactive Computing , Georgia Institute for Technology , Atlanta, GA . Catherine Grevet. 2016. Being Nice on the Internet: Designing for the Coexistence of Diverse Opinions Oneline, PhD. diss., School of Interactive Computing, Georgia Institute for Technology, Atlanta, GA."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2017.267"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038558.2038575"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883023"},{"key":"e_1_2_1_20_1","volume-title":"Content analysis: An introduction to its methodology","author":"Krippendorff Klaus","unstructured":"Klaus Krippendorff . 2004. Content analysis: An introduction to its methodology ( 2 nd ed.). Sage Publications . Klaus Krippendorff. 2004. Content analysis: An introduction to its methodology (2nd ed.). Sage Publications.","edition":"2"},{"key":"e_1_2_1_21_1","volume-title":"Politeness in Hindi Online Texts: Pragmatic and Computational Aspects. PhD. diss","author":"Kumar Ritesh","unstructured":"Ritesh Kumar . 2014. Politeness in Hindi Online Texts: Pragmatic and Computational Aspects. PhD. diss ., Centre for Linguistics, Jawaharlal Nehru University, New Delhi , India . Ritesh Kumar. 2014. Politeness in Hindi Online Texts: Pragmatic and Computational Aspects. PhD. diss., Centre for Linguistics, Jawaharlal Nehru University, New Delhi, India."},{"key":"e_1_2_1_22_1","volume-title":"Automated language-based feedback for teamwork behaviors. PhD. diss","author":"Leshed Gilly","unstructured":"Gilly Leshed . 2009. Automated language-based feedback for teamwork behaviors. PhD. diss ., Cornell University , Ithaca, NY . Gilly Leshed. 2009. Automated language-based feedback for teamwork behaviors. PhD. diss., Cornell University, Ithaca, NY."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1159"},{"key":"e_1_2_1_24_1","volume-title":"Mining software repositories: measuring effectiveness and affectiveness in software systems. PhD diss., Electronic and Computer Engineering","author":"Ortu Marco","unstructured":"Marco Ortu . 2015. Mining software repositories: measuring effectiveness and affectiveness in software systems. PhD diss., Electronic and Computer Engineering , University of Cagliari , Cagliari, Italy . Marco Ortu. 2015. Mining software repositories: measuring effectiveness and affectiveness in software systems. PhD diss., Electronic and Computer Engineering, University of Cagliari, Cagliari, Italy."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/2820518.2820555"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18612-2_11"},{"key":"e_1_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Marco Ortu Guiseppe Destefanis Steve Counsell Stephen Swift Michele Marchesi and Roberto Tonelli. 2016. How diverse is your team? Investigating gender and nationality diversity in GitHub teams (No. e2285v1). PeerJ Preprints. Marco Ortu Guiseppe Destefanis Steve Counsell Stephen Swift Michele Marchesi and Roberto Tonelli. 2016. How diverse is your team? Investigating gender and nationality diversity in GitHub teams (No. e2285v1). PeerJ Preprints.","DOI":"10.7287\/peerj.preprints.2285v1"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901739.2903505"},{"key":"e_1_2_1_29_1","article-title":"An Empirical Analysis of Formality in Online Communication","author":"Pavlick Ellie","year":"2016","unstructured":"Ellie Pavlick and Joel Tetreault . 2016 . An Empirical Analysis of Formality in Online Communication . Transactions of the Association for Computational Linguistics, 4(61--74). Ellie Pavlick and Joel Tetreault. 2016. An Empirical Analysis of Formality in Online Communication. Transactions of the Association for Computational Linguistics, 4(61--74).","journal-title":"Transactions of the Association for Computational Linguistics, 4(61--74)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958953"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531731"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635882"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470659"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134687","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134687","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,31]],"date-time":"2022-12-31T20:42:37Z","timestamp":1672519357000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134687"}},"subtitle":["Challenges for the Application of Machine Classification to Social Computing Data"],"short-title":[],"issued":{"date-parts":[[2017,12,6]]},"references-count":33,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2017,12,6]]}},"alternative-id":["10.1145\/3134687"],"URL":"https:\/\/doi.org\/10.1145\/3134687","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,12,6]]},"assertion":[{"value":"2017-12-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}