Abstract
This paper explores the use of crowdsourcing to classify statement types in film reviews to assess their information quality. Employing the Argument Type Identification Procedure which uses the Periodic Table of Arguments to categorize arguments, the study aims to connect statement types to the overall argument strength and information reliability. Focusing on non-expert annotators in a crowdsourcing environment, the research assesses their reliability based on various factors including language proficiency and annotation experience. Results indicate the importance of careful annotator selection and training to achieve high inter-annotator agreement and highlight challenges in crowdsourcing statement classification for information quality assessment.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
See, for instance, www.rottentomatoes.com/m/split_2017.
References
Addawood, A., Bashir, M.: “What is your evidence?” A study of controversial topics on social media. In: ArgMining2016, pp. 1–11. ACL, August 2016
Bosc, T., Cabrio, E., Villata, S.: DART: a dataset of arguments and their relations on Twitter. In: LREC, pp. 1258–1263. ACL (2016)
Ceolin, D., Primiero, G., Soprano, M., Wielemaker, J.: Transparent assessment of information quality of online reviews using formal argumentation theory. Inf. Syst. 110, 102107 (2022)
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20, 37–46 (1960)
Dancey, C.P., Reidy, J.: Statistics Without Maths for Psychology: Using SPSS for Windows. Prentice-Hall Inc., USA (2004)
Feier, A.: Reach consensus faster by using IAA charts in the annotation lab. https://www.johnsnowlabs.com/reach-consensus-faster-by-using-iaa-charts-in-the-annotation-lab/. Accessed 03 Apr 2023
Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76, 378–382 (1971)
Goudas, T., Louizos, C., Petasis, G., Karkaletsis, V.: Argument extraction from news, blogs, and social media. In: Likas, A., Blekas, K., Kalles, D. (eds.) SETN 2014. LNCS (LNAI), vol. 8445, pp. 287–299. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07064-3_23
Hripcsak, G., Rothschild, A.S.: Agreement, the F-measure, and reliability in information retrieval. J. Am. Med. Inf. Ass. 12, 296–298 (2005)
Iskender, N., Schaefer, R., Polzehl, T., Möller, S.: Argument mining in tweets: comparing crowd and expert annotations for automated claim and evidence detection. In: Métais, E., Meziane, F., Horacek, H., Kapetanios, E. (eds.) NLDB 2021. LNCS, vol. 12801, pp. 275–288. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-80599-9_25
Jiang, Y., Zhu, H., Kummerfeld, J.K., Li, Y., Lasecki, W.: A novel workflow for accurately and efficiently crowdsourcing predicate senses and argument labels. In: EMNLP, pp. 415–421. ACL, November 2020
Lawrence, J., Reed, C.: Argument mining: a survey. Comput. Linguist. 45(4), 765–818 (2019)
Lee, G.E., Sun, A.: A study on agreement in PICO span annotations. In: Proceedings of the 42nd International ACM SIGIR Conference, pp. 1149–1152. ACM (2019)
Li, M., Geng, S., Gao, Y., Peng, S., Liu, H., Wang, H.: Crowdsourcing argumentation structures in Chinese hotel reviews. In: SMC, pp. 87–92. IEEE (2017)
Lindahl, A.: Annotating argumentation in Swedish social media. In: ArgMining Workshop, pp. 100–105. ACL, December 2020
McHugh, M.L.: Interrater reliability: the kappa statistic. Biochemia Medica 22, 276–282 (2012)
Miller, T., Sukhareva, M., Gurevych, I.: A streamlined method for sourcing discourse-level argumentation annotations from the crowd. In: NAACL, pp. 1790–1796. ACL, June 2019
Nordquist, R.: Definition and examples of vagueness in language. https://www.thoughtco.com/vagueness-language-1692483. Accessed 25 Sept 2023
Plug, H., Wagemans, J.: Argument-checken als een methode voor het identificeren van desinformatie. In: Proceedings of VIOT2024. University of Twente (2024)
Ratner, B.: The correlation coefficient: Its values range between +1/-1 or do they? J. Targ. Measur. Anal. Market. 17(2), 139–142 (2009)
Schaefer, R., Stede, M.: Annotation and detection of arguments in tweets. In: Proceedings of the 7th Workshop on Argument Mining, pp. 53–58. ACL, December 2020
Schober, P., Boer, C., Schwarte, L.A.: Correlation coefficients: appropriate use and interpretation. Anesth. Analg. 126(5) (2018)
Scott, W.A.: Reliability of content analysis: the case of nominal scale coding. Public Opin. Q. 19(3), 321–325 (1955)
Soprano, M., Roitero, K., Bombassei De Bona, F., Mizzaro, S.: Crowd_frame: a simple and complete framework to deploy complex crowdsourcing tasks off-the-shelf. In: WSDM 2022, pp. 1605–1608. ACM (2022)
Soprano, M., et al.: The many dimensions of truthfulness: crowdsourcing misinformation assessments on a multidimensional scale. IP &M 58(6), 102710 (2021)
Wagemans, J.: Constructing a periodic table of arguments. In: Argumentation, Objectivity, and Bias: Proceedings of the 11th International Conference of the OSSA, pp. 1–12, May 2016
Wagemans, J.H.M.: Argument Type Identification Procedure (ATIP) - Version 4. https://periodic-table-of-arguments.org/argument-type-identification-procedure. Accessed 21 Mar 2022
World Health Organization: Infodemic. https://www.who.int/health-topics/infodemic/understanding-the-infodemic-and-misinformation-in-the-fight-against-covid-19. Accessed 29 Apr 2024
Acknowledgements
This research is supported by the Netherlands eScience Center project “The Eye of the Beholder” (project nr. 027.020.G15), and it is part of the AI, Media & Democracy Lab (Dutch Research Council project number: NWA.1332.20.009). For more information about the lab and its further activities, visit https://www.aim4dem.nl/.
This research is also supported by the European Union’s NextGenerationEU PNRR M4.C2.1.1 – PRIN 2022 project “20227F2ZN3 MoT–The Measure of Truth: An Evaluation-Centered Machine-Human Hybrid Framework for Assessing Information Truthfulness” (20227F2ZN3_001, CUP G53D23002800006), and by the Strategic Plan of the University of Udine–Interdepartmental Project on Artificial Intelligence (2020–2025).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Singh, J., Soprano, M., Roitero, K., Ceolin, D. (2024). Crowdsourcing Statement Classification to Enhance Information Quality Prediction. In: Preuss, M., Leszkiewicz, A., Boucher, JC., Fridman, O., Stampe, L. (eds) Disinformation in Open Online Media. MISDOOM 2024. Lecture Notes in Computer Science, vol 15175. Springer, Cham. https://doi.org/10.1007/978-3-031-71210-4_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-71210-4_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71209-8
Online ISBN: 978-3-031-71210-4
eBook Packages: Computer ScienceComputer Science (R0)