Abstract
A typical crowdsourcing task is concept labeling, where participants annotate e.g. images using a list of predefined concepts. Recent popular campaigns for environmental bird monitoring even use hierarchies of concepts (taxonomies of species) to obtain the most precise labeling of bird images. But in most applications, volunteer opinions are isolated from each other, and decision is taken upon majority voting. In this work we propose a new iterative labeling process where participants express their opinions together, on ascending levels of the taxonomy. Level changes are performed to minimize opinion conflict, according to the belief function theory. This complex task is orchestrated by a finite-state automaton driven by conflict measures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Amsterdamer, Y., Davidson, S.B., Milo, T., Novgorodov, S., Somech, A.: Oassis: query driven crowd mining. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 589–600. ACM (2014)
Blanco, H.H.R.: Machine-learning for spammer detection in crowd-sourcing. Human Computation AAAI Technical Report (2012)
Boim, R., Greenshpan, O., Milo, T., Novgorodov, S., Polyzotis, N., Tan, W.C.: Asking the right questions in crowd data sourcing. In: 2012 IEEE 28th International Conference on Data Engineering, pp. 1261–1264. IEEE (2012)
Delmotte, F., Smets, P.: Target identification based on the transferable belief model interpretation of dempster-shafer model. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 34(4), 457–471 (2004)
Dempster, A.P.: Upper and lower probabilities induced by a multivalued mapping. Ann. Math. Stat. 38, 325–339 (1967)
Farrell III, W.J., Knapp, A.M.: Multisource taxonomy-based classication using the transferable belief model. In: Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications 2012. vol. 8407, pp. 35–41. SPIE (2012)
Folorunso, O., Mustapha, O.A.: A fuzzy expert system to trust-based access control in crowdsourcing environments. Appl. Comput. Inf. 11(2), 116–129 (2015)
Gross-Amblard, D., Tommasi, M., Rakotoniaina, I., Thierry, C., Singh, R., Jacoboni, L.: Headwork: a data-centric crowdsourcing platform for complex tasks and participants. In: EDBT 2024 (2024)
Jousselme, A.L., Grenier, D., Bossé, É.: A new distance between two bodies of evidence. Inf. Fusion 2(2), 91–101 (2001)
Karampinas, D., Triantafillou, P.: Crowdsourcing taxonomies. In: Simperl, E., Cimiano, P., Polleres, A., Corcho, O., Presutti, V. (eds.) The Semantic Web: Research and Applications, ESWC 2012, LNCS, vol. 7295, pp. 545–559. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-30284-8_43
Kazai, G., Kamps, J., Milic-Frayling, N.: The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy (2012)
Khattak, F.K., Salleb-Aouissi, A.: Quality control of crowd labeling through expert evaluation. In: Proceedings of the NIPS 2nd Workshop on Computational Social Science and the Wisdom of Crowds, vol. 2, p. 5 (2011)
Martin, A.: About conflict in the theory of belief functions. In: Denoeux, T., Masson, MH. (eds.) Belief Functions: Theory and Applications. Advances in Intelligent and Soft Computing, vol. 164, pp. 161–168. Springer, Berlin, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29461-7_19
Martin, A.: Conflict Management in Information Fusion with Belief Functions. In: Bossé, É., Rogova, G. (eds.) Information Quality in Information Fusion and Decision Making. Information Fusion and Data Science, pp. 79–97. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-03643-0_4
Martin, A., Jousselme, A.L., Osswald, C.: Conflict measure for the discounting operation on belief functions. In: Information Fusion, 2008 11th International Conference on, pp. 1–8. IEEE (2008)
Ross, J., Zaldivar, A., Irani, L., Tomlinson, B.: Who are the turkers? worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Technical Report (2009)
Roy, S.B., Lykourentzou, I., Thirumuruganathan, S., Amer-Yahia, S., Das, G.: Crowds, not drones: modeling human factors in interactive crowdsourcing. In: DBCrowd 2013-VLDB Workshop on Databases and Crowdsourcing, pp. 39–42. CEUR-WS (2013)
Shafer, G.: A mathematical theory of evidence, vol. 42. Princeton University Press, Princeton (1976)
Thierry, C., Dubois, J.C., Le Gall, Y., Martin, A.: Modeling uncertainty and inaccuracy on data from crowdsourcing plateforms: Monitor. In: Proceedings of the 31st International Conference on Tools with Artificial Intelligence (2019)
Thierry, C., Martin, A., Dubois, J.C., Le Gall, Y.: Estimation of the qualification and behavior of a contributor and aggregation of his answers in a crowdsourcing context. Expert Syst. Appl. 216, 119496 (2023)
Thierry, C., Martin, A., Le Gall, Y., Dubois, J.C.: Modeling evolutionary responses in crowdsourcing MCQ using belief function theory. Procedia Comput. Sci. 225, 2575–2584 (2023)
Yongxin, T., Caleb, C.C., Chen, J.Z., Yatao, L., Lei, C.: Crowdcleaner: data cleaning for multi-version data on the web via crowdsourcing. In: 2014 IEEE 30th International Conference Data Engineering (ICDE) (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Thierry, C., Gross-Amblard, D., Le Gall, Y., Dubois, JC. (2024). Automated Hierarchical Conflict Reduction for Crowdsourced Annotation Tasks Using Belief Functions. In: Bi, Y., Jousselme, AL., Denoeux, T. (eds) Belief Functions: Theory and Applications. BELIEF 2024. Lecture Notes in Computer Science(), vol 14909. Springer, Cham. https://doi.org/10.1007/978-3-031-67977-3_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-67977-3_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-67976-6
Online ISBN: 978-3-031-67977-3
eBook Packages: Computer ScienceComputer Science (R0)