Abstract
In many decision-making scenarios, it is necessary to aggregate information from a number of different agents, be they people, sensors or computer systems. Each agent may have complementary analysis skills or access to different information, and their reliability may vary greatly. An example is using crowdsourcing to employ multiple human workers to perform analytical tasks. This chapter presents an information-theoretic approach to selecting informative decision-making agents, assigning them to specific tasks and combining their responses using a Bayesian method. For settings in which the agents are paid to undertake tasks, we introduce an automated algorithm for selecting a cohort of agents (workers) to complete informative tasks, hiring new members of the cohort and identifying those members whose services are no longer needed. We demonstrate empirically how our intelligent task assignment approach improves the accuracy of combined decisions while requiring fewer responses from the crowd.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
The Text REtrieval Conference, or TREC, consists of several competitions. For the crowdsourcing challenge, see https://sites.google.com/site/treccrowd/.
References
Bashir, M., Anderton, J., Wu, J., Ekstrand-Abueg, M., Golbus, P.B., Pavlu, V., Aslam, J.A.: Northeastern university runs at the TREC12 crowdsourcing track. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Berger, J.O.: Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics. Springer, New York (1985)
Bishop, C.M.: Pattern recognition and machine learning. Information Science and Statistics, 4th edn. Springer, Heidelberg (2006)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Bloodgood, M., Callison-Burch, C.: Using mechanical turk to build machine translation evaluation sets. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazons Mechanical Turk, pp. 208–211. Association for Computational Linguistics (2010)
Chen, X., Bennett, P.N., Collins-Thompson, K., Horvitz, E.: Pairwise ranking aggregation in a crowdsourced setting. In: Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, pp. 193–202. ACM (2013)
Dasgupta, S.: Analysis of a greedy active learning strategy. In: Advances in Neural Information Processing Systems, vol. 17, pp. 337–344. MIT Press, Cambridge (2004)
Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.) 28(1), 20–28 (1979)
Donmez, P., Carbonell, J., Schneider, J.: A probabilistic framework to learn from multiple annotators with time-varying accuracy. In: SIAM International Conference on Data Mining (SDM), pp. 826–837. Society for Industrial and Applied Mathematics (2010)
Fawcett, T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27(8), 861–874 (2006)
Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell. PAMI–6(6), 721–741 (1984)
Harris, C., Srinivasan, P.: Using hybrid methods for relevance assessment in TREC Crowd ’12. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Hu, Q., Xu, Z., Huang, X., Ye, Z.: York university at TREC 2012: crowdsourcing track. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on Amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64–67. ACM (2010)
Kamar, E., Hacker, S., Horvitz, E.: Combining human and machine intelligence in large-scale crowdsourcing. In: Proceedings of the 11th International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS’12, pp. 467–474. International Foundation for Autonomous Agents and Multi-Agent Systems (2012)
Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)
Levenberg, A., Pulman, S., Moilanen, K., Simpson, E., Roberts, S.: Economic indicators from web text using sentiment composition. Int. J. Comput. Commun. Eng. (2014)
Liu, Q., Peng, J., Ihler, A.: Variational inference for crowdsourcing. In: Advances in Neural Information Processing Systems, vol. 25, pp. 701–709. MIT Press, Cambridge (2012)
Nellapati, R., Peerreddy, S., Singhal, P.: Skierarchy: extending the power of crowdsourcing using a hierarchy of domain experts, crowd and machine learning. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Quinn, A.J., Bederson, B.B., Yeh, T., Lin, J.: Crowdflow: integrating machine learning with mechanical turk for speed-cost-quality flexibility. Technical Report HCIL-2010-09, University of Maryland, College Park (2010)
Raykar, V.C., Yu, S.: Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res. 13, 491–518 (2012)
Simpson, E., Roberts, S., Psorakis, I., Smith, A.: Dynamic Bayesian combination of multiple imperfect classifiers. In: Decision Making and Imperfection, pp. 1–35. Springer, Heidelberg (2013)
Simpson, E., Reece, S., Penta, A., Ramchurn, G., Roberts, S.: Using a Bayesian model to combine LDA features with crowdsourced responses. In: The Twenty-First Text REtrieval Conference (TREC 2012), Crowdsourcing Track. NIST (2013)
Smith, A.M., Lynn, S., Sullivan, M., Lintott, C.J., Nugent, P.E., Botyanszki, J., Kasliwal, M., Quimby, R., Bamford, S.P., Fortson, L.F., Schawinski, K., Hook, I., Blake, S., Podsadlowski, P., Jonsson, J.J., Gal-Yam, A., Arcavi, I., Howell, D.A., Bloom, J.S., Jacobsen, J., Kulkarni, S.R., Law, N.M., Ofek, E.O., Walters, R. Galaxy Zoo supernovae. Monthly Notices R. Astron. Soc. (2010)
Smith, A., Lintott, C.: Web-scale citizen science: from Galaxy Zoo to the Zooniverse. In: Proceedings of the Royal Society Discussion Meeting ‘Web Science: A New Frontier’. The Royal Society (2010)
Smucker, M.D., Kazai, G., Lease, M.: Overview of the TREC 2012 crowdsourcing track. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Smucker, M.D., Kazai, G., Lease, M.: TREC 2012 crowdsourcing track TRAT task results. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Yan, Y., Fung, G.M., Rosales, R., Dy, J.G.: Active learning from crowds. In: Proceedings of the 28th International Conference on Machine Learning, ICML’11, pp. 1161–1168 (2011)
Zhang, C., Zeng, M., Sang, X., Zhang, K., Kang, H.: BUPT\_PRIS at TREC 2012 crowdsourcing track 1. In: The Twenty-First Text REtrieval Conference (TREC 2012). NIST (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Simpson, E., Roberts, S. (2015). Bayesian Methods for Intelligent Task Assignment in Crowdsourcing Systems. In: Guy, T., Kárný, M., Wolpert, D. (eds) Decision Making: Uncertainty, Imperfection, Deliberation and Scalability. Studies in Computational Intelligence, vol 538. Springer, Cham. https://doi.org/10.1007/978-3-319-15144-1_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-15144-1_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-15143-4
Online ISBN: 978-3-319-15144-1
eBook Packages: EngineeringEngineering (R0)