Abstract
Microtask crowdsourcing has become an appealing approach to collecting large-scale high-quality labeled data across a wide range of domains. As the crowd workers may be unreliable, the most fundamental question is how to aggregate the noisy annotations provided by these potentially unreliable workers. Although various factors such as worker reliability and task features are considered in the literature, they are not meaningfully combined in a unified and consistent framework. In this work, we propose a semi-crowdsourced deep generative approach called S-DARFC which combines Bayesian graphical models and deep learning techniques and unifies factors including the worker reliability, task features, task clustering structure, and semi-crowdsourcing. Graphical models are good at finding a structure that is interpretable and generalizes to new tasks easily and deep learning techniques are able to learn a flexible representation of complex high-dimensional unstructured data (e.g., task features). Extensive experiments based on six real-world tasks including text and image classification demonstrate the effectiveness of our proposed approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
BCCwords is not included because it only works in text classification datasets.
References
Bi, W., Wang, L., Kwok, J.T., Tu, Z.: Learning to predict from crowdsourced data. In: UAI, pp. 82–91 (2014)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
Blei, D.M., Kucukelbir, A., McAuliffe, J.D.: Variational inference: a review for statisticians. J. Am. Stat. Assoc. 112(518), 859–877 (2017)
Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Ding, X., Liu, T., Duan, J., Nie, J.Y.: Mining user consumption intention from social media using domain adaptive convolutional neural network. In: AAAI, vol. 15, pp. 2389–2395 (2015)
Dizaji, K.G., Huang, H.: Sentiment analysis via deep hybrid textual-crowd learning model. In: Thirty-Second AAAI Conference on Artificial Intelligence (AAAI 2018) (2018)
Gadiraju, U., Kawase, R., Dietze, S.: A taxonomy of microtasks on the web. In: Proceedings of the 25th ACM Conference on Hypertext and Social Media, pp. 218–223. ACM (2014)
Ipeirotis, P.G., Provost, F., Wang, J.: Quality management on Amazon mechanical turk. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 64–67. ACM (2010)
Johnson, M.J., Duvenaud, D.K., Wiltschko, A., Adams, R.P., Datta, S.R.: Composing graphical models with neural networks for structured representations and fast inference. In: Advances in Neural Information Processing Systems, pp. 2946–2954 (2016)
Karger, D.R., Oh, S., Shah, D.: Iterative learning for reliable crowdsourcing systems. In: Advances in Neural Information Processing Systems, pp. 1953–1961 (2011)
Kazai, G., Kamps, J., Milic-Frayling, N.: Worker types and personality traits in crowdsourcing relevance labels. In: Proceedings of the 20th ACM International Conference on Information and Knowledge Management, pp. 1941–1944. ACM (2011)
Kim, H.C., Ghahramani, Z.: Bayesian classifier combination. In: Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, pp. 619–627 (2012)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: International Conference on Learning Representations (ICLR) (2014). arXiv:1312.6114
Luo, Y., Tian, T., Shi, J., Zhu, J., Zhang, B.: Semi-crowdsourced clustering with deep generative models. In: Advances in Neural Information Processing Systems, pp. 3212–3222 (2018)
Moreno, A., Terwiesch, C.: Doing business with strangers: reputation in online service marketplaces. Inf. Syst. Res. 25(4), 865–886 (2014)
Pang, B., Lee, L.: Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 115–124 (2005)
Raykar, V.C., et al.: Learning from crowds. J. Mach. Learn. Res. 11(Apr), 1297–1322 (2010)
Rodrigues, F., Lourenco, M., Ribeiro, B., Pereira, F.C.: Learning supervised topic models for classification and regression from crowds. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2409–2422 (2017)
Rodrigues, F., Pereira, F., Ribeiro, B.: Learning from multiple annotators: distinguishing good from random labelers. Pattern Recogn. Lett. 34(12), 1428–1436 (2013)
Rodrigues, F., Pereira, F.C.: Deep learning from crowds. In: The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), pp. 1611–1618. AAAI Press (2018)
Simpson, E., Roberts, S.J., Psorakis, I., Smith, A.: Dynamic Bayesian combination of multiple imperfect classifiers. Decis. Making Imperfection 474, 1–35 (2013)
Simpson, E.D., et al.: Language understanding in the wild: combining crowdsourcing and machine learning. In: Proceedings of the 24th International Conference on World Wide Web, pp. 992–1002 (2015)
van Engelen, J.E., Hoos, H.H.: A survey on semi-supervised learning. Mach. Learn. 109(2), 373–440 (2019). https://doi.org/10.1007/s10994-019-05855-6
Wainwright, M.J., Jordan, M.I.: Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. 1(1–2), 1–305 (2008)
Wang, J., Ipeirotis, P.G., Provost, F.: Cost-effective quality assurance in crowd labeling. Inf. Syst. Res. 28(1), 137–158 (2017)
Wei, X., Zhang, Z., Zhang, M., Zeng, D.D.: Combining crowd and machine intelligence to detect false news in social media. SSRN 3355763 (2019)
Yi, J., Jin, R., Jain, S., Yang, T., Jain, A.K.: Semi-crowdsourced clustering: generalizing crowd labeling by robust distance metric learning. In: Advances in Neural Information Processing Systems, pp. 1772–1780 (2012)
Yin, L., Liu, Y., Zhang, W., Yu, Y., et al.: Truth inference with a deep clustering-based aggregation model. IEEE Access 8, 16662–16675 (2020)
Zhang, M., Wei, X., Guo, X., Chen, G., Wei, Q.: Identifying complements and substitutes of products: a neural network framework based on product embedding. ACM Trans. Knowl. Discov. Data (TKDD) 13(3), 1–29 (2019)
Zhang, M., Wei, X., Zeng, D.D.: A matter of reevaluation: incentivizing users to contribute reviews in online platforms. Decis. Support Syst. 128, 113158 (2020)
Acknowledgement
The authors would like to thank Dr. Yong Ge, Dr. Wei Chen, and Dr. Jason Pacheco for their useful comments. Mingyue Zhang is the corresponding author. This work is partially supported by the following grants: the National Key Research and Development Program of China under Grant Nos. 2016QY02D0305 and 2017YFC0820105; the National Natural Science Foundation of China under Grant Nos. 71621002 and 71802024.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wei, X., Zhang, M., Zeng, D.D. (2021). Learning from Crowd Labeling with Semi-crowdsourced Deep Generative Models. In: Sun, Y., Liu, D., Liao, H., Fan, H., Gao, L. (eds) Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2020. Communications in Computer and Information Science, vol 1330. Springer, Singapore. https://doi.org/10.1007/978-981-16-2540-4_8
Download citation
DOI: https://doi.org/10.1007/978-981-16-2540-4_8
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-2539-8
Online ISBN: 978-981-16-2540-4
eBook Packages: Computer ScienceComputer Science (R0)
