{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T20:45:00Z","timestamp":1730321100916,"version":"3.28.0"},"publisher-location":"New York, NY, USA","reference-count":49,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,13]]},"DOI":"10.1145\/3308558.3313599","type":"proceedings-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T12:17:59Z","timestamp":1557749879000},"page":"2158-2168","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Scalpel-CD: Leveraging Crowdsourcing and Deep Probabilistic Modeling for Debugging Noisy Training Data"],"prefix":"10.1145","author":[{"given":"Jie","family":"Yang","sequence":"first","affiliation":[{"name":"U. of Fribourg, Switzerland"}]},{"given":"Alisa","family":"Smirnova","sequence":"additional","affiliation":[{"name":"University of Fribourg, Switzerland"}]},{"given":"Dingqi","family":"Yang","sequence":"additional","affiliation":[{"name":"U. of Fribourg, Switzerland"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Australia"}]},{"given":"Yuan","family":"Lu","sequence":"additional","affiliation":[{"name":"ING Bank, Netherlands"}]},{"given":"Philippe","family":"Cudre-Mauroux","sequence":"additional","affiliation":[{"name":"U. of Fribourg, Switzerland"}]}],"member":"320","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association of Computational Linguistics, 54-59","author":"Alfonseca Enrique","year":"2012","unstructured":"Enrique Alfonseca , Katja Filippova , Jean-Yves Delort , and Guillermo Garrido . 2012 . Pattern learning for relation extraction with a hierarchical topic model . In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association of Computational Linguistics, 54-59 . Enrique Alfonseca, Katja Filippova, Jean-Yves Delort, and Guillermo Garrido. 2012. Pattern learning for relation extraction with a hierarchical topic model. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association of Computational Linguistics, 54-59."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2791285"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1285773"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718501"},{"key":"e_1_3_2_1_6_1","unstructured":"Aleksandar Chakarov Aditya Nori Sriram Rajamani Shayak Sen and Deepak Vijaykeerthy. 2016. Debugging machine learning tasks. arXiv preprint arXiv:1603.07292(2016). Aleksandar Chakarov Aditya Nori Sriram Rajamani Shayak Sen and Deepak Vijaykeerthy. 2016. Debugging machine learning tasks. arXiv preprint arXiv:1603.07292(2016)."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.42"},{"key":"e_1_3_2_1_8_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics","author":"Dawid Alexander Philip","year":"1979","unstructured":"Alexander Philip Dawid and Allan M Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics ( 1979 ), 20-28. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics (1979), 20-28."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080832"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_1_11_1","unstructured":"Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608(2017). Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608(2017)."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.64"},{"key":"e_1_3_2_1_13_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning (ICML), 1183-1192","author":"Gal Yarin","year":"2017","unstructured":"Yarin Gal , Riashat Islam , and Zoubin Ghahramani . 2017 . Deep bayesian active learning with image data . Proceedings of the 34th International Conference on Machine Learning (ICML), 1183-1192 . Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. Proceedings of the 34th International Conference on Machine Learning (ICML), 1183-1192."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866701"},{"volume-title":"European Conference on Computer Vision (ECCV). Springer, 67-84","author":"Joulin Armand","key":"e_1_3_2_1_15_1","unstructured":"Armand Joulin , Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. 2016. Learning visual features from large weakly supervised data . In European Conference on Computer Vision (ECCV). Springer, 67-84 . Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. 2016. Learning visual features from large weakly supervised data. In European Conference on Computer Vision (ECCV). Springer, 67-84."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882(2014). Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882(2014).","DOI":"10.3115\/v1\/D14-1181"},{"key":"e_1_3_2_1_17_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014).","author":"Kingma Diederik P","year":"2014","unstructured":"Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014)."},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 2nd International Conference on Learning Representations (ICLR).","author":"Kingma Diederik P","year":"2013","unstructured":"Diederik P Kingma and Max Welling . 2013 . Auto-encoding variational bayes . In Proceedings of the 2nd International Conference on Learning Representations (ICLR). Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_1_19_1","volume-title":"International Conference on Machine Learning (ICML). 1885-1894","author":"Koh Pang Wei","year":"2017","unstructured":"Pang Wei Koh and Percy Liang . 2017 . Understanding black-box predictions via influence functions . In International Conference on Machine Learning (ICML). 1885-1894 . Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML). 1885-1894."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052646"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994514"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Edith Law and Luis von Ahn. 2011. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning5 3(2011) 1-121. Edith Law and Luis von Ahn. 2011. Human computation. Synthesis Lectures on Artificial Intelligence and Machine Learning5 3(2011) 1-121.","DOI":"10.2200\/S00371ED1V01Y201107AIM013"},{"key":"e_1_3_2_1_23_1","volume-title":"ICML Workshop: Machine Learning Meets Crowdsourcing","author":"Li Hongwei","year":"2013","unstructured":"Hongwei Li , Bin Yu , and Dengyong Zhou . 2013 . Error rate analysis of labeling by crowdsourcing . In ICML Workshop: Machine Learning Meets Crowdsourcing . Atalanta, Georgia, USA. Hongwei Li, Bin Yu, and Dengyong Zhou. 2013. Error rate analysis of labeling by crowdsourcing. In ICML Workshop: Machine Learning Meets Crowdsourcing. Atalanta, Georgia, USA."},{"key":"e_1_3_2_1_24_1","unstructured":"Jiwei Li Will Monroe and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220(2016). Jiwei Li Will Monroe and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220(2016)."},{"key":"e_1_3_2_1_25_1","unstructured":"Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490(2016). Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490(2016)."},{"key":"e_1_3_2_1_26_1","first-page":"2579","article-title":"Visualizing data using t-SNE","author":"van der Maaten Laurens","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton . 2008 . Visualizing data using t-SNE . Journal of Machine Learning Research9 , Nov (2008), 2579 - 2605 . Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research9, Nov (2008), 2579-2605.","journal-title":"Journal of Machine Learning Research9"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13287"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866714"},{"key":"e_1_3_2_1_29_1","volume-title":"International Conference on Machine Learning (ICML). 1727-1736","author":"Miao Yishu","year":"2016","unstructured":"Yishu Miao , Lei Yu , and Phil Blunsom . 2016 . Neural variational inference for text processing . In International Conference on Machine Learning (ICML). 1727-1736 . Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In International Conference on Machine Learning (ICML). 1727-1736."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.3115\/1218955.1218990"},{"key":"e_1_3_2_1_31_1","first-page":"3567","article-title":"Data programming: Creating large training sets, quickly","author":"Ratner Alexander J","year":"2016","unstructured":"Alexander J Ratner , Christopher M De Sa , Sen Wu , Daniel Selsam , and Christopher Re\u00b4 . 2016 . Data programming: Creating large training sets, quickly . In Advances in Neural Information Processing Systems (NIPS). 3567 - 3575 . Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Re\u00b4. 2016. Data programming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems (NIPS). 3567-3575.","journal-title":"Advances in Neural Information Processing Systems (NIPS)."},{"key":"e_1_3_2_1_32_1","first-page":"1297","article-title":"Learning from crowds","author":"Raykar Vikas C","year":"2010","unstructured":"Vikas C Raykar , Shipeng Yu , Linda H Zhao , Gerardo Hermosillo Valadez , Charles Florin , Luca Bogoni , and Linda Moy . 2010 . Learning from crowds . Journal of Machine Learning Research11 , Apr (2010), 1297 - 1322 . Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research11, Apr (2010), 1297-1322.","journal-title":"Journal of Machine Learning Research11"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052708"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/1889788.1889799"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741083"},{"key":"e_1_3_2_1_37_1","volume-title":"Active Learning Literature Survey","author":"Settles Burr","year":"2010","unstructured":"Burr Settles . 2010. Active Learning Literature Survey . University of Wisconsin , Madison52, 55-66 ( 2010 ), 11. Burr Settles. 2010. Active Learning Literature Survey. University of Wisconsin, Madison52, 55-66 (2010), 11."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_2_1_39_1","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association of Computational Linguistics, 721-729","author":"Takamatsu Shingo","year":"2012","unstructured":"Shingo Takamatsu , Issei Sato , and Hiroshi Nakagawa . 2012 . Reducing wrong labels in distant supervision for relation extraction . In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association of Computational Linguistics, 721-729 . Shingo Takamatsu, Issei Sato, and Hiroshi Nakagawa. 2012. Reducing wrong labels in distant supervision for relation extraction. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). Association of Computational Linguistics, 721-729."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339571"},{"key":"e_1_3_2_1_41_1","first-page":"1","article-title":"Making better use of the crowd: How crowdsourcing can advance machine learning research","volume":"193","author":"Vaughan Jennifer Wortman","year":"2018","unstructured":"Jennifer Wortman Vaughan . 2018 . Making better use of the crowd: How crowdsourcing can advance machine learning research . Journal of Machine Learning Research18 , 193 (2018), 1 - 46 . Jennifer Wortman Vaughan. 2018. Making better use of the crowd: How crowdsourcing can advance machine learning research. Journal of Machine Learning Research18, 193 (2018), 1-46.","journal-title":"Journal of Machine Learning Research18"},{"key":"e_1_3_2_1_42_1","first-page":"2035","article-title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier R Movellan , and Paul L Ruvolo . 2009 . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Advances in Neural Information Processing Systems (NIPS). 2035 - 2043 . Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems (NIPS). 2035-2043.","journal-title":"Advances in Neural Information Processing Systems (NIPS)."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2463706"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.14778\/1952376.1952378"},{"key":"e_1_3_2_1_45_1","volume-title":"Proceedings of the 28th International Conference on Machine Learning (ICML). 1161-1168","author":"Yan Yan","year":"2011","unstructured":"Yan Yan , Glenn M Fung , R\u00f3mer Rosales , and Jennifer G Dy . 2011 . Active learning from crowds . In Proceedings of the 28th International Conference on Machine Learning (ICML). 1161-1168 . Yan Yan, Glenn M Fung, R\u00f3mer Rosales, and Jennifer G Dy. 2011. Active learning from crowds. In Proceedings of the 28th International Conference on Machine Learning (ICML). 1161-1168."},{"key":"e_1_3_2_1_46_1","volume-title":"International Conference on Artificial Intelligence and Statistics (AISTATS). 932-939","author":"Yan Yan","year":"2010","unstructured":"Yan Yan , R\u00f3mer Rosales , Glenn Fung , Mark W Schmidt , Gerardo H Valadez , Luca Bogoni , Linda Moy , and Jennifer G Dy . 2010 . Modeling annotator expertise: Learning when everybody knows a bit of something . In International Conference on Artificial Intelligence and Statistics (AISTATS). 932-939 . Yan Yan, R\u00f3mer Rosales, Glenn Fung, Mark W Schmidt, Gerardo H Valadez, Luca Bogoni, Linda Moy, and Jennifer G Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In International Conference on Artificial Intelligence and Statistics (AISTATS). 932-939."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186033"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1203"},{"key":"e_1_3_2_1_49_1","volume-title":"Proceedings of the 5th International Conference on Learning Representations (ICLR).","author":"Zhang Chiyuan","year":"2016","unstructured":"Chiyuan Zhang , Samy Bengio , Moritz Hardt , Benjamin Recht , and Oriol Vinyals . 2016 . Understanding deep learning requires rethinking generalization . In Proceedings of the 5th International Conference on Learning Representations (ICLR). Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2016. Understanding deep learning requires rethinking generalization. In Proceedings of the 5th International Conference on Learning Representations (ICLR)."}],"event":{"name":"WWW '19: The Web Conference","sponsor":["IW3C2 International World Wide Web Conference Committee"],"location":"San Francisco CA USA","acronym":"WWW '19"},"container-title":["The World Wide Web Conference"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3308558.3313599","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T06:39:22Z","timestamp":1681799962000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3308558.3313599"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":49,"alternative-id":["10.1145\/3308558.3313599","10.1145\/3308558"],"URL":"https:\/\/doi.org\/10.1145\/3308558.3313599","relation":{},"subject":[],"published":{"date-parts":[[2019,5,13]]},"assertion":[{"value":"2019-05-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}