{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T19:55:13Z","timestamp":1716494113788},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,7,12]],"date-time":"2016-07-12T00:00:00Z","timestamp":1468281600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2016,7,14]]},"abstract":"Despite technological advances, algorithmic search systems still have difficulty with complex or subtle information needs. For example, scenarios requiring deep semantic interpretation are a challenge for computers. People, on the other hand, are well suited to solving such problems. As a result, there is an opportunity for humans and computers to collaborate during the course of a search in a way that takes advantage of the unique abilities of each. While search tools that rely on human intervention will never be able to respond as quickly as current search engines do, recent research suggests that there are scenarios where a search engine could take more time if it resulted in a much better experience. This article explores how crowdsourcing can be used at query time to augment key stages of the search pipeline. We first explore the use of crowdsourcing to improve search result ranking. When the crowd is used to replace or augment traditional retrieval components such as query expansion and relevance scoring, we find that we can increase robustness against failure for query expansion and improve overall precision for results filtering. However, the gains that we observe are limited and unlikely to make up for the extra cost and time that the crowd requires. We then explore ways to incorporate the crowd into the search process that more drastically alter the overall experience. We find that using crowd workers to support rich query understanding and result processing appears to be a more worthwhile way to make use of the crowd during search. Our results confirm that crowdsourcing can positively impact the search experience but suggest that significant changes to the search process may be required for crowdsourcing to fulfill its potential in search systems.<\/jats:p>","DOI":"10.1145\/2897368","type":"journal-article","created":{"date-parts":[[2016,7,15]],"date-time":"2016-07-15T14:36:21Z","timestamp":1468593381000},"page":"1-24","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Using the Crowd to Improve Search Result Ranking and the Search Experience"],"prefix":"10.1145","volume":"7","author":[{"given":"Yubin","family":"Kim","sequence":"first","affiliation":[{"name":"Carnegie Mellon University"}]},{"given":"Kevyn","family":"Collins-Thompson","sequence":"additional","affiliation":[{"name":"University of Michigan"}]},{"given":"Jaime","family":"Teevan","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA"}]}],"member":"320","published-online":{"date-parts":[[2016,7,12]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390419"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1507509.1507511"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207710"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187971"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541884"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433420"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646059"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1099554.1099727"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9162-z"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281192.1281216"},{"key":"e_1_2_1_14_1","volume-title":"Franklin","author":"Demartini Gianluca","year":"2013","unstructured":"Gianluca Demartini , Beth Trushkowsky , Tim Kraska , and Michael J . Franklin . 2013 . CrowdQ: Crowdsourced query understanding. In Proc. CIDR. Gianluca Demartini, Beth Trushkowsky, Tim Kraska, and Michael J. Franklin. 2013. CrowdQ: Crowdsourced query understanding. In Proc. CIDR."},{"key":"e_1_2_1_15_1","volume-title":"Task-based search: A search engine perspective. Talk at NSF Task-Based Information Search Systems Workshop. (March 14--15","author":"Dumais Susan T.","year":"2013","unstructured":"Susan T. Dumais . 2013. Task-based search: A search engine perspective. Talk at NSF Task-Based Information Search Systems Workshop. (March 14--15 , 2013 ). Retrieved from http:\/\/bit.ly\/15rK5tD. Susan T. Dumais. 2013. Task-based search: A search engine perspective. Talk at NSF Task-Based Information Search Systems Workshop. (March 14--15, 2013). Retrieved from http:\/\/bit.ly\/15rK5tD."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544865"},{"key":"e_1_2_1_18_1","first-page":"138","article-title":"SearchBuddies: Bringing search engines into the conversation","volume":"12","author":"Hecht Brent","year":"2012","unstructured":"Brent Hecht , Jaime Teevan , Meredith Ringel Morris , and Daniel J Liebling . 2012 . SearchBuddies: Bringing search engines into the conversation . In Proc. ICWSM , Vol. 12. 138 -- 145 . Brent Hecht, Jaime Teevan, Meredith Ringel Morris, and Daniel J Liebling. 2012. SearchBuddies: Bringing search engines into the conversation. In Proc. ICWSM, Vol. 12. 138--145.","journal-title":"Proc. ICWSM"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518820"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(99)00056-4"},{"key":"e_1_2_1_22_1","volume-title":"Proc. ICWSM.","author":"Jeong Jin-Woo","year":"2013","unstructured":"Jin-Woo Jeong , Meredith R. Morris , Jaime Teevan , and Daniel Liebling . 2013 . A crowd-powered socially embedded search engine . In Proc. ICWSM. Jin-Woo Jeong, Meredith R. Morris, Jaime Teevan, and Daniel Liebling. 2013. A crowd-powered socially embedded search engine. In Proc. ICWSM."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860449"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009947"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572038"},{"key":"e_1_2_1_26_1","volume-title":"Bigham","author":"Lasecki Walter S.","year":"2014","unstructured":"Walter S. Lasecki and Jeffrey P . Bigham . 2014 . Interactive crowds: Real-time crowdsourcing and crowd agents. In Handbook of Human Computation . Walter S. Lasecki and Jeffrey P. Bigham. 2014. Interactive crowds: Real-time crowdsourcing and crowd agents. In Handbook of Human Computation."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383972"},{"key":"e_1_2_1_28_1","volume-title":"Proc. AAAI.","author":"Law Edith","year":"2011","unstructured":"Edith Law and H Zhang . 2011 . Towards large-scale collaborative planning: Answering high-level search queries using human computation . In Proc. AAAI. Edith Law and H Zhang. 2011. Towards large-scale collaborative planning: Answering high-level search queries using human computation. In Proc. AAAI."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470730"},{"key":"e_1_2_1_30_1","volume-title":"Miller","author":"Marcus Adam","year":"2011","unstructured":"Adam Marcus , Eugene Wu , Samuel Madden , and Robert C . Miller . 2011 . Crowdsourced databases: Query processing with people. In Proc. CIDR. CIDR , 211--214. Adam Marcus, Eugene Wu, Samuel Madden, and Robert C. Miller. 2011. Crowdsourced databases: Query processing with people. In Proc. CIDR. CIDR, 211--214."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2594510"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1963405.1963511"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807251"},{"key":"e_1_2_1_35_1","unstructured":"Eric Schurman and Jake Brutlag. 2009. Performance related changes and their user impact. Velocity. (2009). http:\/\/oreil.ly\/fTmYwz. Eric Schurman and Jake Brutlag. 2009. Performance related changes and their user impact. Velocity. (2009). http:\/\/oreil.ly\/fTmYwz."},{"key":"e_1_2_1_36_1","volume-title":"Proc. HCIR. ACM","author":"Teevan Jaime","year":"2013","unstructured":"Jaime Teevan , Kevyn Collins-Thompson , Ryen White , Susan Dumais , and Yubin Kim . 2013 . Slow search or: How search engines can learn to stop hurrying and take their time . In Proc. HCIR. ACM , New York, NY. Jaime Teevan, Kevyn Collins-Thompson, Ryen White, Susan Dumais, and Yubin Kim. 2013. Slow search or: How search engines can learn to stop hurrying and take their time. In Proc. HCIR. ACM, New York, NY."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2633041"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2559206.2581181"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1814433.1814443"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207708"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2897368","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,31]],"date-time":"2022-12-31T06:51:22Z","timestamp":1672469482000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897368"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,12]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,14]]}},"alternative-id":["10.1145\/2897368"],"URL":"https:\/\/doi.org\/10.1145\/2897368","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,7,12]]},"assertion":[{"value":"2015-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-07-12","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}