{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,12,13]],"date-time":"2024-12-13T23:40:01Z","timestamp":1734133201954,"version":"3.30.2"},"reference-count":94,"publisher":"Wiley","issue":"13","license":[{"start":{"date-parts":[[2003,7,7]],"date-time":"2003-07-07T00:00:00Z","timestamp":1057536000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Am. Soc. Inf. Sci."],"published-print":{"date-parts":[[2003,11]]},"abstract":"Abstract<\/jats:title>The project proposes and tests a comprehensive and systematic model of user evaluation of Web search engines. The project contains two parts. Part I describes the background and the model including a set of criteria and measures, and a method for implementation. It includes a literature review for two periods. The early period (1995\u20131996) portrays the settings for developing the model and the later period (1997\u20132000) places two applications of the model among contemporary evaluation work. Part II presents one of the applications that investigated the evaluation of four major search engines by 36 undergraduates from three academic disciplines. It reports results from statistical analyses of quantitative data for the entire sample and among disciplines, and content analysis of verbal data containing users' reasons for satisfaction. The proposed model aims to provide systematic feedback to engine developers or service providers for system improvement and to generate useful insight for system design and tool choice. The model can be applied to evaluating other compatible information retrieval systems or information retrieval (IR) techniques. It intends to contribute to developing a theory of relevance that goes beyond topicality to include value and usefulness for designing user\u2010oriented information retrieval systems.<\/jats:p>","DOI":"10.1002\/asi.10303","type":"journal-article","created":{"date-parts":[[2003,10,7]],"date-time":"2003-10-07T08:03:09Z","timestamp":1065513789000},"page":"1175-1192","source":"Crossref","is-referenced-by-count":34,"title":["A comprehensive and systematic model of user evaluation of Web search engines: I. Theory and background"],"prefix":"10.1002","volume":"54","author":[{"given":"Louise T.","family":"Su","sequence":"first","affiliation":[]}],"member":"311","published-online":{"date-parts":[[2003,7,7]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.46895\/lis.37.1","article-title":"A measure for evaluating search engines on the World Wide Web: Retrieval test with ESL (Expected Search Length)","volume":"37","author":"Agata T.","year":"1997","journal-title":"Library and Information Science"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1108\/S1876-0562(1996)96"},{"volume-title":"Presearch interview project","year":"1980","author":"Atherton P.A.","key":"e_1_2_9_4_1"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630350205"},{"key":"e_1_2_9_6_1","first-page":"128","volume-title":"Information retrieval experiments","author":"Barraclough E.D.","year":"1981"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb045413"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.5860\/crl_57_06_514"},{"volume-title":"User\u2010oriented evaluation of information systems and services","year":"1990","author":"Bawden D.","key":"e_1_2_9_10_1"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<85::AID-ASI8>3.0.CO;2-Z"},{"key":"e_1_2_9_12_1","first-page":"88","volume-title":"Interaction in information systems: A review of research from document retrieval to knowledge\u2010based system","author":"Belkin N.J.","year":"1985"},{"key":"e_1_2_9_13_1","unstructured":"Bilal D.(1999).Web search engines for children: A comparative study and performance evaluation of Yahooligans! Ask Jeeves for Kids and Super Snooper. Proceedings of the 62nd Annual Meeting of the American Society for Information Science Washington DC 36 70\u201383."},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3166.3197"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(198611)37:6<387::AID-ASI3>3.0.CO;2-8"},{"key":"e_1_2_9_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(89)90042-3"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199510)46:9<663::AID-ASI4>3.0.CO;2-2"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199607)47:7<568::AID-ASI8>3.0.CO;2-S"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007198"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<142::AID-ASI4>3.0.CO;2-6"},{"key":"e_1_2_9_21_1","unstructured":"Brunk B. &Bidgood S.(2000).An empirical comparison of Web overview techniques. Proceedings of the 63rd Annual Meeting of the American Society for Information Science Chicago IL 37 382\u2013391."},{"volume-title":"Internet agents: Spiders, wanderers, brokers, and 'bots","year":"1996","author":"Cheong F.C.","key":"e_1_2_9_22_1"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(98)00065-X"},{"key":"e_1_2_9_24_1","unstructured":"Chu H.T. &Rosenthal M.(1996).Search engines for the World Wide Web: A comparative study and evaluation methodology. Proceedings of the 59th Annual Meeting of the American Society for Information Science Baltimore MD 33 127\u2013135."},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb051463"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026574"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090190108"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630240204"},{"key":"e_1_2_9_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630240603"},{"key":"e_1_2_9_30_1","unstructured":"Courtois M.P. Baer W. &Stark M.(1995 November\/December).Cool tools for searching the Web: A performance evaluation. Online 15\u201332."},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(99)00057-6"},{"key":"e_1_2_9_32_1","unstructured":"Ding W. &Marchionini G.(1996).A comparative study of Web search service performance. Proceedings of the 59th Annual Meeting of the American Society for Information Science Baltimore MD 33 136\u2013142."},{"key":"e_1_2_9_33_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb024608"},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-444-70536-5.50029-4"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630320104"},{"key":"e_1_2_9_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630340302"},{"key":"e_1_2_9_37_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<124::AID-ASI2>3.0.CO;2-8"},{"key":"e_1_2_9_38_1","first-page":"74","article-title":"Meta Search Engines","volume":"23","author":"Garman N.","year":"1999","journal-title":"Online"},{"key":"e_1_2_9_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(98)00041-7"},{"key":"e_1_2_9_40_1","doi-asserted-by":"crossref","unstructured":"Greene S.L. Gomez L.M. &Devlin S.J.(1986).A cognitive analysis of database query production. Proceedings of the Human Factors Society 9\u201313.","DOI":"10.1177\/154193128603000103"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3"},{"key":"e_1_2_9_42_1","series-title":"Annual review of information science and technology","first-page":"3","author":"Harter S.P.","year":"1997"},{"key":"e_1_2_9_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00024-9"},{"key":"e_1_2_9_44_1","doi-asserted-by":"crossref","unstructured":"Hirsh S.G.(1996).The effect of domain knowledge on elementary school children's information retrieval behavior on an automated library catalog. Unpublished doctoral dissertation University of California Los Angeles CA.","DOI":"10.1145\/223355.223424"},{"key":"e_1_2_9_45_1","unstructured":"Hirsh S.G. &Borgman C.L.(1995).Comparing children's use of browsing and keyword searching on the Science Library Catalog. Proceedings of the 58th Annual Meeting of the American Society for Information Science Chicago IL 32 19\u201326."},{"key":"e_1_2_9_46_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<172::AID-ASI7>3.0.CO;2-V"},{"key":"e_1_2_9_47_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199304)44:3<161::AID-ASI5>3.0.CO;2-8"},{"key":"e_1_2_9_48_1","first-page":"281","article-title":"Getting the best from Web search engines","author":"Kassel A.","year":"1998","journal-title":"Online Information"},{"volume-title":"An introduction to the design and analysis of experiments in behavioral research","year":"1985","author":"Kennedy J.J.","key":"e_1_2_9_49_1"},{"key":"e_1_2_9_50_1","doi-asserted-by":"crossref","unstructured":"Koenemann J. Quatrain R. Cool C. &Belkin N.J.(1995).New tools and old habits: The interactive searching behavior of expert online searchers using INQUIRY. Overview of the Third Text Retrieval Conference (TREC\u20103) 145\u2013177.","DOI":"10.6028\/NIST.SP.500-225.rutgers"},{"key":"e_1_2_9_51_1","unstructured":"Koster M.(1996). List of robots [On\u2010line]. Available:http:\/\/info.webcrawler.com\/mak\/projects\/robots\/active.html"},{"key":"e_1_2_9_52_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199106)42:5<361::AID-ASI6>3.0.CO;2-#"},{"key":"e_1_2_9_53_1","doi-asserted-by":"publisher","DOI":"10.1177\/089443939901700405"},{"volume-title":"Evaluation of MEDLARS demand search service","year":"1968","author":"Lancaster F.W.","key":"e_1_2_9_54_1"},{"key":"e_1_2_9_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00095-6"},{"key":"e_1_2_9_56_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.280.5360.98"},{"key":"e_1_2_9_57_1","doi-asserted-by":"publisher","DOI":"10.1038\/21987"},{"key":"e_1_2_9_58_1","unstructured":"Lebedov A.(1996). Best search engines for finding scientific information in the Net [On\u2010line]. Available:http:\/\/www.cnem.msu.su\/eng\/comparison.html"},{"key":"e_1_2_9_59_1","unstructured":"Leighton H.V.(1996). Performance of four World Wide Web index services: Infoseek Lycos Webcrawler and WWW Worm [On\u2010line]. Available:http:\/\/inona.msus.edu\/servicesI\/library\u2010I.weblna.html#A4"},{"key":"e_1_2_9_60_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(1999)50:10<870::AID-ASI4>3.0.CO;2-G"},{"key":"e_1_2_9_61_1","unstructured":"Logan E.(1995).The Internet challenge. In Proceedings of the 16th National Online Meeting New York NY 285\u2013290."},{"key":"e_1_2_9_62_1","unstructured":"Marchionini G. Brunk B. Komolodi A. Conrad F. &Bosley J.(2000).Look before you click: A relational browser for federal statistics Websites. Proceedings of the 63rd Annual Meeting of the American Society for Information Science Chicago IL 37 392\u2013403."},{"key":"e_1_2_9_63_1","first-page":"591","article-title":"User\u2010based data collection techniques and strategies for evaluating networked information services","volume":"4","author":"McClure C.R.","year":"1994","journal-title":"Library Trends"},{"key":"e_1_2_9_64_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007653114902"},{"key":"e_1_2_9_65_1","unstructured":"NPD Media Service(2000 July 13). Google's success story continues: Google ranked No. 1 in loyalty and satisfaction measures by search engine users in independent survey. BusinessWire. [On\u2010line]. Available:http:\/\/www.businesswire.com"},{"key":"e_1_2_9_66_1","first-page":"48","article-title":"On the Nets: Searching the Worldwide Web: Lycos, WebCrawler and more","author":"Notess G.R.","year":"1995","journal-title":"Online"},{"key":"e_1_2_9_67_1","first-page":"85","article-title":"On the Nets: The Infoseek databases","volume":"18","author":"Notess G.R.","year":"1995","journal-title":"Database"},{"key":"e_1_2_9_68_1","first-page":"33","article-title":"On the Nets: Internet directions in 1996","volume":"20","author":"Notess G.R.","year":"1996","journal-title":"Online"},{"key":"e_1_2_9_69_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb024557"},{"key":"e_1_2_9_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-9236(99)00054-8"},{"key":"e_1_2_9_71_1","first-page":"23","volume-title":"Education for information science","author":"Paisley W.J.","year":"1965"},{"key":"e_1_2_9_72_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<135::AID-ASI3>3.0.CO;2-1"},{"key":"e_1_2_9_73_1","unstructured":"Rieh S.Y. &Belkin N.J.(1998).Understanding judgment of information quality and cognitive authority in the WWW. Proceedings of the 61st Annual Meeting of the American Society for Information Science Pittsburgh PA 35 279\u2013290."},{"key":"e_1_2_9_74_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630260604"},{"key":"e_1_2_9_75_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(198805)39:3<161::AID-ASI2>3.0.CO;2-0"},{"key":"e_1_2_9_76_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(198805)39:3<177::AID-ASI3>3.0.CO;2-F"},{"key":"e_1_2_9_77_1","unstructured":"Saracevic T. &Su L.T.(1989).Modeling and measuring user\u2010intermediary\u2010computer interaction in online searching: Design of a study. Proceedings of the 52nd Annual Meeting of the American Society for Information Science Washington DC 26 75\u201380."},{"key":"e_1_2_9_78_1","first-page":"132","article-title":"Diversity on the World Wide Web: Using robots to search the Web","volume":"14","author":"Scales B.J.","year":"1995","journal-title":"Library Software Review"},{"key":"e_1_2_9_79_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199306)44:5<273::AID-ASI3>3.0.CO;2-X"},{"volume-title":"Information retrieval experiments","year":"1981","author":"Sparck Jones K.","key":"e_1_2_9_80_1"},{"key":"e_1_2_9_81_1","unstructured":"Su L.T.(1991).An investigation to find appropriate measures for evaluating interactive information retrieval. Doctoral dissertation Rutgers University New Brunswick NJ."},{"key":"e_1_2_9_82_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(92)90007-M"},{"key":"e_1_2_9_83_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<207::AID-ASI10>3.0.CO;2-1"},{"key":"e_1_2_9_84_1","unstructured":"Su L.T.(1997).Developing a comprehensive and systematic model of user evaluation of Web\u2010based search engines.Proceedings of the 18th National Online Meeting New York NY 335\u2013344."},{"key":"e_1_2_9_85_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(98)00023-5"},{"key":"e_1_2_9_86_1","unstructured":"Su L.T. Chen H. &Dong X.(1998).Evaluation of Web\u2010based search engines from the end\u2010user's perspective: A pilot study. Proceedings of the 61st American Society for Information Science Pittsburgh PA 35 348\u2013361."},{"key":"e_1_2_9_87_1","unstructured":"Su L.T. &Chen H.(1999).Evaluation of Web search engines by undergraduate students. Proceedings of the 62nd American Society for Information Science Washington DC 36 98\u2013114."},{"key":"e_1_2_9_88_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<186::AID-ASI8>3.0.CO;2-F"},{"key":"e_1_2_9_89_1","first-page":"243","article-title":"Estimating the satisfaction of information users","volume":"65","author":"Tagliacozzo R.","year":"1977","journal-title":"Bulletin of the Medical Library Association"},{"key":"e_1_2_9_90_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<1::AID-ASI1>3.0.CO;2-3"},{"volume-title":"Satisfaction measures in presearch interviews. A report of the presearch interview project","year":"1981","author":"Tessier J.","key":"e_1_2_9_91_1"},{"key":"e_1_2_9_92_1","first-page":"383","article-title":"New measures of user satisfaction with computer\u2010based literature searches","volume":"68","author":"Tessier J.","year":"1977","journal-title":"Special Libraries"},{"key":"e_1_2_9_93_1","doi-asserted-by":"publisher","DOI":"10.1177\/016555159902500606"},{"volume-title":"Statistical principles in experimental design","year":"1991","author":"Winer B.J.","key":"e_1_2_9_94_1"},{"key":"e_1_2_9_95_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199503)46:2<133::AID-ASI6>3.0.CO;2-Z"}],"container-title":["Journal of the American Society for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fasi.10303","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/asi.10303","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,13]],"date-time":"2024-12-13T23:09:00Z","timestamp":1734131340000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/asi.10303"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2003,7,7]]},"references-count":94,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2003,11]]}},"alternative-id":["10.1002\/asi.10303"],"URL":"https:\/\/doi.org\/10.1002\/asi.10303","archive":["Portico"],"relation":{},"ISSN":["1532-2882","1532-2890"],"issn-type":[{"type":"print","value":"1532-2882"},{"type":"electronic","value":"1532-2890"}],"subject":[],"published":{"date-parts":[[2003,7,7]]}}}