{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T21:22:24Z","timestamp":1740172944664,"version":"3.37.3"},"reference-count":79,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2015,10,22]],"date-time":"2015-10-22T00:00:00Z","timestamp":1445472000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Asso for Info Science & Tech"],"published-print":{"date-parts":[[2016,1]]},"abstract":"Tools for automatic subject assignment help deal with scale and sustainability in creating and enriching metadata, establishing more connections across and between resources and enhancing consistency. Although some software vendors and experimental researchers claim the tools can replace manual subject indexing, hard scientific evidence of their performance in operating information environments is scarce. A major reason for this is that research is usually conducted in laboratory conditions, excluding the complexities of real\u2010life systems and situations. The article reviews and discusses issues with existing evaluation approaches such as problems of aboutness and relevance assessments, implying the need to use more than a single \u201cgold standard\u201d method when evaluating indexing and retrieval, and proposes a comprehensive evaluation framework. The framework is informed by a systematic review of the literature on evaluation approaches: evaluating indexing quality directly through assessment by an evaluator or through comparison with a gold standard, evaluating the quality of computer\u2010assisted indexing directly in the context of an indexing workflow, and evaluating indexing quality indirectly through analyzing retrieval performance.<\/jats:p>","DOI":"10.1002\/asi.23600","type":"journal-article","created":{"date-parts":[[2015,10,23]],"date-time":"2015-10-23T18:27:03Z","timestamp":1445624823000},"page":"3-16","source":"Crossref","is-referenced-by-count":19,"title":["A framework for evaluating automatic indexing or classification in the context of retrieval"],"prefix":"10.1002","volume":"67","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4169-4777","authenticated-orcid":false,"given":"Koraljka","family":"Golub","sequence":"first","affiliation":[{"name":"Department of Library and Information Science School of Cultural Sciences Faculty of Arts and Humanities Linnaeus University V\u00e4xj\u00f6 351 95 Sweden"}]},{"given":"Dagobert","family":"Soergel","sequence":"additional","affiliation":[{"name":"Department of Library and Information Studies University of Buffalo Room 534 Baldy Buffalo NY 14260\u20101020"}]},{"given":"George","family":"Buchanan","sequence":"additional","affiliation":[{"name":"Department of Computer Science City University Northampton Square London EC1V 0HB United Kingdom"}]},{"given":"Douglas","family":"Tudhope","sequence":"additional","affiliation":[{"name":"Hypermedia Research Group University of South Wales Pontypridd Wales CF37 1DL United Kingdom"}]},{"given":"Marianne","family":"Lykke","sequence":"additional","affiliation":[{"name":"E\u2010Learning Lab University of Aalborg Rensburggade 14 Aalborg 9000 Denmark"}]},{"given":"Debra","family":"Hiom","sequence":"additional","affiliation":[{"name":"IT Services R&D University of Bristol 8\u201010 Berkeley Square Bristol BS8 1HH United Kingdom"}]}],"member":"311","published-online":{"date-parts":[[2015,10,22]]},"reference":[{"key":"e_1_2_9_2_1","unstructured":"20 Newsgroups DataSet. (1998).The 4 universities data set. Retrieved fromhttp:\/\/www\u20102.cs.cmu.edu\/afs\/cs.cmu.edu\/project\/theo\u201020\/www\/data\/news20.html"},{"key":"e_1_2_9_3_1","unstructured":"Albright J.B.(1979).Some limits to subject retrieval from a large published index(Doctoral dissertation Graduate School of Library Science Urbana\u2010Champaign University of Illinois ."},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00046-7"},{"key":"e_1_2_9_5_1","first-page":"268","volume-title":"Proceedings of the 11th World Congress On Medical Informatics","author":"Aronson A.R.","year":"2004"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378889.1378962"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/01655515030294002"},{"key":"e_1_2_9_8_1","first-page":"117","volume-title":"Evaluation Methodologies in Information Retrieval (Dagstuhl Seminar 13441","author":"Belkin N.","year":"2014"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/996350.996360"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10286"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:INRT.0000009438.69013.fa"},{"key":"e_1_2_9_12_1","first-page":"340","volume-title":"Punched cards","author":"Brenner C.W.","year":"1958"},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/345508.345543"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-007-9032-x"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/276675.276682"},{"volume-title":"Factors determining the performance of indexing systems","year":"1968","author":"Cleverdon C.W.","key":"e_1_2_9_16_1"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-6-S1-S12"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630200314"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.35.8.982"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199409)45:8<572::AID-ASI11>3.0.CO;2-X"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1108\/00220410610666501"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1080\/13614560600774313"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1108\/00220410910998915"},{"key":"e_1_2_9_24_1","unstructured":"Hagedorn K.(2001).Extracting value from automated classification tools: The role of manual involvement and controlled vocabularies. Retrieved fromhttp:\/\/argus\u2010acia.com\/white_papers\/classification.html"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.1994.95236136"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2008.11.002"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28997-2_16"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2002.0090001"},{"issue":"2","key":"e_1_2_9_29_1","article-title":"Functional relevance and inductive development of an e\u2010retailing product information typology","volume":"18","author":"Huang X.","year":"2013","journal-title":"Information Research"},{"key":"e_1_2_9_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(95)80034-Q"},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026960"},{"volume-title":"The turn: Integration of information seeking and retrieval in context","year":"2005","author":"Ingwersen P.","key":"e_1_2_9_32_1"},{"key":"e_1_2_9_33_1","unstructured":"International Organization for Standardization. (1985).Documentation\u2014methods for examining documents determining their subjects and selecting indexing terms: ISO 5963\u20101985. Geneva: International Organization for Standardization."},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9205-0"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10137"},{"key":"e_1_2_9_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.11.008"},{"key":"e_1_2_9_37_1","unstructured":"Kim S. &Soergel D.(2005).Selecting and measuring task characteristics as independent variables. In B. Wildemuth M. Crandall & A.S. Grove (Eds.) Proceedings of the 68th Annual Meeting of the American Society for Information Science and Technology(pp.163\u2013186)."},{"key":"e_1_2_9_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273221.1273225"},{"volume-title":"Evaluation of the MEDLARS demand search service","year":"1968","author":"Lancaster F.W.","key":"e_1_2_9_39_1"},{"volume-title":"Indexing and abstracting in theory and practice","year":"2003","author":"Lancaster F.W.","key":"e_1_2_9_40_1"},{"issue":"1","key":"e_1_2_9_41_1","first-page":"361","article-title":"RCV1: A new benchmark collection for text categorization research","volume":"5","author":"Lewis D.D.","year":"2004","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_9_42_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23160"},{"key":"e_1_2_9_43_1","first-page":"87","volume-title":"The Janus faced scholar: A Festschrift in honour of Peter Ingwersen","author":"Lykke M.","year":"2010"},{"key":"e_1_2_9_44_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007095"},{"issue":"2","key":"e_1_2_9_45_1","first-page":"155","article-title":"Interindexer consistency tests: A literature review and report of a test of consistency in indexing visual materials","volume":"6","author":"Markey K.","year":"1984","journal-title":"Library & Information Science Research"},{"key":"e_1_2_9_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1141753.1141819"},{"key":"e_1_2_9_47_1","first-page":"473","volume-title":"Proceedings of the 13th European Conference on Artificial Intelligence","author":"Mladenic D.","year":"1998"},{"volume-title":"Automatic indexing and abstracting of document texts","year":"2000","author":"Moens M.\u2010F.","key":"e_1_2_9_48_1"},{"key":"e_1_2_9_49_1","unstructured":"National Library of Medicine. (2009).Indexing Initiative. Retrieved fromhttp:\/\/ii.nlm.nih.gov\/"},{"key":"e_1_2_9_50_1","unstructured":"National Library of Medicine. (2010).Medical Text Indexer. Retrieved fromhttp:\/\/ii.nlm.nih.gov\/mti.shtml"},{"key":"e_1_2_9_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009002"},{"volume-title":"Subject analysis in online catalogs","year":"2001","author":"Olson H.A.","key":"e_1_2_9_52_1"},{"key":"e_1_2_9_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065385.1065454"},{"issue":"10","key":"e_1_2_9_54_1","first-page":"888","article-title":"An association based method for automatic indexing with a controlled vocabulary","volume":"49","author":"Plaunt C.","year":"1997","journal-title":"Journal of the American Association for Information Science"},{"key":"e_1_2_9_55_1","unstructured":"Polfreman M. &Grace S.(2008).MetaTools stage 1 report: Test framework: A methodology for evaluating metadata generation tools. Retrieved fromhttp:\/\/www.jisc.ac.uk\/media\/documents\/programmes\/reppres\/metatoolsfinalreport.pdf"},{"key":"e_1_2_9_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/1146598.1146660"},{"key":"e_1_2_9_57_1","doi-asserted-by":"publisher","DOI":"10.1002\/1532-2890(2001)9999:9999<::AID-ASI1083>3.0.CO;2-1"},{"key":"e_1_2_9_58_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007030"},{"key":"e_1_2_9_59_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21233"},{"key":"e_1_2_9_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(81)90028-5"},{"key":"e_1_2_9_61_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630220404"},{"key":"e_1_2_9_62_1","unstructured":"Ruiz M.E. &Aronson A.(2007).User\u2010centered evaluation of the Medical Text Indexing (MTI) system. Retrieved fromhttp:\/\/ii.nlm.nih.gov\/Publications\/Papers\/MTIEvaluation\u2010Final.pdf"},{"issue":"1","key":"e_1_2_9_63_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/meet.2008.1450450143","article-title":"Adoption and evaluation issues of automatic and computer aided indexing systems","volume":"45","author":"Ruiz M.E.","year":"2008","journal-title":"Proceedings of the American Society for Information Science and Technology"},{"key":"e_1_2_9_64_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20682"},{"key":"e_1_2_9_65_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20681"},{"key":"e_1_2_9_66_1","doi-asserted-by":"publisher","DOI":"10.1353\/lib.0.0000"},{"key":"e_1_2_9_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"issue":"24","key":"e_1_2_9_68_1","first-page":"76","article-title":"Computer supported indexing: A history and evaluation of NASA's MAI system","volume":"61","author":"Silvester J.P.","year":"1997","journal-title":"Encylopedia of Library and Information Services"},{"volume-title":"Organizing information: Principles of data base and retrieval systems","year":"1985","author":"Soergel D.","key":"e_1_2_9_69_1"},{"key":"e_1_2_9_70_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199409)45:8<589::AID-ASI14>3.0.CO;2-E"},{"key":"e_1_2_9_71_1","unstructured":"Soergel D. &Golub K.(2015).Formal analysis of similarity measures with emphasis on consistency and performance measures in indexing and retrieval. Manuscript in preparation."},{"key":"e_1_2_9_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564433"},{"key":"e_1_2_9_73_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026616"},{"key":"e_1_2_9_74_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-006-6387-3"},{"key":"e_1_2_9_75_1","doi-asserted-by":"publisher","DOI":"10.5771\/0943-7444-2014-1-76"},{"key":"e_1_2_9_76_1","first-page":"30","volume-title":"Proceedings of the 12th International Conference on Electronic Publishing","author":"Tonkin E.","year":"2008"},{"key":"e_1_2_9_77_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2004.11.001"},{"key":"e_1_2_9_78_1","doi-asserted-by":"crossref","unstructured":"Urbano J. Marrero M. &Mart\u00edn D.(2013).On the measurement of test collection reliability. InG.J.F.Jones P.Sheridan D.Kelly M.deRijke &T.Sakai(Eds.) Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval(pp.393\u2013402).New York:ACM","DOI":"10.1145\/2484028.2484038"},{"key":"e_1_2_9_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"key":"e_1_2_9_80_1","first-page":"3","volume-title":"TREC: Experiments and evaluation in information retrieval","author":"Voorhees E.M.","year":"2005"}],"container-title":["Journal of the Association for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fasi.23600","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full\/10.1002\/asi.23600","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,23]],"date-time":"2023-09-23T13:59:25Z","timestamp":1695477565000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/asi.23600"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,10,22]]},"references-count":79,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2016,1]]}},"alternative-id":["10.1002\/asi.23600"],"URL":"https:\/\/doi.org\/10.1002\/asi.23600","archive":["Portico"],"relation":{},"ISSN":["2330-1635","2330-1643"],"issn-type":[{"type":"print","value":"2330-1635"},{"type":"electronic","value":"2330-1643"}],"subject":[],"published":{"date-parts":[[2015,10,22]]}}}