Abstract
When searching for health information, results quality can be judged against available scientific evidence: Do search engines return advice consistent with evidence based medicine? We compared the performance of domain-specific health and depression search engines against a general-purpose engine (Google) on both relevance of results and quality of advice. Over 101 queries, to which the term ‘depression’ was added if not already present, Google returned more relevant results than those of the domain-specific engines. However, over the 50 treatment-related queries, Google returned 70 pages recommending for or against a well studied treatment, of which 19 strongly disagreed with the scientific evidence. A domain-specific index of 4 sites selected by domain experts was only wrong in 5 of 50 recommendations. Analysis suggests a tension between relevance and quality. Indexing more pages can give a greater number of relevant results, but selective inclusion can give better quality.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aggarwal CC, Al-Garawi F and Yu PS (2001) On the design of a learning crawler for topical resource discovery. ACM Trans. Inf. Syst., 19(3):286–309
Andrews G, Issakidis C and Andrews G, Issakidis C and Carter G (2001) Shortfall in mental health service utilisation. British Journal of Psychiatry, 179:417–25
American Psychiatric Association (1994) Diagnostic and statistical manual of mental disorders. Author, Washington, DC, 4th edition.
Baker L, Wagner TH, Singer S and Bundorf KM (2003). Use of the internet and e-mail for health care information. Journal of the American Medical Association, 289(18):2400–2406
Bin L and Lun KC (2001) The retrieval effectiveness of medical information on the web, International Journal of Medical Informatics, 62:155–163
Chakrabarti S, Berg M and Dom B (1999) Focused crawling: A new approach to topic-specific web resource discovery. In Proceeding of the 8th International World Wide Web Conference (WWW8).
Cho J, Garcia-Molina H and Page L (1998) Efficient crawling through url ordering. In Proceeding of the Seventh World Wide Web Conference.
Christensen H, Griffiths KM and Jorm AF (2004) Delivering interventions for depression by using the internet: Randomised controlled trial, BMJ, 328(7434):265–0
Clarke M and Oxman AD (2001)Cochrane Reviewers’ handbook 4.1.1. The Cochrane Library, Oxford.
Eysebach G and Kohler C (2002) How do consumers search for and appraise health information on the world wide web? qualitative study using focus groups, usability tests, and in-depth interviews. BMJ, 324:573-577. http://bmj.com.
Gretchen BK, Elliot MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, Kanouse DE, Munoz JA, Puyol JA, Lara M, Watkins KE, Yang H and McGlynn EA (2001) Health information on the internet: accessibility, quality, and readability in English and Spanish. JAMA, 285(20):2612–2621
Griffiths KM and Christensen H (2002) The quality and accessibility of australian depression sites on the world wide web. MJA, 176:S97–S104
Hawking D, Craswell N, Bailey P and Griffiths K (2001) Measuring search engine quality. Information Retrieval, http://es.cmis.csiro.au/pubs/hawking_ir01.pdf, 4(1):33–59
Hersovici M, Jacovi M, Maarek YS, Pelleg D, Shtalhaim M and Ur S (1998) The shark-search algorithm, an application: Tailored web site mapping. In Proceeding of the Seventh World Wide Web Conference.
Ilic D, Bessell TL, Silagy CA and Green S (2003) Specialized medical search-engines are no better than general search-engines in sourcing consumer information about androgen deficiency. Human Reproduction, 18(3):557–561
Jarvelin K and Kekalainen J (2002) Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422–446
Jorm AF, Christensen H, Griffiths KM, Korten A and Rodgers B (2001) Help for depression: What works (and what doesn’t). Centre for Mental Health Research, Canberra, Australia.
McCallum A, Nigam K, Rennie J and Sey-more K (1999) Building domain-specific search engines with machine learning technique. In Proceedings of AAAI Spring Symposium on Intelligents Engine in Cyberspace.
Menczer F, Pant G and Srinivasan P (2001) Evaluating topic-driven web crawlers. In Proceeding of the 24th Annual Intl. ACM SIGIR Conf. On Research and Development in Information Retrieval.
Mukherjea S (2000). Organizing topic-specific web information. In Proceedings of the eleventh ACM on Hypertext and hypermedia, ACM Press pp. 133–141.
Murray CJL and Lopez AD (1996) (eds.) The global burden of disease and injury series. Harvard University Press, Cambridge MA.
Sellen AJ, Murphy R and Shaw KL (2002). How knowledge workers use the web. In: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press pp. 227–234.
Sormunen E (2000) A method for measuring wide range performance of Boolean queries in full-text databases. PhD thesis, University of Tampere. http://acta.uta.fi/teos.phtml?3786.
Spink A, Jansen BJ, Wolfram D and Saracevic T (2002) From e-sex to e-commerce: Web search changes. Computer, 35(3):107–109
Wu G and Li J (1999) Comparing web search engine performance in searching consumer health information: Evaluation and recommendations, Bull Med Libr Assoc, 87(4):456–461
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tang, T.T., Craswell, N., Hawking, D. et al. Quality and relevance of domain-specific search: A case study in mental health. Inf Retrieval 9, 207–225 (2006). https://doi.org/10.1007/s10791-006-7150-5
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10791-006-7150-5