Beyond Topicality: Including Multidimensional Relevance in Cross-encoder Re-ranking | SpringerLink
Skip to main content
Springer Nature Link
Log in

Beyond Topicality: Including Multidimensional Relevance in Cross-encoder Re-ranking

The Health Misinformation Case Study

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14608))

Included in the following conference series:

Abstract

In this paper, we propose a novel approach to consider multiple dimensions of relevance in cross-encoder re-ranking. On the one hand, cross-encoders constitute an effective solution for re-ranking when considering a single relevance dimension such as topicality, but are not designed to straightforwardly account for additional relevance dimensions. On the other hand, the majority of re-ranking models accounting for multdimensional relevance are often based on the aggregation of multiple relevance scores at the re-ranking stage, leading to potential compensatory effects. To address these issues, in the proposed solution we enhance the candidate documents retrieved by a first-stage lexical retrieval model with suitable relevance statements related to distinct relevance dimensions, and then perform a re-ranking on them with cross-encoders. In this work we focus, in particular, on an extra dimension of relevance beyond topicality, namely, credibility, to address health misinformation in the Consumer Health Search task. Experimental evaluations are performed by considering publicly available datasets; our results show that the proposed approach statistically outperforms state-of-the-art aggregation-based and cross-encoder re-rankers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9380
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11725
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    I.e., open-source scientific articles extracted from reputed and trustworthy medical journals such as the Journal of the American Medical Association (JAMA) and eLife. .

  2. 2.

    https://huggingface.co/dmis-lab/biobert-v1.1.

References

  1. Abualsaud, M., et al.: Uwaterloomds at the TREC 2021 health misinformation track. In: Proceedings of the Thirtieth Retrieval Conference Proceedings (TREC 2021), pp. 1–18. National Institute of Standards and Technology (NIST), Special Publication (2021)

    Google Scholar 

  2. Abualsaud, M., Lioma, C., Maistro, M., Smucker, M.D., Guido, Z.: Overview of the TREC 2019 decision track (2020). https://api.semanticscholar.org/CorpusID:221857114

  3. Al-Hajj, M., Jarrar, M.: Arabglossbert: fine-tuning bert on context-gloss pairs for WSD. arXiv preprint arXiv:2205.09685 (2022)

  4. Anand, M., Zhang, J., Ding, S., Xin, J., Lin, J.: Serverless bm25 search and bert reranking. In: DESIRES, pp. 3–9 (2021)

    Google Scholar 

  5. Askari, A., Abolghasemi, A., Pasi, G., Kraaij, W., Verberne, S.: Injecting the bm25 score as text improves bert-based re-rankers. arXiv preprint arXiv:2301.09728 (2023)

  6. Aslam, J.A., Montague, M.: Models for metasearch. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 276–284 (2001)

    Google Scholar 

  7. Bondarenko, A., et al.: Webis at TREC 2021: deep learning, health misinformation, and podcasts tracks. In: The Thirtieth Retrieval Conference Proceedings (TREC 2021), pp. 500, 335 (2021)

    Google Scholar 

  8. Boualili, L., Moreno, J.G., Boughanem, M.: Markedbert: integrating traditional IR cues in pre-trained language models for passage retrieval. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), pp. 1977–1980. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3397271.3401194

  9. Boualili, L., Moreno, J.G., Boughanem, M.: Highlighting exact matching via marking strategies for ad hoc document ranking with pretrained contextualized language models. Inf. Retriev. J. 25(4), 414–460 (2022). https://doi.org/10.1007/s10791-022-09414-x

  10. Chen, Y., et al.: Cdevalsumm: an empirical study of cross-dataset evaluation for neural summarization systems. arXiv preprint arXiv:2010.05139 (2020)

  11. Clarke, C.L.A., Maistro, M., Rizvi, S., Smucker, M.D., Zuccon, G.: Overview of the TREC 2020 health misinformation track (2020). https://trec.nist.gov/pubs/trec29/papers/OVERVIEW.HM.pdf

  12. Clarke, C.L.A., Maistro, M., Seifikar, M., Smucker, M.D.: Overview of the TREC 2022 health misinformation track. In: 30th Retrieval Conference, TREC 2021, vol. 500, 338, pp. 15–19. Gaithersburg, Maryland (2021)

    Google Scholar 

  13. Clarke, C.L.A., Rizvi, S., Smucker, M.D., Maistro, M., Zuccon, G.: Overview of the TREC 2021 health misinformation track. In: Text Retrieval Conference (2021). https://api.semanticscholar.org/CorpusID:235600234

  14. Cong, G., Jensen, C.S., Wu, D.: Efficient retrieval of the top-k most relevant spatial web objects. Proc. VLDB Endowm. 2(1), 337–348 (2009)

    Article  Google Scholar 

  15. Cormack, G.V., Clarke, C.L., Buettcher, S.: Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 758–759 (2009)

    Google Scholar 

  16. da Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensional relevance: a new aggregation criterion. In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds.) Advances in Information Retrieval. ECIR 2009. LNCS, vol. 5478, pp. 264–275. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-00958-7_25

  17. da Costa Pereira, C., Dragoni, M., Pasi, G.: A prioritized “and” aggregation operator for multidimensional relevance assessment. In: Serra, R., Cucchiara, R. (eds.) AI*IA 2009: Emergent Perspectives in Artificial Intelligence. AI*IA 2009. LNCS, vol. 5883, pp. 72–81. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10291-2_8

  18. da Costa Pereira, C., Dragoni, M., Pasi, G.: Multidimensional relevance: prioritized aggregation in a personalized information retrieval setting. Inf. Process. Manag. 48(2), 340–357 (2012). https://doi.org/10.1016/j.ipm.2011.07.001

  19. Daoud, M., Tamine, L., Boughanem, M.: A personalized graph-based document ranking model using a semantic user profile. In: De Bra, P., Kobsa, A., Chin, D. (eds.) User Modeling, Adaptation, and Personalization. UMAP 2010. LNCS, vol. 6075, pp. 171–182. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13470-8_17

  20. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis (2019). https://doi.org/10.18653/v1/N19-1423

  21. Fernández-Pichel, M., Losada, D.E., Pichel, J.C.: A multistage retrieval system for health-related misinformation detection. Eng. Appl. Artif. Intell. 115, 105211 (2022), https://api.semanticscholar.org/CorpusID:250932569

  22. Fernández-Pichel, M., Losada, D.E., Pichel, J.C., Elsweiler, D.: Citius at the trec 2020 health misinformation track. In: TREC (2020)

    Google Scholar 

  23. Fox, E.A.: Combination of multiple searches. In: Proceedings of the Second Text Retrieval Conference, August/September 1993 (1993)

    Google Scholar 

  24. Fox, E.A., Koushik, M.P., Shaw, J., Modlin, R., Rao, D., et al.: Combining evidence from multiple searches. In: The First Text Retrieval Conference (TREC-1), pp. 319–328 (1993)

    Google Scholar 

  25. Gao, L., Dai, Z., Chen, T., Fan, Z., Van Durme, B., Callan, J.:. Complement lexical retrieval model with semantic residual embeddings. In: Hiemstra, D., Moens, MF., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2021. LNCS, vol. 12656, pp. 146–160. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72113-8_10

  26. Goeuriot, L., et al.: Clef 2017 ehealth evaluation lab overview. In: Conference and Labs of the Evaluation Forum (2017). https://api.semanticscholar.org/CorpusID:206705118

  27. Goeuriot, L., et al.: CLEF eHealth evaluation lab 2021. In: Hiemstra, D., Moens, M.-F., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds.) Advances in Information Retrieval, ECIR 2021. LNCS, vol. 12657, pp. 593–600. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72240-1_69

  28. Goeuriot, L., et al.: Overview of the clef ehealth 2020 task 2: consumer health search with ad hoc and spoken queries. In: Conference and Labs of the Evaluation Forum (2020). https://api.semanticscholar.org/CorpusID:225073918

  29. Goeuriot, L., et al.: Overview of the clef ehealth evaluation lab 2020. In: Arampatzis, A., et al. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. LNCS, vol. 12260, pp. 255–271. Springer, Cham (2020)

    Google Scholar 

  30. Huang, Y., Xu, Q., Wu, S., Nugent, C., Moore, A.: Fight against covid-19 misinformation via clustering-based subset selection fusion methods. In: ROMCIR 2022 CEUR Workshop Proceedings, vol. 3138, pp. 11–26 (2022)

    Google Scholar 

  31. Kamphuis, C., de Vries, A.P., Boytsov, L., Lin, J.: Which bm25 do you mean? a large-scale reproducibility study of scoring variants. In: Jose, J.M., et al. (eds.) Advances in Information Retrieval. LNCS, vol. 12036, pp. 28–34. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45442-5_4

  32. Khattab, O., Zaharia, M.: Colbert: efficient and effective passage search via contextualized late interaction over bert. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 39–48 (2020)

    Google Scholar 

  33. Lee, J., et al.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)

    Google Scholar 

  34. Li, L., et al.: Markbert: marking word boundaries improves chinese bert. arXiv preprint arXiv:2203.06378 (2022)

  35. Lima, L.C., Wright, D.B., Augenstein, I., Maistro, M.: University of copenhagen participation in TREC health misinformation track 2020. arXiv preprint arXiv:2103.02462 (2021)

  36. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 30 (2017)

    Google Scholar 

  37. Macdonald, C., Tonellotto, N., MacAvaney, S., Ounis, I.: Pyterrier: declarative experimentation in python from bm25 to dense retrieval. In: Proceedings of the 30th ACM International Conference on Information Knowledge Management (CIKM 2021), pp. 4526–4533. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3459637.3482013

  38. Moulahi, B., Tamine, L., Yahia, S.B.: i a ggregator: multidimensional relevance aggregation based on a fuzzy operator. J. Am. Soc. Inf. Sci. 65(10), 2062–2083 (2014)

    Google Scholar 

  39. Nguyen, M., Kishan, K., Nguyen, T., Chadha, A., Vu, T.: Efficient fine-tuning large language models for knowledge-aware response planning. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds.) Joint European Conference on Machine Learning and Knowledge Discovery in Databases. LNCS, vol. 14170, pp. 593–611. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43415-0_35

  40. Nogueira, R., Cho, K.: Passage re-ranking with bert. arXiv preprint arXiv:1901.04085 (2019)

  41. Paszke, A., et al.: PyTorch: An Imperative Style, High-Performance Deep Learning Library, p. 12. Curran Associates Inc., Red Hook (2019)

    Google Scholar 

  42. Pradeep, R., Ma, X., Nogueira, R., Lin, J.J., Cheriton, D.R.: Vera: prediction techniques for reducing harmful misinformation in consumer health search. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (2021). https://api.semanticscholar.org/CorpusID:235477259

  43. Pradeep, R., et al.: H2oloo at TREC 2020: when all you got is a hammer... deep learning, health misinformation, and precision medicine. Corpus 5(d3), d2 (2020)

    Google Scholar 

  44. Putri, D.G.P., Viviani, M., Pasi, G.: Social search and task-related relevance dimensions in microblogging sites. In: Aref, S., et al. (eds.) Social Informatics (SocInfo 2020). LNCS, vol. 12467, pp. 297–311. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60975-7_22

  45. Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (2019). https://arxiv.org/abs/1908.10084

  46. Ren, R., et al.: Rocketqav2: a joint training method for dense passage retrieval and passage re-ranking. arXiv preprint arXiv:2110.07367 (2021)

  47. Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Croft, B.W., van Rijsbergen, C.J. (eds.) SIGIR 1994, pp. 232–241. Springer, Heidelberg (1994). https://doi.org/10.1007/978-1-4471-2099-5_24

  48. Robertson, S., Zaragoza, H., et al.: The probabilistic relevance framework: Bm25 and beyond. Found. Trends® Inf. Retriev. 3(4), 333–389 (2009)

    Google Scholar 

  49. Schlicht, I.B., de Paula, A.F.M., Rosso, P.: UPV at TREC health misinformation track 2021 ranking with SBERT and quality estimators. arXiv preprint arXiv:2112.06080 (2021)

  50. Upadhyay, R., Pasi, G., Viviani, M. (2022). An unsupervised approach to genuine health information retrieval based on scientific evidence. In: Chbeir, R., Huang, H., Silvestri, F., Manolopoulos, Y., Zhang, Y. (eds.) Web Information Systems Engineering (WISE 2022). LNCS, vol. 13724, pp. 119–135. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20891-1_10

  51. Van Opijnen, M., Santos, C.: On the concept of relevance in legal information retrieval. Artif. Intell. Law 25, 65–87 (2017)

    Article  Google Scholar 

  52. Viviani, M., Pasi, G.: Credibility in social media: opinions, news, and health information-a survey. Wiley Interdiscip. Rev.: Data Mining Knowl. Discov. 7(5), e1209 (2017)

    Google Scholar 

  53. Wallace, E., Wang, Y., Li, S., Singh, S., Gardner, M.: Do NLP models know numbers? probing numeracy in embeddings. arXiv preprint arXiv:1909.07940 (2019)

  54. Weisstein, E.W.: Bonferroni correction (2004). https://mathworld.wolfram.com/

  55. Wolf, T., et al.: Huggingface’s transformers: state-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)

  56. Zhang, B., Naderi, N., Jaume-Santero, F., Teodoro, D.: Ds4dh at TREC health misinformation 2021: multi-dimensional ranking models with transfer learning and rank fusion. arXiv preprint arXiv:2202.06771 (2022)

  57. Zhang, B., Naderi, N., Mishra, R., Teodoro, D.: Improving online health search via multi-dimensional information quality models based on deep learning. medRxiv, pp. 2023–04 (2023)

    Google Scholar 

  58. Zhang, D., Vakili Tahami, A., Abualsaud, M., Smucker, M.D.: Learning trustworthy web sources to derive correct answers and reduce health misinformation in search. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2099–2104 (2022)

    Google Scholar 

  59. Zhuang, S., Zuccon, G.: Tilde: term independent likelihood model for passage re-ranking. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1483–1492 (2021)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the EU Horizon 2020 ITN/ETN on Domain Specific Systems for Information Extraction and Retrieval (DoSSIER), H2020-EU.1.3.1., ID: 860721, https://dossier-project.eu/.

Author information

Authors and Affiliations

  1. Università degli Studi di Milano-Bicocca, Department of Informatics, Systems, and Communication, Viale Sarca 336, 20126, Milan, Italy

    Rishabh Upadhyay, Gabriella Pasi & Marco Viviani

  2. Universiteit Leiden, Leiden Institute of Advanced Computer Science, Snellius Gebouw, Niels Bohrweg 1, 2333 CA, Leiden, The Netherlands

    Arian Askari

Authors
  1. Rishabh Upadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arian Askari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gabriella Pasi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marco Viviani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marco Viviani .

Editor information

Editors and Affiliations

  1. Georgetown University, Washington, WA, USA

    Nazli Goharian

  2. University of Pisa, Pisa, Italy

    Nicola Tonellotto

  3. King's College London, London, UK

    Yulan He

  4. University College London, London, UK

    Aldo Lipani

  5. University of Glasgow, Glasgow, UK

    Graham McDonald

  6. University of Glasgow, Glasgow, UK

    Craig Macdonald

  7. University of Glasgow, Glasgow, UK

    Iadh Ounis

Ethics declarations

Data and Code Availability

The data used in this article are publicly accessible at: https://github.com/ikr3-lab/TREC-CLEF-HealthMisinfoSubdatasets. The code for the implementation and evaluation of the proposed model is publicly accessible at: https://github.com/ikr3-lab/Multidimensional-Cross-Encoder-Reranking.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Upadhyay, R., Askari, A., Pasi, G., Viviani, M. (2024). Beyond Topicality: Including Multidimensional Relevance in Cross-encoder Re-ranking. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14608. Springer, Cham. https://doi.org/10.1007/978-3-031-56027-9_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56027-9_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56026-2

  • Online ISBN: 978-3-031-56027-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation