Metrics for Evaluating Explainable Recommender Systems | SpringerLink
Skip to main content
Springer Nature Link
Log in

Metrics for Evaluating Explainable Recommender Systems

  • Conference paper
  • First Online:
Explainable and Transparent AI and Multi-Agent Systems (EXTRAAMAS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14127))

Abstract

Recommender systems aim to support their users by reducing information overload so that they can make better decisions. Recommender systems must be transparent, so users can form mental models about the system’s goals, internal state, and capabilities, that are in line with their actual design. Explanations and transparent behaviour of the system should inspire trust and, ultimately, lead to more persuasive recommendations. Here, explanations convey reasons why a recommendation is given or how the system forms its recommendations. This paper focuses on the question how such claims about effectiveness of explanations can be evaluated. Accordingly, we investigate various models that are used to assess the effects of explanations and recommendations. We discuss objective and subjective measurement and argue that both are needed. We define a set of metrics for measuring the effectiveness of explanations and recommendations. The feasibility of using these metrics is discussed in the context of a specific explainable recommender system in the food and health domain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 6634
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 8293
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 734–749 (2005)

    Article  Google Scholar 

  2. Anjomshoae, S., Calvaresi, D., Najjar, A., Främling, K.: Explainable agents and robots: results from a systematic literature review. In: Autonomous Agents and Multi Agent Systems (AAMAS 2019), pp. 1078–1088 (2019)

    Google Scholar 

  3. Atkinson, K., Bench-Capon, T., McBurney, P.: Computational representation of practical argument. Synthese 152(2), 157–206 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bernstein, E.: Making transparency transparent: the evolution of observation in management theory. Acad. Manag. Ann. 11(1), 217–266 (2017)

    Article  Google Scholar 

  5. Burke, R., Felfernig, A., Göker, M.H.: Recommender systems: an overview. AI Mag. 32, 13–18 (2011)

    Google Scholar 

  6. Buzcu, B., Varadhajaran, V., Tchappi, I.H., Najjar, A., Calvaresi, D., Aydoğan, R.: Explanation-based negotiation protocol for nutrition virtual coaching. In: PRIMA 2022. LNCS, vol. 13753, pp. 20–36. Springer (2022). https://doi.org/10.1007/978-3-031-21203-1_2

  7. Calvaresi, D.: Ethical and legal considerations for nutrition virtual coaches. In: AI and Ethics, pp. 1–28 (2022)

    Google Scholar 

  8. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13(3), 319–340 (1989)

    Article  Google Scholar 

  9. V. Dignum. Responsible Artificial Intelligence: How to Develop and Use AI in a Responsible Way. Springer (2019). https://doi.org/10.1007/978-3-030-30371-6

  10. European Commission. Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain union legislative acts (2021)

    Google Scholar 

  11. Falcone, R., Castelfranchi, C.: Trust and relational capital. Comput. Math. Organ. Theory 17(2), 179–195 (2011)

    Article  Google Scholar 

  12. Goodhue, D.L.: Understanding user evaluations of information systems. Manage. Sci. 41(12), 1827–1844 (1995)

    Article  Google Scholar 

  13. Grice, H.P.: Logic and conversation. In: Cole, P., Morgan, J.L. (eds.) Syntax and Semantics, vol. 3, pp. 41–58. Academic Press, New York (1975)

    Google Scholar 

  14. HLEG. Ethics guidelines for trustworthy AI (2019)

    Google Scholar 

  15. Hoffman, R.R., Mueller, S.T., Klein, G., Litman, O.: Metrics for explainable ai: challenges and prospects. arXiv:1812.04608 [cs.AI] (2018)

  16. Jannach, D., Pearl, P., Ricci, F., Zanker, M.: Recommender systems: past, present, future. AI Mag. 42, 3–6 (2021)

    Google Scholar 

  17. Kriz, S., Ferro, T.D., Damera, P., Porter, J.R.: Fictional Robots as a Data Source in HRI Research, pp. 458–463. IEEE (2010)

    Google Scholar 

  18. Lewicki, R.J., Bunker, B.B.: Developing and maintaining trust in work relationships. In: Trust in Organizations, pp. 114–139. Sage Publications (1996)

    Google Scholar 

  19. Lewis, D.: Causal explanation, pp. 214–240. Oxford University Press, Oxford (1986)

    Google Scholar 

  20. Lewis, J.R., Sauro, J.: Item benchmarks for the system usability scale. J. Usability Stud. 13(3), 158–167 (2018)

    Google Scholar 

  21. Lima, G., Grgić-Hlača, N., Jeong, J.K., Cha, M.: The conflict between explainable and accountable decision-making algorithms. In: FACCT, pp. 2103–2113. ACM, Seoul, Republic of Korea (2022)

    Google Scholar 

  22. Lyons, J.B.: Being transparent about transparency: A model for human-robot interaction, pp. 48–53. AAAI (2013)

    Google Scholar 

  23. Lyons, J.B., Havig, P.R.: Transparency in a human-machine context: approaches for fostering shared awareness/intent. In: Shumaker, R., Lackey, S. (eds.) VAMR 2014. LNCS, vol. 8525, pp. 181–190. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07458-0_18

    Chapter  Google Scholar 

  24. Malle, B.F.: How people explain behavior: a new theoretical framework. Pers. Soc. Psychol. Rev. 3(1), 23–48 (1999)

    Article  Google Scholar 

  25. Mayer, R.C., Davis, J.H., Schoorman, F.D.: An integrative model of organizational trust. Acad. Manag. Rev. 20(3), 709–734 (1995)

    Article  Google Scholar 

  26. Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  27. Miller, T., Hoffman, R., Amir, O., Holzinger, A.: Special issue on explainable artificial intelligence. Artif. Intell. 307, 103705 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  28. Mualla, Y., et al.: The quest of parsimonious XAI: a human-agent architecture for explanation formulation. Artif. Intell. 302, 103573 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  29. O’Leary, K., Wobbrock, J.O., Riskin, E.A.: Q-methodology as a research and design tool for HCI, pp. 1941–1950. ACM, Paris (2013)

    Google Scholar 

  30. Pavlou, P.A., Gefen, D.: Building effective online marketplaces with institution-based trust. Inf. Syst. Res. 15(1), 37–59 (2004)

    Article  Google Scholar 

  31. Rosenfeld, A.: Better metrics for evaluating explainable artificial intelligence. In: AAMAS, pp. 45–50, Richland, SC (2021)

    Google Scholar 

  32. Smith, R.W., Hipp, D.R.: Spoken Language Dialog Systems: A Practical Approach. Oxford University Press, Oxford (1994)

    Google Scholar 

  33. Christina Soyoung Song and Youn-Kyung Kim: The role of the human-robot interaction in consumers’ acceptance of humanoid retail service robots. J. Bus. Res. 146, 489–503 (2022)

    Article  Google Scholar 

  34. Tintarev, N., Masthoff, J.: Explaining recommendations: design and evaluation. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 353–382. Springer, Boston, MA (2015). https://doi.org/10.1007/978-1-4899-7637-6_10

    Chapter  Google Scholar 

  35. Trang Tran, T.N., Atas, M., Felfernig, A., Stettinger, M.: An overview of recommender systems in the healthy food domain. J. Intell. Inform. Syst. 50(3), 501–526 (2018)

    Article  Google Scholar 

  36. van der Waa, J., Nieuwburg, E., Cremers, A., Neerincx, M.: Evaluating XAI: A comparison of rule-based and example-based explanations. Artif. Intell. 291, 103404 (2023)

    MathSciNet  MATH  Google Scholar 

  37. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Q. 27(3), 425–478 (2003)

    Article  Google Scholar 

  38. Vermaas, P.E., Tan, Y.-H., van den Hoven, J., Burgemeestre, B., Hulstijn, J.: Designing for trust: a case of value-sensitive design. Knowl. Technol. Policy 23(3–4), 491–505 (2010)

    Article  Google Scholar 

  39. Vorm, E.S., Combs, D.J.Y.: Integrating transparency, trust, and acceptance: The intelligent systems technology model (ISTAM). Int. J. Hum.-Comput. Interact., 1–19 (2022)

    Google Scholar 

  40. Vorm, E.S., Miller, A.D.: Modeling user information needs to enable successful human-machine teams: designing transparency for autonomous systems. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) HCII 2020. LNCS (LNAI), vol. 12197, pp. 445–465. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50439-7_31

    Chapter  Google Scholar 

  41. Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explanations without opening the black box: automated decisions and the GDPR. Harvard J. Law Technol. 31(2), 841–887 (2018)

    Google Scholar 

  42. Walker, M.A., Litman, D.J., Kamm, A., Abella, A.: PARADISE: A framework for evaluating spoken dialogue agents. In: Proceedings of the 35th Annual meeting of the ACL/EACL, pp. 271–280, Madrid (1997)

    Google Scholar 

  43. Wallkötter, S., Tulli, S., Castellano, G., Paiva, A., Chetouani, M.: Explainable embodied agents through social cues: a review. ACM Trans. Hum.-Robot Interact. 10(3), 27:2–27:24 (2021)

    Google Scholar 

Download references

Acknowledgments

This work has been supported by CHIST-ERA grant CHIST-ERA19-XAI-005, and by (i) the Swiss National Science Foundation (G.A. 20CH21_195530), (ii) the Italian Ministry for Universities and Research, (iii) the Luxembourg National Research Fund (G.A. INTER/CHIST/19/14589586), (iv) the Scientific and Research Council of Turkey (TÜBİTAK, G.A. 120N680).

Author information

Authors and Affiliations

  1. University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Joris Hulstijn, Igor Tchappi & Amro Najjar

  2. Luxembourg Institute of Science and Technology (LIST), Esch-sur-Alzette, Luxembourg

    Amro Najjar

  3. Computer Science, Özyeğin University, Istanbul, Turkey

    Reyhan Aydoğan

  4. Interactive Intelligence, Delft University of Technology, Delft, Netherlands

    Reyhan Aydoğan

Authors
  1. Joris Hulstijn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Igor Tchappi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amro Najjar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Reyhan Aydoğan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joris Hulstijn .

Editor information

Editors and Affiliations

  1. University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland

    Davide Calvaresi

  2. Luxembourg Institute of Science and Technology, Esch-sur-Alzette, Luxembourg

    Amro Najjar

  3. Alma Mater Studiorum, Università di Bologna, Bologna, Italy

    Andrea Omicini

  4. Ozyegin University, Istanbul, Türkiye

    Reyhan Aydogan

  5. Alma Mater Studiorum, Università di Bologna, Bologna, Italy

    Rachele Carli

  6. Alma Mater Studiorum, Università di Bologna, Bologna, Italy

    Giovanni Ciatto

  7. Université de Technologie de Belfort-Montbéliard, Belfort Cedex, France

    Yazan Mualla

  8. Umeå University, Umeå, Sweden

    Kary Främling

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hulstijn, J., Tchappi, I., Najjar, A., Aydoğan, R. (2023). Metrics for Evaluating Explainable Recommender Systems. In: Calvaresi, D., et al. Explainable and Transparent AI and Multi-Agent Systems. EXTRAAMAS 2023. Lecture Notes in Computer Science(), vol 14127. Springer, Cham. https://doi.org/10.1007/978-3-031-40878-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-40878-6_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-40877-9

  • Online ISBN: 978-3-031-40878-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation