{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T07:19:07Z","timestamp":1740122347592,"version":"3.37.3"},"reference-count":62,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,1,25]],"date-time":"2021-01-25T00:00:00Z","timestamp":1611532800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,1,25]],"date-time":"2021-01-25T00:00:00Z","timestamp":1611532800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000288","name":"Royal Society","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000288","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Min Knowl Disc"],"published-print":{"date-parts":[[2021,3]]},"abstract":"Abstract<\/jats:title>Moral responsibility is a major concern in autonomous systems, with applications ranging from self-driving cars to kidney exchanges. Although there have been recent attempts to formalise responsibility and blame, among similar notions, the problem of learning within these formalisms has been unaddressed. From the viewpoint of such systems, the urgent questions are: (a) How can models of moral scenarios and blameworthiness be extracted and learnt automatically from data? (b) How can judgements be computed effectively and efficiently, given the split-second decision points faced by some systems? By building on constrained tractable probabilistic learning, we propose and implement a hybrid (between data-driven and rule-based methods) learning framework for inducing models of such scenarios automatically from data and reasoning tractably from them. We report on experiments that compare our system with human judgement in three illustrative domains: lung cancer staging, teamwork management, and trolley problems.<\/jats:p>","DOI":"10.1007\/s10618-020-00726-4","type":"journal-article","created":{"date-parts":[[2021,1,25]],"date-time":"2021-01-25T08:05:19Z","timestamp":1611561919000},"page":"621-659","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Learning tractable probabilistic models for moral responsibility and blame"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1695-0871","authenticated-orcid":false,"given":"Lewis","family":"Hammond","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5573-8465","authenticated-orcid":false,"given":"Vaishak","family":"Belle","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,1,25]]},"reference":[{"key":"726_CR1","unstructured":"Abel D, MacGlashan J, Littman ML (2016) Reinforcement learning as a framework for ethical decision making. In: Workshops at the 30th AAAI conference on artificial intelligence"},{"key":"726_CR2","unstructured":"Acharya J, Bhattacharyya A, Constantinos D, Saravanan K (2018) Learning and testing causal models with interventions. arXiv preprint arXiv:1805.09697"},{"issue":"3","key":"726_CR3","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1007\/s10676-006-0004-4","volume":"7","author":"C Allen","year":"2005","unstructured":"Allen C, Smit I, Wallach W (2005) Artificial morality: top\u2013down, bottom\u2013up, and hybrid approaches. Ethics Inf Technol 7(3):149\u2013155","journal-title":"Ethics Inf Technol"},{"key":"726_CR4","doi-asserted-by":"crossref","unstructured":"Anderson M, Anderson SL (2014) Geneth: a general ethical dilemma analyzer. In: Proceedings of the 28th AAAI conference on artificial intelligence, pp 253\u2013261","DOI":"10.1609\/aaai.v28i1.8737"},{"key":"726_CR5","doi-asserted-by":"crossref","unstructured":"Arkin RC, Ulam PD, Duncan B (2009) An ethical governor for constraining lethal action in an autonomous system. Technical Report GIT-GVU-09-02, Georgia Institute of Technology Mobile Robot Lab","DOI":"10.21236\/ADA493563"},{"issue":"886","key":"726_CR6","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1017\/S1816383112000768","volume":"94","author":"P Asaro","year":"2012","unstructured":"Asaro P (2012) On banning autonomous weapon systems: human rights, automation, and the dehumanization of lethal decision-making. Int Rev Red Cross 94(886):687\u2013709","journal-title":"Int Rev Red Cross"},{"key":"726_CR7","unstructured":"Automated Reasoning Group (University Of California, Los Angeles). The SDD Package 2.0, 2018. http:\/\/reasoning.cs.ucla.edu\/sdd, Accessed 2018-08-17"},{"issue":"7729","key":"726_CR8","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1038\/s41586-018-0637-6","volume":"563","author":"E Awad","year":"2018","unstructured":"Awad E, Dsouza S, Kim R, Schulz J, Henrich J, Shariff A, Bonnefon J-F, Rahwan I (2018) The moral machine experiment. Nature 563(7729):59","journal-title":"Nature"},{"key":"726_CR9","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1613\/jair.2648","volume":"34","author":"F Bacchus","year":"2009","unstructured":"Bacchus F, Dalmao S, Pitassi T (2009) Solving #SAT and Bayesian inference with backtracking search. J Artif Intell Res 34:391\u2013442","journal-title":"J Artif Intell Res"},{"key":"726_CR10","first-page":"1","volume":"18","author":"SH Bach","year":"2017","unstructured":"Bach SH, Broecheler M, Huang B, Getoor L (2017) Hinge-loss markov random fields and probabilistic soft logic. J Mach Learn Res 18:1\u201367","journal-title":"J Mach Learn Res"},{"issue":"3","key":"726_CR11","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/j.cognition.2009.07.005","volume":"113","author":"CL Baker","year":"2009","unstructured":"Baker CL, Saxe R, Tenenbaum JB (2009) Action understanding as inverse planning. Cognition 113(3):329\u2013349","journal-title":"Cognition"},{"key":"726_CR12","unstructured":"Bhattacharjya Debarun, Shachter RD (2012) Evaluating influence diagrams with decision circuits. arXiv preprint arXiv:1206.5257"},{"key":"726_CR13","unstructured":"Boutilier C, Friedman N, Goldszmidt M, Koller D (1996) Context-specific independence in Bayesian networks. In: Proceedings of the twelfth international conference on uncertainty in artificial intelligence, San Francisco, CA, USA, pp 115\u2013123"},{"key":"726_CR14","doi-asserted-by":"crossref","unstructured":"Briggs G, Scheutz M (2014) Modeling blame to avoid positive face threats in natural language generation. In: Proceedings of the INLG and SIGDIAL 2014 joint session, pp 157\u2013161","DOI":"10.3115\/v1\/W14-5001"},{"key":"726_CR15","unstructured":"Charisi V, Dennis L, Fisher M, Lieck R, Matthias A, Slavkovik M, Sombetzki J, Winfield AFT, Yampolskiy R (2017) Towards moral autonomous systems. arXiv preprint arXiv:1703.04741"},{"key":"726_CR16","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1613\/jair.1391","volume":"22","author":"H Chockler","year":"2004","unstructured":"Chockler H, Halpern JY (2004) Responsibility and blame: a structural-model approach. J Artif Intell Res 22:93\u2013115","journal-title":"J Artif Intell Res"},{"key":"726_CR17","unstructured":"Choi A, Van den Broeck G, Darwiche A (2015) Tractable learning for structured probability spaces: a case study in learning preference distributions. In: Proceedings of the 24th international joint conference on artificial intelligence, pp 2861\u20132868"},{"key":"726_CR18","doi-asserted-by":"crossref","unstructured":"Conitzer V, Sinnott-Armstrong W, Borg JS, Deng Y, Kramer M (2017) Moral decision making frameworks for artificial intelligence. In: Proceedings of the 31st AAAI conference on artificial intelligence, pp 4831\u20134835","DOI":"10.1609\/aaai.v31i1.11140"},{"issue":"2","key":"726_CR19","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1037\/a0031021","volume":"104","author":"P Conway","year":"2013","unstructured":"Conway P, Gawronski B (2013) Deontological and utilitarian inclinations in moral decision making: a process dissociation approach. J Pers Soc Psychol 104(2):216","journal-title":"J Pers Soc Psychol"},{"key":"726_CR20","unstructured":"Darwiche A (2011) SDD: a new canonical representation of propositional knowledge bases. In: Proceedings of the 22nd international joint conference on artificial intelligence, p 81"},{"key":"726_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.robot.2015.11.012","volume":"77","author":"L Dennis","year":"2016","unstructured":"Dennis L, Fisher M, Slavkovik M, Webster M (2016) Formal verification of ethical choices in autonomous systems. Robot Auton Syst 77:1\u201314","journal-title":"Robot Auton Syst"},{"issue":"4","key":"726_CR22","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1007\/s10892-017-9252-2","volume":"21","author":"A Etzioni","year":"2017","unstructured":"Etzioni A, Etzioni O (2017) Incorporating ethics into artificial intelligence. J Ethics 21(4):403\u2013418","journal-title":"J Ethics"},{"key":"726_CR23","doi-asserted-by":"crossref","unstructured":"Evans O, Stuhlm\u00fcller A, Goodman ND (2016) Learning the preferences of ignorant, inconsistent agents. In: Proceedings of the 30th AAAI conference on artificial intelligence, pp 323\u2013329","DOI":"10.1609\/aaai.v30i1.10010"},{"key":"726_CR24","unstructured":"Gens R, Domingos P (2013) Learning the structure of sum-product networks. In: Proceedings of the 30th international conference on machine learning, pp 873\u2013880"},{"key":"726_CR25","doi-asserted-by":"crossref","unstructured":"Gratch J, Mao W (2003) Automating after action review: Attributing blame or credit in team training. In: Proceedings of the 12th conference on behavior representation in modeling and simulation","DOI":"10.21236\/ADA460293"},{"key":"726_CR26","doi-asserted-by":"crossref","unstructured":"Greene J, Rossi F, Tasioulas J, Venable KB, Williams BC (2016) Embedding ethical principles in collective decision support systems. In: Proceedings of the 30th AAAI conference on artificial intelligence, pp 4147\u20134151","DOI":"10.1609\/aaai.v30i1.9804"},{"key":"726_CR27","doi-asserted-by":"crossref","unstructured":"Groom V, Chen J, Johnson T, Kara FA, Nass C (2010) Critic, compatriot, or chump?: Responses to robot blame attribution. In: Proceedings of the 5th ACM\/IEEE international conference on human-robot interaction, pp 211\u2013218","DOI":"10.1145\/1734454.1734545"},{"key":"726_CR28","doi-asserted-by":"crossref","unstructured":"Halpern JY, Kleiman-Weiner M (2018) Towards formal definitions of blameworthiness, intention, and moral responsibility. In: Proceedings of the 32nd AAAI conference on artificial intelligence, pp 1853\u20131860","DOI":"10.1609\/aaai.v32i1.11557"},{"issue":"4","key":"726_CR29","doi-asserted-by":"publisher","first-page":"843","DOI":"10.1093\/bjps\/axi147","volume":"56","author":"JY Halpern","year":"2005","unstructured":"Halpern JY, Pearl J (2005) Causes and explanations: a structural-model approach. Part I: causes. Br J Philos Sci 56(4):843\u2013887","journal-title":"Br J Philos Sci"},{"key":"726_CR30","unstructured":"Hammond L (2018) LTPM4MAB code package. University of Edinburgh. https:\/\/github.com\/lrhammond\/ltpm4mrAb. Accessed 05 Oct 2018"},{"key":"726_CR31","doi-asserted-by":"publisher","first-page":"160127","DOI":"10.1038\/sdata.2016.127","volume":"4","author":"Yu Han","year":"2017","unstructured":"Han Yu, Shen Z, Miao C, Leung C, Chen Y, Fauvel S, Lin J, Cui L, Pan Z, Yang Q (2017) A dataset of human decision-making in teamwork management. Sci Data 4:160127","journal-title":"Sci Data"},{"key":"726_CR32","doi-asserted-by":"crossref","unstructured":"Jentzsch S, Schramowski P, Rothkopf C, Kersting K (2019) Semantics derived automatically from language corpora contain human-like moral choices. In: Proceedings of the 2019 AAAI\/ACM conference on AI, ethics, and society, AIES \u201919. pp 37\u201344","DOI":"10.1145\/3306618.3314267"},{"issue":"3","key":"726_CR33","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1177\/0272989X9701700302","volume":"17","author":"RF Jr Nease","year":"1997","unstructured":"Jr Nease RF, Owens DK (1997) Use of influence diagrams to structure medical decisions. Med Decis Mak 17(3):263\u2013275","journal-title":"Med Decis Mak"},{"key":"726_CR34","unstructured":"Kalra A (2017) Tachyon. University of Waterloo. https:\/\/github.com\/KalraA\/Tachyon, Accessed 23 Aug 2018"},{"key":"726_CR35","doi-asserted-by":"crossref","unstructured":"Kaniarasu P, Steinfeld AM (2014) Effects of blame on trust in human robot interaction. In: The 23rd IEEE international symposium on robot and human interactive communication, pp 850\u2013855","DOI":"10.1109\/ROMAN.2014.6926359"},{"key":"726_CR36","doi-asserted-by":"crossref","unstructured":"Kim T, Hinds P (2006) Who should I blame? Effects of autonomy and transparency on attributions in human-robot interaction. In: The 15th IEEE international symposium on robot and human interactive communication, pp 80\u201385","DOI":"10.1109\/ROMAN.2006.314398"},{"key":"726_CR37","doi-asserted-by":"crossref","unstructured":"Kim R, Kleiman-Weiner M, Abeliuk A, Awad E, Dsouza S, Tenenbaum JB, Rahwan I (2018) A computational model of commonsense moral decision making. In: Proceedings of the 2018 AAAI\/ACM conference on AI, ethics, and society, pp 197\u2013203","DOI":"10.1145\/3278721.3278770"},{"key":"726_CR38","unstructured":"Kisa D, Van den Broeck G, Choi A, Darwiche A (2014) Probabilistic sentential decision diagrams. In: Proceedings of the 14th international conference on principles of knowledge representation and reasoning, pp 558\u2013567"},{"key":"726_CR39","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/j.cognition.2017.03.005","volume":"167","author":"M Kleiman-Weiner","year":"2017","unstructured":"Kleiman-Weiner M, Saxe R, Tenenbaum JB (2017) Learning a commonsense moral theory. Cognition 167:107\u2013123","journal-title":"Cognition"},{"key":"726_CR40","unstructured":"Kleiman-Weiner M, Gerstenberg T, Levine S, Tenenbaum JB (2015) Inference of intention and permissibility in moral decision making. In: Proceedings of the 37th annual conference of the cognitive science society, pp 1123\u20131128"},{"key":"726_CR41","unstructured":"Liang Y, Bekker J, Van den Broeck G (2017) Learning the structure of probabilistic sentential decision diagrams. In: Proceedings of the 33rd conference on uncertainty in artificial intelligence, pp 134\u2013145"},{"key":"726_CR42","doi-asserted-by":"crossref","unstructured":"Lindner F, Bentzen MM, Nebel B (2017) The Hera approach to morally competent robots. In: Proceedings of the IEEE\/RSJ intelligent robots and systems, pp 6991\u20136997","DOI":"10.1109\/IROS.2017.8206625"},{"issue":"2","key":"726_CR43","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1080\/1047840X.2014.877340","volume":"25","author":"BF Malle","year":"2014","unstructured":"Malle BF, Guglielmo S, Monroe AE (2014) A theory of blame. Psychol Inq 25(2):147\u2013186","journal-title":"Psychol Inq"},{"key":"726_CR44","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1613\/jair.3526","volume":"44","author":"W Mao","year":"2012","unstructured":"Mao W, Gratch J (2012) Modeling social causality and responsibility judgement in multi-agent interactions. J Artif Intell Res 44:223\u2013273","journal-title":"J Artif Intell Res"},{"issue":"1","key":"726_CR45","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1006\/game.1995.1023","volume":"10","author":"RD McKelvey","year":"1995","unstructured":"McKelvey RD, Palfrey TR (1995) Quantal response equilibria for normal form games. Games Econ Behav 10(1):6\u201338","journal-title":"Games Econ Behav"},{"key":"726_CR46","unstructured":"Melibari MA, Poupart P, Doshi P (2016) Sum-product-max networks for tractable decision making. In: Proceedings of the 15th international conference on autonomous agents & multiagent systems, pp 1419\u20131420"},{"issue":"4","key":"726_CR47","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1109\/MIS.2006.80","volume":"21","author":"JH Moor","year":"2006","unstructured":"Moor JH (2006) The nature, importance, and difficulty of machine ethics. IEEE Intell Syst 21(4):18\u201321","journal-title":"IEEE Intell Syst"},{"key":"726_CR48","unstructured":"Ng AY, Russell SJ (2000) Algorithms for inverse reinforcement learning. In: Proceedings of the 17th international conference on machine learning, pp 663\u2013670"},{"issue":"1\u20132","key":"726_CR49","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1016\/j.artint.2004.08.003","volume":"160","author":"TD Nielsen","year":"2004","unstructured":"Nielsen TD, Jensen FV (2004) Learning a decision maker\u2019s utility function from (possibly) inconsistent behavior. Artif Intell 160(1\u20132):53\u201378","journal-title":"Artif Intell"},{"key":"726_CR50","unstructured":"Noothigattu R, Gaikwad SNS, Awad E, Dsouza S, Rahwan I, Ravikumar P, Procaccia AD (2017) A voting-based system for ethical decision making. arXiv preprint arXiv:1709.06692"},{"key":"726_CR51","unstructured":"Papantonis I, Belle V (2019) Interventions and counterfactuals in tractable probabilistic models. In: NeurIPS workshop on knowledge representation & reasoning meets machine learning"},{"key":"726_CR52","doi-asserted-by":"crossref","unstructured":"Pearl J (1998) Graphical models for probabilistic and causal reasoning. In: Quantified representation of uncertainty and imprecision. Springer, pp 367\u2013389","DOI":"10.1007\/978-94-017-1735-9_12"},{"key":"726_CR53","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1214\/09-SS057","volume":"3","author":"J Pearl","year":"2009","unstructured":"Pearl J (2009) Causal inference in statistics: an overview. Stat Surv 3:96\u2013146","journal-title":"Stat Surv"},{"issue":"10","key":"726_CR54","doi-asserted-by":"publisher","first-page":"2030","DOI":"10.1109\/TPAMI.2016.2618381","volume":"39","author":"R Peharz","year":"2017","unstructured":"Peharz R, Gens R, Pernkopf F, Domingos P (2017) On the latent variable interpretation in sum-product networks. IEEE Trans Pattern Anal Mach Intell 39(10):2030\u20132044","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"726_CR55","doi-asserted-by":"crossref","unstructured":"Poon H, Domingos P (2011) Sum-product networks: a new deep architecture. In: IEEE international conference on computer vision workshops, pp 689\u2013690","DOI":"10.1109\/ICCVW.2011.6130310"},{"issue":"1\u20132","key":"726_CR56","first-page":"107","volume":"62","author":"M Richardson","year":"2006","unstructured":"Richardson M, Domingos P (2006) Markov logic networks. Mach Lang 62(1\u20132):107\u2013136","journal-title":"Mach Lang"},{"key":"726_CR57","doi-asserted-by":"crossref","unstructured":"Shaw NP, St\u00f6ckel A, Orr RW, Lidbetter TF, Cohen R (2018) Towards provably moral AI agents in bottom-up learning frameworks. In: Proceedings of the 2018 AAAI\/ACM conference on AI, ethics, and society, pp 271\u2013277","DOI":"10.1145\/3278721.3278728"},{"issue":"3\u20134","key":"726_CR58","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1007\/s10892-005-3508-y","volume":"9","author":"P Singer","year":"2005","unstructured":"Singer P (2005) Ethics and intuitions. J Ethics 9(3\u20134):331\u2013352","journal-title":"J Ethics"},{"issue":"6","key":"726_CR59","doi-asserted-by":"publisher","first-page":"1395","DOI":"10.2307\/796133","volume":"94","author":"JJ Thomson","year":"1985","unstructured":"Thomson JJ (1985) The trolley problem. Yale Law J 94(6):1395\u20131415","journal-title":"Yale Law J"},{"key":"726_CR60","doi-asserted-by":"crossref","unstructured":"Van den Broeck G, Thon I, van Otterlo M, De Raedt L (2010) Dtproblog: a decision-theoretic probabilistic prolog. In: Proceedings of the twenty-fourth AAAI conference on artificial intelligence, AAAI\u201910. AAAI Press, pp 1217\u20131222","DOI":"10.1609\/aaai.v24i1.7755"},{"key":"726_CR61","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.cogsys.2017.04.002","volume":"48","author":"D Vanderelst","year":"2018","unstructured":"Vanderelst D, Winfield A (2018) An architecture for ethical robots inspired by the simulation theory of cognition. Cogn Syst Res 48:56\u201366","journal-title":"Cogn Syst Res"},{"key":"726_CR62","unstructured":"Volkovich I (2016) A guide to learning arithmetic circuits. In: Proceedings of the 29th conference on learning theory, pp 1540\u20131561"}],"container-title":["Data Mining and Knowledge Discovery"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-020-00726-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10618-020-00726-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10618-020-00726-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,13]],"date-time":"2022-12-13T04:43:33Z","timestamp":1670906613000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10618-020-00726-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,1,25]]},"references-count":62,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["726"],"URL":"https:\/\/doi.org\/10.1007\/s10618-020-00726-4","relation":{},"ISSN":["1384-5810","1573-756X"],"issn-type":[{"type":"print","value":"1384-5810"},{"type":"electronic","value":"1573-756X"}],"subject":[],"published":{"date-parts":[[2021,1,25]]},"assertion":[{"value":"8 November 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 November 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 January 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}