{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T07:40:05Z","timestamp":1733125205726,"version":"3.30.0"},"reference-count":50,"publisher":"SAGE Publications","issue":"1","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IA"],"published-print":{"date-parts":[[2022,7,8]]},"abstract":"A common practice in modern explainable AI is to post-hoc explain black-box machine learning (ML) predictors \u2013 such as neural networks \u2013 by extracting symbolic knowledge out of them, in the form of either rule lists or decision trees. By acting as a surrogate model, the extracted knowledge aims at revealing the inner working of the black box, thus enabling its inspection, representation, and explanation. Various knowledge-extraction algorithms have been presented in the literature so far. Unfortunately, running implementations of most of them are currently either proofs of concept or unavailable. In any case, a unified, coherent software framework supporting them all \u2013 as well as their interchange, comparison, and exploitation in arbitrary ML workflows \u2013 is currently missing. Accordingly, in this paper we discuss the design of PSyKE, a platform providing general-purpose support to symbolic knowledge extraction from different sorts of black-box predictors via many extraction algorithms. Notably, PSyKE targets symbolic knowledge in logic form, allowing the extraction of first-order logic clauses. The extracted knowledge is thus both machine- and human-interpretable, and can be used as a starting point for further symbolic processing\u2014e.g. automated reasoning.<\/jats:p>","DOI":"10.3233\/ia-210120","type":"journal-article","created":{"date-parts":[[2022,7,8]],"date-time":"2022-07-08T15:08:57Z","timestamp":1657292937000},"page":"27-48","source":"Crossref","is-referenced-by-count":11,"title":["Symbolic knowledge extraction from opaque ML predictors in PSyKE: Platform design & experiments"],"prefix":"10.1177","volume":"16","author":[{"given":"Federico","family":"Sabbatini","sequence":"first","affiliation":[{"name":"Dipartimento di Scienze Pure e Applicate (DiSPeA), Universit\u00e0 degli Studi di Urbino \u201cCarlo Bo\u201d, Italy"},{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria (DISI), ALMA MATER STUDIORUM\u2014Universit\u00e0 di Bologna, Italy"}]},{"given":"Giovanni","family":"Ciatto","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria (DISI), ALMA MATER STUDIORUM\u2014Universit\u00e0 di Bologna, Italy"}]},{"given":"Roberta","family":"Calegari","sequence":"additional","affiliation":[{"name":"Alma Mater Research Institute for Human-Centered Artificial Intelligence (AlmaAI), ALMA MATER STUDIORUM\u2014Universit\u00e0 di Bologna, Italy"}]},{"given":"Andrea","family":"Omicini","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica \u2013 Scienza e Ingegneria (DISI), ALMA MATER STUDIORUM\u2014Universit\u00e0 di Bologna, Italy"}]}],"member":"179","reference":[{"issue":"6","key":"10.3233\/IA-210120_ref1","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1016\/0950-7051(96)81920-4","article-title":"Survey and critique of techniques for extracting rules from trained artificial neural networks","volume":"8","author":"Andrews","year":"1995","journal-title":"Knowledge-Based Systems"},{"key":"10.3233\/IA-210120_ref2","doi-asserted-by":"crossref","unstructured":"Azcarraga A. , Liu M.D. and Setiono R. , Keyword extraction using backpropagation neural networks and rule extraction. In The 2012 International Joint Conference on Neural Networks (IJCNN 2012), pp. 1\u20137. IEEE, 2012.","DOI":"10.1109\/IJCNN.2012.6252618"},{"key":"10.3233\/IA-210120_ref3","unstructured":"Baesens B. , Setiono R. , De Lille V. , ViaeneS. and VanthienenJ., Building credit-risk evaluation expert systems using neural network rule extraction and decision tables. In StoreyVeda C., SarkarSumit, and DeGrossJanice I., editors, ICIS 2001 Proceedings, pp. 159\u2013168. Association for Information Systems, 2001."},{"issue":"3","key":"10.3233\/IA-210120_ref4","doi-asserted-by":"crossref","first-page":"312","DOI":"10.1287\/mnsc.49.3.312.12739","article-title":"Using neural network rule extraction and decision tables for credit-risk evaluation","volume":"49","author":"Baesens","year":"2003","journal-title":"Management Science"},{"issue":"5","key":"10.3233\/IA-210120_ref5","first-page":"1672","article-title":"Eclectic rule-extraction from support vector machines","volume":"2","author":"Barakat","year":"2008","journal-title":"International Journal of Computer and Information Engineering"},{"key":"10.3233\/IA-210120_ref6","first-page":"183","article-title":"Three medical examples in neural network rule extraction","volume":"13","author":"Bologna","year":"1997","journal-title":"Physica Medica"},{"issue":"1","key":"10.3233\/IA-210120_ref7","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1023\/B:MACH.0000019804.29836.05","article-title":"Khiops: A statistical discretization method of continuous attributes","volume":"55","author":"Boull\u00e9","year":"2004","journal-title":"Machine Learning"},{"key":"10.3233\/IA-210120_ref8","unstructured":"Breiman L. , Friedman J. , Stone C.J. and Olshen R.A. , Classification and Regression Trees. CRC Press, 1984."},{"issue":"1","key":"10.3233\/IA-210120_ref9","doi-asserted-by":"crossref","first-page":"7","DOI":"10.3233\/IA-190036","article-title":"On the integration of symbolic and sub-symbolic techniques for XAI: A survey","volume":"14","author":"Calegari","year":"2020","journal-title":"Intelligenza Artificiale"},{"key":"10.3233\/IA-210120_ref10","doi-asserted-by":"crossref","unstructured":"Calvaresi D. , Ciatto G. , Najjar A. , Aydo\u011fan R. , Van der TorreL., OmiciniA. and SchumacherM., EXPECTATION: Personalized explainable artificial intelligence for decentralized agents with heterogeneous knowledge. In CalvaresiDavide, NajjarAmro, WinikoffMichael, and Fr\u00e4mlingKary, editors, Explainable and Transparent AI and Multi-Agent Systems. Third International Workshop, EXTRAAMAS Virtual Event, May 3\u20137, Revised Selected Papers, volume 12688 of LNCS, pp. 331\u2013343. Springer Nature, Basel, Switzerland, 2021.","DOI":"10.1007\/978-3-030-82017-6_20"},{"issue":"1","key":"10.3233\/IA-210120_ref11","doi-asserted-by":"crossref","first-page":"173","DOI":"10.1007\/s00500-014-1488-1","article-title":"ur-caim: improved CAIM discretization for unbalanced and balanced data","volume":"20","author":"Cano","year":"2016","journal-title":"Soft Comput"},{"key":"10.3233\/IA-210120_ref12","doi-asserted-by":"crossref","unstructured":"Ciatto G. , Calegari R. and Omicini A. , 2PKt: A logic-based ecosystem for symbolic AI, SoftwareX 16(100817) (2021), 1\u20137, December 2021.","DOI":"10.1016\/j.softx.2021.100817"},{"key":"10.3233\/IA-210120_ref13","unstructured":"Ciatto G. , Calegari R. , Omicini A. and Calvaresi D. , Towards XMAS: eXplainability through Multi-Agent Systems. In SavaglioClaudio, FortinoGiancarlo, CiattoGiovanni, and OmiciniAndrea, editors, AI&IoT 2019 \u2013 Artificial Intelligence and Internet of Things volume 2502 of CEUR Workshop Proceedings, pp. 40\u201353. CEUR WS, November, 2019."},{"key":"10.3233\/IA-210120_ref14","unstructured":"Ciatto G. , Calvaresi D. , Schumacher M.I. and Omicini A. , An abstract framework for agent-based explanations in AI. In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke-Smith, editors, pp. \u2013 IFAAMAS, May, 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1816\u20131818. IFAAMAS, May 2020."},{"key":"10.3233\/IA-210120_ref16","doi-asserted-by":"crossref","unstructured":"Craven M.W. and Shavlik J.W. , Using sampling and queries to extract rules from trained neural networks. In Machine Learning Proceedings 1994, pp. 37\u201345, Elsevier, 1994.","DOI":"10.1016\/B978-1-55860-335-6.50013-1"},{"key":"10.3233\/IA-210120_ref17","unstructured":"Craven M.W. and Shavlik J.W. , Extracting treestructured representations of trained networks. In David S. Touretzky, Michael C. Mozer, and Michael E. Hasselmo, editors, Advances in Neural Information Processing Systems 8. Proceedings of the 1995 Conference, pp. 24\u201330. The MIT Press, June 1996."},{"key":"10.3233\/IA-210120_ref18","doi-asserted-by":"crossref","unstructured":"Dougherty J. , Kohavi R. and Sahami M. , Supervised and unsupervised discretization of continuous features. In Armand Prieditis and Stuart J. Russell, editors, Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, July 9-12, 1995, pp. 194\u2013202. Morgan Kaufmann, 1995.","DOI":"10.1016\/B978-1-55860-377-6.50032-3"},{"key":"10.3233\/IA-210120_ref19","doi-asserted-by":"crossref","unstructured":"Elhilbawi H. , Eldawlatly S. and Mahdi H. , The importance of discretization methods in machine learning applications: A case study of predicting ICU mortality. In Aboul Ella Hassanien, Kuo-Chi Chang, and Mincong Tang, editors, Advanced Machine Learning Technologies and Applications - Proceedings of AMLTA 2021, Cairo, Egypt, March 22-24, 2021, volume 1339 of Advances in Intelligent Systems and Computing, pp. 214\u2013224. Springer, 2021.","DOI":"10.1007\/978-3-030-69717-4_23"},{"key":"10.3233\/IA-210120_ref20","doi-asserted-by":"crossref","unstructured":"Franco L. , Luis Subirats J. , MolinaI., AlbaE. and JerezJ.M., Early breast cancer prognosis prediction and rule extraction using a new constructive neural network algorithm. In Computational and Ambient Intelligence (IWANN 2007), volume 4507 of LNCS, pp. 1004\u20131011, Springer, 2007.","DOI":"10.1007\/978-3-540-73007-1_121"},{"key":"10.3233\/IA-210120_ref21","doi-asserted-by":"crossref","unstructured":"FreitasAlex A., Comprehensible classification models: a position paper, ACM SIGKDD Explorations Newsletter 15(1) (2014),1\u201310, June 2014.","DOI":"10.1145\/2594473.2594475"},{"key":"10.3233\/IA-210120_ref22","doi-asserted-by":"crossref","unstructured":"Gavanelli M. and Rossi F. , Constraint logic programming. In Agostino Dovier and Enrico Pontelli, editors, A 25-Year Perspective on Logic Programming: Achievements of the Italian Association for Logic Programming, GULP, volume 6125 of LNCS, pp. 64\u201386. Springer, 2010.","DOI":"10.1007\/978-3-642-14309-0_4"},{"issue":"5","key":"10.3233\/IA-210120_ref23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3236009","article-title":"A survey of methods for explaining black box models","volume":"51","author":"Guidotti","year":"2018","journal-title":"ACM Computing Surveys"},{"issue":"3","key":"10.3233\/IA-210120_ref24","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/S0933-3657(00)00064-6","article-title":"A comparison between two neural network rule extraction techniques for the diagnosis of hepatobiliary disorders","volume":"20","author":"Hayashi","year":"2000","journal-title":"Artificial Intelligence in Medicine"},{"key":"10.3233\/IA-210120_ref25","unstructured":"Ho K.M. and Scott Paul D.. Zeta: A global method for discretization of continuous variables. In David Heckerman, Heikki Mannila, and Daryl Pregibon, editors, Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (KDD-97), Newport Beach, California, USA, August 14-17, 1997, pp. 191\u2013194. AAAI Press, 1997."},{"key":"10.3233\/IA-210120_ref26","doi-asserted-by":"crossref","unstructured":"Hofmann A. , Schmitz C. and Sick B. , Rule extraction from neural networks for intrusion detection in computer networks. In 2003 IEEE International Conference on Systems, Man and Cybernetics, volume 2, (2003), pp. 1259\u20131265. IEEE.","DOI":"10.1109\/ICSMC.2003.1244584"},{"key":"10.3233\/IA-210120_ref27","doi-asserted-by":"crossref","unstructured":"Huysmans J. , Baesens B. and Vanthienen J. , ITER: An algorithm for predictive regression rule extraction. In Data Warehousing and Knowledge Discovery (DaWaK 2006), (2006), pp. 270\u2013279. Springer.","DOI":"10.1007\/11823728_26"},{"issue":"1","key":"10.3233\/IA-210120_ref28","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.dss.2010.12.003","article-title":"An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models","volume":"51","author":"Huysmans","year":"2011","journal-title":"Decision Support Systems"},{"key":"10.3233\/IA-210120_ref29","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1016\/0743-1066(94)90033-7","article-title":"Constraint logic programming: A survey","author":"Jaffar","year":"1994","journal-title":"Journal of Logic Programming"},{"key":"10.3233\/IA-210120_ref30","doi-asserted-by":"crossref","first-page":"103459","DOI":"10.1016\/j.artint.2021.103459","article-title":"Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and errorrates in XAI user studies","volume":"294","author":"Kenny","year":"2021","journal-title":"Artificial Intelligence"},{"key":"10.3233\/IA-210120_ref31","unstructured":"Kerber R. , Chimerge: Discretization of numeric attributes. In William R. Swartout, editor, Proceedings of the 10th National Conference on Artificial Intelligence, San Jose, CA, USA, July 12-16, 1992, pp. 123\u2013128. AAAI Press \/ The MIT Press, 1992."},{"key":"10.3233\/IA-210120_ref32","doi-asserted-by":"crossref","first-page":"971","DOI":"10.1109\/ICDMW.2008.117","article-title":"G-REX: A versatile framework for evolutionary data mining","author":"Konig","year":"2008","journal-title":"2008 IEEE International Conference on Data Mining Workshops (ICDM 2008 Workshops)"},{"issue":"2","key":"10.3233\/IA-210120_ref33","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1109\/TKDE.2004.1269594","article-title":"CAIM discretization algorithm","volume":"16","author":"Kurgan","year":"2004","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"10.3233\/IA-210120_ref34","doi-asserted-by":"crossref","unstructured":"Lipton Z.C. , The mythos of model interpretability, Queue 16(3) (2018),31\u201357, June 2018.","DOI":"10.1145\/3236386.3241340"},{"issue":"3","key":"10.3233\/IA-210120_ref36","doi-asserted-by":"crossref","first-page":"1466","DOI":"10.1016\/j.ejor.2006.04.051","article-title":"Comprehensible credit scoring models using rule extraction from support vector machines","volume":"183","author":"Martens","year":"2007","journal-title":"European Journal of Operational Research"},{"key":"10.3233\/IA-210120_ref37","unstructured":"Metakides G. and Nerode A. , Principles of logic and logic programming, volume 13 of Studies in Computer Science and Artificial Intelligence, Elsevier, 1996."},{"key":"10.3233\/IA-210120_ref38","doi-asserted-by":"crossref","unstructured":"Murphy P.M. and Pazzani M.J. , Id2-of-3: Constructive induction of m-of-n concepts for discriminators in decision trees. In Machine Learning Proceedings (1991), pp. 183\u2013187, Elsevier.","DOI":"10.1016\/B978-1-55860-200-7.50040-4"},{"key":"10.3233\/IA-210120_ref39","first-page":"2825","article-title":"Scikitlearn: Machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research (JMLR)"},{"issue":"3","key":"10.3233\/IA-210120_ref40","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1016\/S0020-7373(87)80053-6","article-title":"Simplifying decision trees","volume":"27","author":"Ross Quinlan","year":"1987","journal-title":"International Journal of Man-Machine Studies"},{"key":"10.3233\/IA-210120_ref41","unstructured":"Ross J. , Quinlan, C4.5: Programming for machine learning. Morgan Kauffmann, 1993."},{"key":"10.3233\/IA-210120_ref42","doi-asserted-by":"crossref","unstructured":"Rocha A. , Paulo Papa J. and MeiraLuis A. A., How far do we get using machine learning black-boxes? International Journal of Pattern Recognition and Artificial Intelligence 26(02) (2012), 1261001\u2013(1\u201323).","DOI":"10.1142\/S0218001412610010"},{"issue":"5","key":"10.3233\/IA-210120_ref43","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1038\/s42256-019-0048-x","article-title":"Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead","volume":"1","author":"Rudin","year":"2019","journal-title":"Nature Machine Intelligence"},{"key":"10.3233\/IA-210120_ref44","doi-asserted-by":"crossref","unstructured":"Sabbatini F. , Ciatto G. , Calegari R. and Omicini A. , On the design of PSyKE: A platform for symbolic knowledge extraction. In Roberta Calegari, Giovanni Ciatto, Enrico Denti, Andrea Omicini, and Giovanni Sartor, editors, WOA 2021 \u2013 22nd Workshop \u201cFrom Objects to Agents\u201d, volume of CEUR Workshop Proceedings, pp. 29\u201348. Sun SITE Central Europe, RWTH Aachen University, October 2021. 22ndWorkshop \u201cFrom Objects to Agents\u201d (WOA 2021), Bologna, Italy, 1\u20133 September 2021. Proceedings.","DOI":"10.3233\/IA-220141"},{"key":"10.3233\/IA-210120_ref45","doi-asserted-by":"crossref","unstructured":"Sabbatini F. , Ciatto G. and Omicini A. , GridEx: An algorithm for knowledge extraction from black-box regressors. In Davide Calvaresi, Amro Najjar, Michael Winikoff, and Kary Fr\u00e4mling, editors, Explainable and Transparent AI and Multi-Agent Systems. Third International Workshop, EXTRAAMAS 2021, Virtual Event, May 3\u20137, 2021, Revised Selected Papers, volume 12688 of LNCS, pp. 18\u2013 38. Springer Nature, Basel, Switzerland, 2021.","DOI":"10.1007\/978-3-030-82017-6_2"},{"issue":"10","key":"10.3233\/IA-210120_ref46","doi-asserted-by":"crossref","first-page":"1279","DOI":"10.1016\/S0893-6080(02)00089-8","article-title":"Extracting regression rules from neural networks","volume":"15","author":"Saito","year":"2002","journal-title":"Neural Networks"},{"key":"10.3233\/IA-210120_ref47","doi-asserted-by":"crossref","unstructured":"Gregor P. J. Schmitz, C. Aldrich and Fran\u00e7ois S. Gouws, ANN-DT: an algorithm for extraction of decision trees from artificial neural networks, IEEE Transactions on Neural Networks 10(6) (1999), 1392\u20131401.","DOI":"10.1109\/72.809084"},{"issue":"04","key":"10.3233\/IA-210120_ref48","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1142\/S0129065711002821","article-title":"Rule extraction from minimal neural networks for credit card screening","volume":"21","author":"Setiono","year":"2011","journal-title":"International Journal of Neural Systems"},{"issue":"3","key":"10.3233\/IA-210120_ref49","doi-asserted-by":"crossref","first-page":"564","DOI":"10.1109\/TNN.2002.1000125","article-title":"Extraction of rules from artificial neural networks for nonlinear regression","volume":"13","author":"Setiono","year":"2002","journal-title":"IEEE Transactions on Neural Networks"},{"issue":"5A","key":"10.3233\/IA-210120_ref51","first-page":"6","article-title":"Using neural network rule extraction for credit-risk evaluation","volume":"6","author":"Steiner","year":"2006","journal-title":"International Journal of Computer Science and Network Security"},{"issue":"1","key":"10.3233\/IA-210120_ref52","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/BF00993103","article-title":"Extracting refined rules from knowledge-based neural networks","volume":"13","author":"Towell","year":"1993","journal-title":"Machine Learning"},{"key":"10.3233\/IA-210120_ref53","doi-asserted-by":"crossref","unstructured":"Yang Y. , Webb G.I. and Wu X. , Discretization methods. In Oded Maimon and Lior Rokach, editors, Data Mining and Knowledge Discovery Handbook, 2nd ed, pp. 101\u2013116. Springer, 2010.","DOI":"10.1007\/978-0-387-09823-4_6"}],"container-title":["Intelligenza Artificiale"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/IA-210120","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T06:59:32Z","timestamp":1733122772000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/IA-210120"}},"subtitle":[],"editor":[{"given":"Roberta","family":"Calegari","sequence":"additional","affiliation":[]},{"given":"Giovanni","family":"Ciatto","sequence":"additional","affiliation":[]},{"given":"Andrea","family":"Omicini","sequence":"additional","affiliation":[]},{"given":"Giuseppe","family":"Vizzari","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,7,8]]},"references-count":50,"journal-issue":{"issue":"1"},"URL":"https:\/\/doi.org\/10.3233\/ia-210120","relation":{},"ISSN":["1724-8035","2211-0097"],"issn-type":[{"type":"print","value":"1724-8035"},{"type":"electronic","value":"2211-0097"}],"subject":[],"published":{"date-parts":[[2022,7,8]]}}}