{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,1,29]],"date-time":"2024-01-29T14:42:58Z","timestamp":1706539378070},"reference-count":51,"publisher":"Springer Science and Business Media LLC","issue":"15","license":[{"start":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T00:00:00Z","timestamp":1675900800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T00:00:00Z","timestamp":1675900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["825619"],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007601","name":"Horizon 2020","doi-asserted-by":"publisher","award":["952215"],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003130","name":"Fonds Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["1239422N"],"id":[{"id":"10.13039\/501100003130","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2023,8]]},"abstract":"Abstract<\/jats:title>Injecting prior knowledge into the learning process of a neural architecture is one of the main challenges currently faced by the artificial intelligence community, which also motivated the emergence of neural-symbolic models. One of the main advantages of these approaches is their capacity to learn competitive solutions with a significant reduction of the amount of supervised data. In this regard, a commonly adopted solution consists of representing the prior knowledge via first-order logic formulas, then relaxing the formulas into a set of differentiable constraints by using a t-norm fuzzy logic. This paper shows that this relaxation, together with the choice of the penalty terms enforcing the constraint satisfaction, can be unambiguously determined by the selection of a t-norm generator, providing numerical simplification properties and a tighter integration between the logic knowledge and the learning objective. When restricted to supervised learning, the presented theoretical framework provides a straight derivation of the popular cross-entropy loss, which has been shown to provide faster convergence and to reduce the vanishing gradient problem in very deep structures. However, the proposed learning formulation extends the advantages of the cross-entropy loss to the general knowledge that can be represented by neural-symbolic methods. In addition, the presented methodology allows the development of novel classes of loss functions, which are shown in the experimental results to lead to faster convergence rates than the approaches previously proposed in the literature.<\/jats:p>","DOI":"10.1007\/s10489-022-04383-6","type":"journal-article","created":{"date-parts":[[2023,2,10]],"date-time":"2023-02-10T02:14:13Z","timestamp":1675995253000},"page":"18775-18789","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["T-norms driven loss functions for machine learning"],"prefix":"10.1007","volume":"53","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-8492-8110","authenticated-orcid":false,"given":"Francesco","family":"Giannini","sequence":"first","affiliation":[]},{"given":"Michelangelo","family":"Diligenti","sequence":"additional","affiliation":[]},{"given":"Marco","family":"Maggini","sequence":"additional","affiliation":[]},{"given":"Marco","family":"Gori","sequence":"additional","affiliation":[]},{"given":"Giuseppe","family":"Marra","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,2,9]]},"reference":[{"issue":"7553","key":"4383_CR1","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436","journal-title":"Nature"},{"key":"4383_CR2","unstructured":"Selbst A, Powles J (2018) meaningful information and the right to explanation. In: Conference on fairness, accountability and transparency. PMLR, pp 48\u201348"},{"key":"4383_CR3","doi-asserted-by":"crossref","unstructured":"De Raedt L, Duman\u010di\u0107 S, Manhaeve R, Marra G (2021) From statistical relational to neural-symbolic artificial intelligence. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 4943\u20134950","DOI":"10.24963\/ijcai.2020\/688"},{"issue":"4","key":"4383_CR4","first-page":"611","volume":"6","author":"A Garcez","year":"2019","unstructured":"Garcez A, Gori M, Lamb L, Serafini L, Spranger M, Tran S (2019) Neural-symbolic computing: an effective methodology for principled integration of machine learning and reasoning. Journal of Applied Logics 6(4):611\u2013631","journal-title":"Journal of Applied Logics"},{"key":"4383_CR5","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1016\/j.artint.2015.08.011","volume":"244","author":"M Diligenti","year":"2017","unstructured":"Diligenti M, Gori M, Sacca C (2017) Semantic-based regularization for learning and inference. Artif Intell 244:143\u2013165","journal-title":"Artif Intell"},{"key":"4383_CR6","doi-asserted-by":"publisher","first-page":"103649","DOI":"10.1016\/j.artint.2021.103649","volume":"303","author":"S Badreddine","year":"2022","unstructured":"Badreddine S, Garcez AD, Serafini L, Spranger M (2022) Logic tensor networks. Artif Intell 303:103649","journal-title":"Artif Intell"},{"key":"4383_CR7","unstructured":"Goodfellow I, Bengio Y, Courville A (2016) Deep learning"},{"key":"4383_CR8","doi-asserted-by":"crossref","unstructured":"Giannini F, Marra G, Diligenti M, Maggini M, Gori M (2019) On the relation between loss functions and t-norms. In: Proceedings of the conference on inductive logic programming (ILP)","DOI":"10.1007\/978-3-030-49210-6_4"},{"key":"4383_CR9","first-page":"1","volume":"342","author":"AD Garcez","year":"2022","unstructured":"Garcez AD, Bader S, Bowman H, Lamb LC, De Penning L, Illuminoo B, Poon H, Gerson Zaverucha C (2022) Neural-symbolic learning and reasoning: a survey and interpretation. Neuro-Symbolic Artificial Intelligence: The State of the Art 342:1","journal-title":"Neuro-Symbolic Artificial Intelligence: The State of the Art"},{"key":"4383_CR10","doi-asserted-by":"crossref","unstructured":"Hitzler P (2022) Neuro-symbolic artificial intelligence: the state of the art","DOI":"10.3233\/FAIA342"},{"issue":"2","key":"4383_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-3-031-01574-8","volume":"10","author":"LD Raedt","year":"2016","unstructured":"Raedt LD, Kersting K, Natarajan S, Poole D (2016) Statistical relational artificial intelligence: logic, probability, and computation. Synthesis Lectures on Artificial Intelligence and Machine Learning 10 (2):1\u2013189","journal-title":"Synthesis Lectures on Artificial Intelligence and Machine Learning"},{"issue":"1","key":"4383_CR12","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s10994-006-5833-1","volume":"62","author":"M Richardson","year":"2006","unstructured":"Richardson M, Domingos P (2006) Markov logic networks. Mach Learn 62(1):107\u2013136","journal-title":"Mach Learn"},{"key":"4383_CR13","first-page":"1","volume":"18","author":"SH Bach","year":"2017","unstructured":"Bach SH, Broecheler M, Huang B, Getoor L (2017) Hinge-loss markov random fields and probabilistic soft logic. J Mach Learn Res 18:1\u201367","journal-title":"J Mach Learn Res"},{"key":"4383_CR14","doi-asserted-by":"crossref","unstructured":"Niu F, R\u00e9 C, Doan A, Shavlik J (2011) Tuffy: scaling up statistical inference in markov logic networks using an rdbms. Proceedings of the VLDB Endowment 4(6)","DOI":"10.14778\/1978665.1978669"},{"key":"4383_CR15","unstructured":"Chekol MW, Huber J, Meilicke C, Stuckenschmidt H (2016) Markov logic networks with numerical constraints. In: Proceedings of the twenty-second european conference on artificial intelligence, pp 1017\u20131025"},{"key":"4383_CR16","unstructured":"Qu M, Bengio Y, Tang J (2019) Gmnn: graph markov neural networks. In: International conference on machine learning. PMLR, pp 5241\u20135250"},{"key":"4383_CR17","doi-asserted-by":"crossref","unstructured":"Khot T, Balasubramanian N, Gribkoff E, Sabharwal A, Clark P, Etzioni O (2015) Exploring markov logic networks for question answering. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp 685\u2013694","DOI":"10.18653\/v1\/D15-1080"},{"key":"4383_CR18","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1016\/j.knosys.2017.01.025","volume":"121","author":"K Gayathri","year":"2017","unstructured":"Gayathri K, Easwarakumar K, Elias S (2017) Probabilistic ontology based activity recognition in smart homes using markov logic network. Knowl-Based Syst 121:173\u2013184","journal-title":"Knowl-Based Syst"},{"key":"4383_CR19","unstructured":"Marra G, Ku\u017eelka O (2021) Neural markov logic networks. In: Uncertainty in artificial intelligence. PMLR, pp 908\u2013917"},{"key":"4383_CR20","doi-asserted-by":"crossref","unstructured":"Diligenti M, Giannini F, Gori M, Maggini M, Marra G (2021) A constraint-based approach to learning and reasoning. In: Neuro-symbolic artificial intelligence: the state of the art, pp 192\u2013 213","DOI":"10.3233\/FAIA210355"},{"key":"4383_CR21","doi-asserted-by":"crossref","unstructured":"Marra G, Giannini F, Diligenti M, Gori M (2019) Lyrics: a general interface layer to integrate logic inference and deep learning. In: Proceedings of the joint european conference on machine learning and knowledge discovery in databases (ECML\/PKDD)","DOI":"10.1007\/978-3-030-46147-8_17"},{"key":"4383_CR22","unstructured":"Xu J, Zhang Z, Friedman T, Liang Y, Broeck G (2018) A semantic loss function for deep learning with symbolic knowledge. In: International conference on machine learning. PMLR, pp 5502\u20135511"},{"key":"4383_CR23","unstructured":"van Krieken E, Acar E, van Harmelen F (2019) Semi-supervised learning using differentiable reasoning. Journal of Applied Logics\u2014IfCoLog Journal of Logics and their Applications 6(4)"},{"key":"4383_CR24","doi-asserted-by":"crossref","unstructured":"Marra G, Giannini F, Diligenti M, Gori M (2019) Integrating learning and reasoning with deep logic models. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 517\u2013532","DOI":"10.1007\/978-3-030-46147-8_31"},{"key":"4383_CR25","unstructured":"Marra G, Diligenti M, Giannini F, Gori M, Maggini M (2020) Relational neural machines. In: Proceedings of the European conference on artificial intelligence (ECAI)"},{"key":"4383_CR26","unstructured":"Manhaeve R, Dumancic S, Kimmig A, Demeester T, De Raedt L (2018) Deepproblog: neural probabilistic logic programming. Adv Neural Inf Process Syst 31"},{"key":"4383_CR27","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1613\/jair.1.11203","volume":"62","author":"G Sourek","year":"2018","unstructured":"Sourek G, Aschenbrenner V, Zelezny F, Schockaert S, Kuzelka O (2018) Lifted relational neural networks: efficient learning of latent relational structures. J Artif Intell Res 62:69\u2013100","journal-title":"J Artif Intell Res"},{"key":"4383_CR28","unstructured":"Rockt\u00e4schel T, Riedel S (2017) End-to-end differentiable proving. In: Advances in neural information processing systems, pp 3788\u20133800"},{"key":"4383_CR29","doi-asserted-by":"crossref","unstructured":"Minervini P, Riedel S, Stenetorp P, Grefenstette E, Rockt\u00e4schel T (2020) Learning reasoning strategies in end-to-end differentiable proving. In: ICML","DOI":"10.3233\/FAIA210359"},{"key":"4383_CR30","doi-asserted-by":"crossref","unstructured":"Serafini L, Donadello I, Garcez AD (2017) Learning and reasoning in logic tensor networks: theory and application to semantic image interpretation. In: Proceedings of the symposium on applied computing. ACM, pp 125\u2013130","DOI":"10.1145\/3019612.3019642"},{"key":"4383_CR31","doi-asserted-by":"crossref","unstructured":"Giannini F, Diligenti M, Gori M, Maggini M (2018) On a convex logic fragment for learning and reasoning. IEEE Transactions on Fuzzy Systems","DOI":"10.1109\/TFUZZ.2018.2879627"},{"key":"4383_CR32","doi-asserted-by":"crossref","unstructured":"van Krieken E, Acar E, van Harmelen F (2020) Analyzing differentiable fuzzy implications. In: KR2020: Proceedings of the 17th Conference on Principles of Knowledge Representation and Reasoning. Rhodes, Greece. September 12\u201318, 2020. IJCAI Organization, pp 893\u2013903","DOI":"10.24963\/kr.2020\/92"},{"key":"4383_CR33","doi-asserted-by":"publisher","first-page":"103602","DOI":"10.1016\/j.artint.2021.103602","volume":"302","author":"E van Krieken","year":"2022","unstructured":"van Krieken E, Acar E, van Harmelen F (2022) Analyzing differentiable fuzzy logic operators. Artif Intell 302:103602","journal-title":"Artif Intell"},{"key":"4383_CR34","doi-asserted-by":"crossref","unstructured":"Donadello I, Serafini L, d\u2019Avila Garcez A (2017) Logic tensor networks for semantic image interpretation. In: IJCAI International joint conference on artificial intelligence, pp 1596\u20131602","DOI":"10.24963\/ijcai.2017\/221"},{"key":"4383_CR35","unstructured":"Klement EP, Mesiar R, Pap E (2013) Triangular norms 8"},{"key":"4383_CR36","unstructured":"H\u00e1jek P. (2013) Metamathematics of Fuzzy Logic 4"},{"issue":"2","key":"4383_CR37","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1016\/S0165-0114(01)00040-9","volume":"126","author":"S Jenei","year":"2002","unstructured":"Jenei S (2002) A note on the ordinal sum theorem and its consequence for the construction of triangular norms. Fuzzy Sets Syst 126(2):199\u2013205","journal-title":"Fuzzy Sets Syst"},{"issue":"2","key":"4383_CR38","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1016\/0165-0114(89)90005-5","volume":"31","author":"M Mizumoto","year":"1989","unstructured":"Mizumoto M (1989) Pictorial representations of fuzzy connectives, part i: cases of t-norms, t-conorms and averaging operators. Fuzzy Sets Syst 31(2):217\u2013242","journal-title":"Fuzzy Sets Syst"},{"key":"4383_CR39","doi-asserted-by":"crossref","unstructured":"Marra G, Giannini F, Diligenti M, Gori M (2019) Constraint-based visual generation. In: International conference on artificial neural networks. Springer, pp 565\u2013577","DOI":"10.1007\/978-3-030-30508-6_45"},{"key":"4383_CR40","unstructured":"Diligenti M, Roychowdhury S, Gori M (2018) Image classification using deep learning and prior knowledge. In: Proceedings of third international workshop on declarative learning based programming (DeLBP)"},{"key":"4383_CR41","unstructured":"Nov\u00e1k V., Perfilieva I, Mockor J (2012) Mathematical Principles of Fuzzy Logic 517"},{"key":"4383_CR42","doi-asserted-by":"crossref","unstructured":"Kolb S, Teso S, Passerini A, De Raedt L (2018) Learning smt (lra) constraints using smt solvers. In: IJCAI, pp 2333\u20132340","DOI":"10.24963\/ijcai.2018\/323"},{"key":"4383_CR43","unstructured":"Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: OSDI, vol 16, pp 265\u2013283"},{"key":"4383_CR44","doi-asserted-by":"publisher","unstructured":"Fakhraei S, Foulds J, Shashanka M, Getoor L (2015) Collective spammer detection in evolving multi-relational social networks. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. KDD \u201915, pp 1769\u20131778, DOI https:\/\/doi.org\/10.1145\/2783258.2788606, (to appear in print)","DOI":"10.1145\/2783258.2788606"},{"key":"4383_CR45","unstructured":"Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. arXiv:1412.6980"},{"issue":"Nov","key":"4383_CR46","first-page":"2399","volume":"7","author":"M Belkin","year":"2006","unstructured":"Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7(Nov):2399\u20132434","journal-title":"J Mach Learn Res"},{"key":"4383_CR47","unstructured":"Neville J, Jensen D (2000) Iterative classification in relational data. In: Proc. AAAI-2000 workshop on learning statistical models from relational data, pp 13\u201320"},{"key":"4383_CR48","unstructured":"Lu Q, Getoor L (2003) Link-based classification. In: Proceedings of the 20th international conference on machine learning (ICML-03), pp 496\u2013503"},{"issue":"3","key":"4383_CR49","first-page":"93","volume":"29","author":"P Sen","year":"2008","unstructured":"Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI Mag 29(3):93","journal-title":"AI Mag"},{"key":"4383_CR50","doi-asserted-by":"crossref","unstructured":"Ketkar N (2017) Introduction to pytorch. In: Deep learning with python, pp 195\u2013208","DOI":"10.1007\/978-1-4842-2766-4_12"},{"key":"4383_CR51","unstructured":"Yi K, Wu J, Gan C, Torralba A, Kohli P, Tenenbaum JB (2018) Neural-Symbolic VQA: disentangling reasoning from vision and language understanding. In: Advances in neural information processing systems (NIPS)"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-022-04383-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-022-04383-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-022-04383-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,7]],"date-time":"2023-07-07T07:07:06Z","timestamp":1688713626000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-022-04383-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,9]]},"references-count":51,"journal-issue":{"issue":"15","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["4383"],"URL":"https:\/\/doi.org\/10.1007\/s10489-022-04383-6","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,9]]},"assertion":[{"value":"5 December 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 February 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interests"}}]}}