{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T05:39:10Z","timestamp":1740893950037,"version":"3.38.0"},"reference-count":76,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2019,6,18]],"date-time":"2019-06-18T00:00:00Z","timestamp":1560816000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61703015"],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["SIMULATION"],"published-print":{"date-parts":[[2020,2]]},"abstract":" Measuring the credibility of a simulation model has always been challenging due to its growing uncertainty and complexity. During the past decades, plenty of metrics and evaluation procedures have been developed for evaluating different sorts of simulation models. Most of the existing research focuses on the direct comparison of numerical results with a group of reference data. However, it is sometimes unsuitable for evolving dynamic models such as the multi-agent models. With the same condition, both the practical system and the simulation model perform highly dynamic actions. The credibility of the model with insufficient information, non-stationary states and changing environment is unable to acquire through a direct pair comparison. This paper presents a pattern-based validation method to complementarily extract hidden patterns that exist in both a simulation model and its reference data, and assess the model performance in a different aspect. Firstly, multi-dimensional perceptually important points strategy is modified to find the patterns exist in time-serial data. Afterward, a pattern organizing topology is applied to automatically depict required pattern from reference data and assess the performance of the corresponding simulation model. The extensive case study on three simulation models shows the effectiveness of the proposed method. <\/jats:p>","DOI":"10.1177\/0037549719856100","type":"journal-article","created":{"date-parts":[[2019,6,18]],"date-time":"2019-06-18T11:59:06Z","timestamp":1560859146000},"page":"151-167","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":6,"title":["A pattern-based validation method for the credibility evaluation of simulation models"],"prefix":"10.1177","volume":"96","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3834-4602","authenticated-orcid":false,"given":"Yuanjun","family":"Laili","sequence":"first","affiliation":[{"name":"School of Automation Science and Electrical Engineering and Beijing Advanced Innovation Center for Big Data-based Precision Medicine, Beihang University, Beijing, China"}]},{"given":"Lin","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Automation Science and Electrical Engineering and Beijing Advanced Innovation Center for Big Data-based Precision Medicine, Beihang University, Beijing, China"}]},{"given":"Yongliang","family":"Luo","sequence":"additional","affiliation":[{"name":"China State Shipbuilding Corporation System Engineering Research Institute, Beijing, China"}]}],"member":"179","published-online":{"date-parts":[[2019,6,18]]},"reference":[{"key":"bibr1-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1002\/9781118152812"},{"key":"bibr2-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-009-0153-2"},{"key":"bibr3-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2013.06.040"},{"key":"bibr4-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1002\/9781119174882"},{"volume-title":"Simulation credibility: advances in verification, validation, and uncertainty quantification","year":"2016","author":"Mehta UB","key":"bibr5-0037549719856100"},{"volume-title":"Simulation modeling and analysis","year":"1991","author":"Law AM","key":"bibr6-0037549719856100"},{"key":"bibr7-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/MCOM.2014.6815915"},{"first-page":"38","volume-title":"Proceedings of the 18th winter simulation conference","author":"Balci O","key":"bibr8-0037549719856100"},{"first-page":"39","volume-title":"Proceedings of the 40th winter simulation conference","author":"Law AM","key":"bibr9-0037549719856100"},{"key":"bibr10-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1145\/268437.268462"},{"key":"bibr11-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1002\/9780470590621"},{"volume-title":"Systems and software verification: model-checking techniques and tools","year":"2013","author":"B\u00e9rard B","key":"bibr12-0037549719856100"},{"key":"bibr13-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21455-4_3"},{"key":"bibr14-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2015.7408153"},{"key":"bibr15-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/0377-2217(93)90316-F"},{"key":"bibr16-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1051\/matecconf\/20166305032"},{"key":"bibr17-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.58"},{"key":"bibr18-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2014.30"},{"key":"bibr19-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2584050"},{"key":"bibr20-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2015.7408297"},{"key":"bibr21-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2017.09.007"},{"key":"bibr22-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-008-0035-z"},{"key":"bibr23-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1142\/S1793962317410021"},{"volume-title":"DoD M&S standards and best practices guide","year":"2010","author":"US Department of Defense","key":"bibr24-0037549719856100"},{"volume-title":"Standard for verification and validation in computational fluid dynamics and heat transfer","year":"2009","author":"American Society of Mechanical Engineers","key":"bibr25-0037549719856100"},{"journal-title":"Guide for the verification and validation of computational fluid dynamics simulations","year":"1998","author":"American Institute of Aeronautics and Astronautics","key":"bibr26-0037549719856100"},{"volume-title":"Standard for models and simulations","year":"2016","author":"National Aeronautics and Space Administration","key":"bibr27-0037549719856100"},{"journal-title":"A credibility assessment scoring (CAS) process for mission risk management","year":"2016","author":"Colombano SP","key":"bibr28-0037549719856100"},{"key":"bibr29-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1002\/sys.21266"},{"key":"bibr30-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2012.20"},{"key":"bibr31-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.05.370"},{"key":"bibr32-0037549719856100","unstructured":"Niazi MA. Towards a novel unified framework for developing formal, network and validated agent-based simulation models of complex adaptive systems. arXiv preprint arXiv:1708.02357, 2017."},{"key":"bibr33-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-15096-3_2"},{"key":"bibr34-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2011.6147749"},{"volume-title":"Proceedings of the 3rd workshop on multi-agent system and simulation","year":"2009","author":"Niazi MA","key":"bibr35-0037549719856100"},{"key":"bibr36-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/s10494-015-9612-6"},{"key":"bibr37-0037549719856100","unstructured":"Mehrabadi NR, Wen B, Burgos R, et al. Verification, validation and uncertainty quantification (VV&UQ) framework applicable to power electronics systems. SAE Technical Paper 2014-01-2176, 2014."},{"volume-title":"Resource allocation framework: validation of numerical models of complex engineering systems against physical experiments","year":"2012","author":"Hegenderfer JL","key":"bibr38-0037549719856100"},{"key":"bibr39-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1177\/1548512914568657"},{"key":"bibr40-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2016.7822087"},{"key":"bibr41-0037549719856100","first-page":"2554","volume-title":"Proceedings of the 37th winter simulation conference","author":"Liu F","year":"2005"},{"key":"bibr42-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2014.11.020"},{"key":"bibr43-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1115\/1.4004223"},{"key":"bibr44-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1111\/exsy.12111"},{"key":"bibr45-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1115\/1.4036182"},{"key":"bibr46-0037549719856100","first-page":"1","volume":"56","author":"Xie Z","year":"2013","journal-title":"Sci China Inf Sci"},{"key":"bibr47-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2015.2418592"},{"key":"bibr48-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.10"},{"key":"bibr49-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2012.53"},{"key":"bibr50-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-47166-2_5"},{"journal-title":"Systems and software verification: model-checking techniques and tools","year":"2013","author":"B\u00e9rard B","key":"bibr51-0037549719856100"},{"volume-title":"Design and analysis of experiments","year":"2017","author":"Montgomery DC","key":"bibr52-0037549719856100"},{"key":"bibr53-0037549719856100","first-page":"398","volume-title":"Proceedings of the 2010 international conference on principles and practice of multi-agent systems","author":"Caillou P","year":"2010"},{"key":"bibr54-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-015-1929-5"},{"volume-title":"Sample survey principles and methods","year":"2003","author":"Barnett V","key":"bibr55-0037549719856100"},{"key":"bibr56-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1002\/sim.6787"},{"key":"bibr57-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2016.02.031"},{"key":"bibr58-0037549719856100","unstructured":"Soper DS. A-priori sample size calculator for multiple regression, www.danielsoper.com\/statcalc (2015, accessed 10 June 2019)."},{"key":"bibr59-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1057\/jos.2009.26"},{"key":"bibr60-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecolmodel.2013.11.024"},{"key":"bibr61-0037549719856100","doi-asserted-by":"publisher","DOI":"10.18564\/jasss.3127"},{"key":"bibr62-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/s11518-016-5303-7"},{"key":"bibr63-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2011.03.016"},{"volume-title":"CEAS 2013 \u2013 International Conference of the European Aerospace Societies","year":"2013","author":"Carlsson M","key":"bibr64-0037549719856100"},{"key":"bibr65-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1111\/risa.12121"},{"key":"bibr66-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.ress.2015.01.023"},{"key":"bibr67-0037549719856100","first-page":"359","volume-title":"KDD Workshop","volume":"10","author":"Berndt DJ","year":"1994"},{"key":"bibr68-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2008.11.030"},{"key":"bibr69-0037549719856100","first-page":"26","volume-title":"Workshop notes of KDD 2001 workshop on temporal data mining","author":"Fu T","year":"2001"},{"key":"bibr70-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1007\/s00500-013-0985-y"},{"key":"bibr71-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2007.07.008"},{"key":"bibr72-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.04.006"},{"key":"bibr73-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1162\/artl.2007.13.3.303"},{"key":"bibr74-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1016\/j.jtbi.2010.07.039"},{"key":"bibr75-0037549719856100","unstructured":"Izquierdo SS, Izquierdo LR. Conditional dissociation. Cyp+By, https:\/\/luis-r-izquierdo.github.io\/Cyp-By\/ (2009, accessed ?)."},{"key":"bibr76-0037549719856100","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2287884"}],"container-title":["SIMULATION"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0037549719856100","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0037549719856100","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0037549719856100","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,1]],"date-time":"2025-03-01T10:57:28Z","timestamp":1740826648000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0037549719856100"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,18]]},"references-count":76,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,2]]}},"alternative-id":["10.1177\/0037549719856100"],"URL":"https:\/\/doi.org\/10.1177\/0037549719856100","relation":{},"ISSN":["0037-5497","1741-3133"],"issn-type":[{"type":"print","value":"0037-5497"},{"type":"electronic","value":"1741-3133"}],"subject":[],"published":{"date-parts":[[2019,6,18]]}}}