{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,30]],"date-time":"2024-08-30T11:40:50Z","timestamp":1725018050481},"reference-count":45,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T00:00:00Z","timestamp":1631664000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001866","name":"Fonds National de la Recherche Luxembourg","doi-asserted-by":"publisher","award":["BRIDGES19\/IS\/13706587"],"id":[{"id":"10.13039\/501100001866","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"Artificial Intelligence (AI) is one of the hottest topics in our society, especially when it comes to solving data-analysis problems. Industry are conducting their digital shifts, and AI is becoming a cornerstone technology for making decisions out of the huge amount of (sensors-based) data available in the production floor. However, such technology may be disappointing when deployed in real conditions. Despite good theoretical performances and high accuracy when trained and tested in isolation, a Machine-Learning (M-L) model may provide degraded performances in real conditions. One reason may be fragility in treating properly unexpected or perturbed data. The objective of the paper is therefore to study the robustness of seven M-L and Deep-Learning (D-L) algorithms, when classifying univariate time-series under perturbations. A systematic approach is proposed for artificially injecting perturbations in the data and for evaluating the robustness of the models. This approach focuses on two perturbations that are likely to happen during data collection. Our experimental study, conducted on twenty sensors\u2019 datasets from the public University of California Riverside (UCR) repository, shows a great disparity of the models\u2019 robustness under data quality degradation. Those results are used to analyse whether the impact of such robustness can be predictable\u2014thanks to decision trees\u2014which would prevent us from testing all perturbations scenarios. Our study shows that building such a predictor is not straightforward and suggests that such a systematic approach needs to be used for evaluating AI models\u2019 robustness.<\/jats:p>","DOI":"10.3390\/s21186195","type":"journal-article","created":{"date-parts":[[2021,9,15]],"date-time":"2021-09-15T16:00:44Z","timestamp":1631721644000},"page":"6195","source":"Crossref","is-referenced-by-count":7,"title":["A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"http:\/\/orcid.org\/0000-0001-7319-3921","authenticated-orcid":false,"given":"Paul-Lou","family":"Benedick","sequence":"first","affiliation":[{"name":"Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, 6 Rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, Luxembourg"}]},{"given":"J\u00e9r\u00e9my","family":"Robert","sequence":"additional","affiliation":[{"name":"Cebi Luxembourg S.A, 30 rue J.F. Kennedy, L-7327 Steinsel, Luxembourg"}]},{"given":"Yves","family":"Le Traon","sequence":"additional","affiliation":[{"name":"Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, 6 Rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, Luxembourg"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1645","DOI":"10.1016\/j.future.2013.01.010","article-title":"Internet of Things (IoT): A vision, architectural elements, and future directions","volume":"29","author":"Gubbi","year":"2013","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1049\/trit.2018.1008","article-title":"Artificial intelligence in Internet of things","volume":"3","author":"Ghosh","year":"2018","journal-title":"CAAI Trans. Intell. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2787","DOI":"10.1016\/j.comnet.2010.05.010","article-title":"The internet of things: A survey","volume":"54","author":"Atzori","year":"2010","journal-title":"Comput. Netw."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Breivold, H.P., and Sandstr\u00f6m, K. (2015, January 11\u201313). Internet of Things for Industrial Automation\u2013Challenges and Technical Solutions. Proceedings of the IEEE International Conference on Data Science and Data Intensive Systems (DSDIS), Sydney, Australia.","DOI":"10.1109\/DSDIS.2015.11"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Varghese, A., and Tandur, D. (2014, January 27\u201329). Wireless requirements and challenges in Industry 4.0. Proceedings of the International Conference on Contemporary Computing and Informatics (IC3I), Mysuru, India.","DOI":"10.1109\/IC3I.2014.7019732"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Khan, W.A., Wisniewski, L., Lang, D., and Jasperneite, J. (2017, January 19\u201321). Analysis of the requirements for offering industrie 4.0 applications as a cloud service. Proceedings of the 2017 IEEE 26th International Symposium on Industrial Electronics (ISIE), Edinburgh, UK.","DOI":"10.1109\/ISIE.2017.8001413"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1142\/S0219622006002258","article-title":"10 challenging problems in data mining research","volume":"5","author":"Yang","year":"2006","journal-title":"Int. J. Inf. Technol. Decis. Mak."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1080\/07421222.1996.11518099","article-title":"Beyond accuracy: What data quality means to data consumers","volume":"12","author":"Wang","year":"1996","journal-title":"J. Manag. Inf. Syst."},{"key":"ref_9","unstructured":"Foidl, H., and Felderer, M. Risk-based data validation in machine learning-based software systems. Proceedings of the 3rd ACM SIGSOFT International Workshop on Machine Learning Techniques for Software Quality Evaluation."},{"key":"ref_10","first-page":"1","article-title":"Challenges for quality of data in smart cities","volume":"6","author":"Barnaghi","year":"2015","journal-title":"J. Data Inf. Qual. JDIQ"},{"key":"ref_11","unstructured":"Griggs, T., and Wakabayashi, D. (The New York Times, 2018). How a Self-Driving Uber Killed a Pedestrian in Arizona, The New York Times."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1016\/j.mfglet.2018.09.002","article-title":"Industrial Artificial Intelligence for industry 4.0-based manufacturing systems","volume":"18","author":"Lee","year":"2018","journal-title":"Manuf. Lett."},{"key":"ref_13","first-page":"1","article-title":"Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations","volume":"10","author":"Gudivada","year":"2017","journal-title":"Int. J. Adv. Softw."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1781","DOI":"10.14778\/3229863.3229867","article-title":"Automating large-scale data quality verification","volume":"11","author":"Schelter","year":"2018","journal-title":"Proc. VLDB Endow."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"699","DOI":"10.1109\/TII.2011.2166788","article-title":"How to access factory floor information using internet technologies and gateways","volume":"7","author":"Sauter","year":"2011","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Benedick, P.L., Robert, J., and Le Traon, Y. (2019, January 14\u201317). TRIDENT: A Three-Steps Strategy to Digitise an Industrial System for Stepping into Industry 4.0. Proceedings of the 2019-45th Annual Conference of the IEEE Industrial Electronics Society (IECON), Lisbon, Portugal.","DOI":"10.1109\/IECON.2019.8927010"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Benedick, P.L., Robert, J., Le Traon, Y., and Kubler, S. (2018, January 15\u201318). O-MI\/O-DF vs. MQTT: A performance analysis. Proceedings of the IEEE Industrial Cyber-Physical Systems (ICPS), Saint Petersburg, Russia.","DOI":"10.1109\/ICPHYS.2018.8387652"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Sch\u00e4fer, P., and Leser, U. (2017, January 6\u201310). Fast and accurate time series classification with weasel. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore.","DOI":"10.1145\/3132847.3132980"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1109\/JAS.2019.1911747","article-title":"The UCR time series archive","volume":"6","author":"Dau","year":"2019","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1016\/j.jmsy.2018.01.003","article-title":"Deep learning for smart manufacturing: Methods and applications","volume":"48","author":"Wang","year":"2018","journal-title":"J. Manuf. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Luckow, A., Cook, M., Ashcraft, N., Weill, E., Djerekarov, E., and Vorster, B. (2016, January 5\u20138). Deep learning in the automotive industry: Applications and tools. Proceedings of the 2016 IEEE International Conference on Big Data (Big Data), Washington, DC, USA.","DOI":"10.1109\/BigData.2016.7841045"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1889","DOI":"10.1007\/s00170-019-03988-5","article-title":"A review of machine learning for the optimization of production processes","volume":"104","author":"Weichert","year":"2019","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.dcan.2017.10.002","article-title":"Machine learning for Internet of Things data analysis: A survey","volume":"4","author":"Mahdavinejad","year":"2018","journal-title":"Digit. Commun. Netw."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1007\/s10618-019-00619-1","article-title":"Deep learning for time series classification: A review","volume":"33","author":"Fawaz","year":"2019","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_25","unstructured":"Gamboa, J.C.B. (2017). Deep learning for time-series analysis. arXiv."},{"key":"ref_26","unstructured":"IEEE (1990). IEEE Standard Glossary of Software Engineering Terminology, IEEE. IEEE Std 610.12-1990."},{"key":"ref_27","unstructured":"Zhang, J.M., Harman, M., Ma, L., and Liu, Y. (2020). Machine learning testing: Survey, landscapes and horizons. IEEE Trans. Softw. Eng."},{"key":"ref_28","unstructured":"Huang, H., and Lin, S. WiDet: Wi-Fi Based Device-Free Passive Person Detection with Deep Convolutional Neural Networks. Proceedings of the 21st ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems."},{"key":"ref_29","unstructured":"Tanisaro, P., Mahner, F., and Heidemann, G. Quasi View-Independent Human Motion Recognition in Subspaces. Proceedings of the 9th International Conference on Machine Learning and Computing."},{"key":"ref_30","unstructured":"Chen, R., Jankovic, F., Marinsek, N., Foschini, L., Kourtis, L., Signorini, A., Pugh, M., Shen, J., Yaari, R., and Maljkovic, V. Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1080\/24725579.2019.1583703","article-title":"Continuous monitoring and detection of post-traumatic stress disorder (PTSD) triggers among veterans: A supervised machine learning approach","volume":"9","author":"McDonald","year":"2019","journal-title":"IISE Trans. Healthc. Syst. Eng."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., and Muller, P. (2019, January 14\u201319). Adversarial Attacks on Deep Neural Networks for Time Series Classification. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8851936"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Karimi-Bidhendi, S., Munshi, F., and Munshi, A. (2018, January 10\u201313). Scalable Classification of Univariate and Multivariate Time Series. Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA.","DOI":"10.1109\/BigData.2018.8621889"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2247","DOI":"10.1007\/s11063-019-10007-x","article-title":"Online Learning for Time Series Prediction of AR Model with Missing Data","volume":"50","author":"Yang","year":"2019","journal-title":"Neural Process. Lett."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"106","DOI":"10.1016\/j.knosys.2013.04.021","article-title":"Instance selection for time series classification based on immune binary particle swarm optimization","volume":"49","author":"Zhai","year":"2013","journal-title":"Knowl.-Based Syst."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1016\/j.isprsjprs.2019.10.003","article-title":"A time-series classification approach based on change detection for rapid land cover mapping","volume":"158","author":"Yan","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2194","DOI":"10.1016\/j.engappai.2013.06.013","article-title":"Robust predictive model for evaluating breast cancer survivability","volume":"26","author":"Park","year":"2013","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/j.neunet.2020.04.030","article-title":"Robust image classification against adversarial attacks using elastic similarity measures between edge count sequences","volume":"128","author":"Oregi","year":"2020","journal-title":"Neural Netw."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1016\/j.eswa.2017.01.056","article-title":"A bio-statistical mining approach for classifying multivariate clinical time series data observed at irregular intervals","volume":"78","author":"Nancy","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1016\/j.patcog.2017.11.030","article-title":"Time series cluster kernel for learning similarities between multivariate time series with missing data","volume":"76","author":"Mikalsen","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.eswa.2017.05.052","article-title":"Non-linear classifiers applied to EEG analysis for epilepsy seizure detection","volume":"86","author":"Santofimia","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_42","unstructured":"Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. (2018). Robustness may be at odds with accuracy. arXiv."},{"key":"ref_43","unstructured":"Pearson, K. (1896). Mathematical contributions to the theory of evolution.\u2014III. Regression, heredity, and panmixia. Philosophical Transactions of the Royal Society of London, Royal Society of London for Improving Natural Knowledge. Series A, Containing Papers of a Mathematical or Physical Character."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1002\/cyto.a.20896","article-title":"Quantifying colocalization by correlation: The Pearson correlation coefficient is superior to the Mander\u2019s overlap coefficient","volume":"77","author":"Adler","year":"2010","journal-title":"Cytom. Part A"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"757","DOI":"10.1109\/TASL.2008.919072","article-title":"On the importance of the Pearson correlation coefficient in noise reduction","volume":"16","author":"Benesty","year":"2008","journal-title":"IEEE Trans. Audio Speech Lang. Process."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/18\/6195\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,18]],"date-time":"2024-07-18T19:21:30Z","timestamp":1721330490000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/18\/6195"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,15]]},"references-count":45,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["s21186195"],"URL":"https:\/\/doi.org\/10.3390\/s21186195","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,15]]}}}