{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,12]],"date-time":"2025-04-12T02:07:37Z","timestamp":1744423657210,"version":"3.37.3"},"reference-count":36,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T00:00:00Z","timestamp":1705622400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"Anomaly detection plays a critical role in ensuring safe, smooth, and efficient operation of machinery and equipment in industrial environments. With the wide deployment of multimodal sensors and the rapid development of Internet of Things (IoT), the data generated in modern industrial production has become increasingly diverse and complex. However, traditional methods for anomaly detection based on a single data source cannot fully utilize multimodal data to capture anomalies in industrial systems. To address this challenge, we propose a new model for anomaly detection in industrial environments using multimodal temporal data. This model integrates an attention-based autoencoder (AAE) and a generative adversarial network (GAN) to capture and fuse rich information from different data sources. Specifically, the AAE captures time-series dependencies and relevant features in each modality, and the GAN introduces adversarial regularization to enhance the model\u2019s ability to reconstruct normal time-series data. We conduct extensive experiments on real industrial data containing both measurements from a distributed control system (DCS) and acoustic signals, and the results demonstrate the performance superiority of the proposed model over the state-of-the-art TimesNet for anomaly detection, with an improvement of 5.6% in F1 score.<\/jats:p>","DOI":"10.3390\/s24020637","type":"journal-article","created":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T08:33:41Z","timestamp":1705653221000},"page":"637","source":"Crossref","is-referenced-by-count":16,"title":["MFGAN: Multimodal Fusion for Industrial Anomaly Detection Using Attention-Based Autoencoder and Generative Adversarial Network"],"prefix":"10.3390","volume":"24","author":[{"given":"Xinji","family":"Qu","sequence":"first","affiliation":[{"name":"School of Information Science and Technology, Northwest University, Xi\u2019an 710127, China"}]},{"given":"Zhuo","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Northwest University, Xi\u2019an 710127, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8218-1209","authenticated-orcid":false,"given":"Chase Q.","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Data Science, New Jersey Institute of Technology, Newark, NJ 07102, USA"}]},{"given":"Aiqin","family":"Hou","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Northwest University, Xi\u2019an 710127, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3265-2954","authenticated-orcid":false,"given":"Xiaoyan","family":"Yin","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Northwest University, Xi\u2019an 710127, China"}]},{"given":"Zhulian","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Information Science and Technology, Northwest University, Xi\u2019an 710127, China"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,19]]},"reference":[{"key":"ref_1","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_2","unstructured":"Xu, J., Wu, H., Wang, J., and Long, M. (2021). Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv."},{"key":"ref_3","first-page":"22419","article-title":"Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting","volume":"34","author":"Wu","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_4","unstructured":"Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. (2022, January 17\u201323). Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wang, Y., Peng, J., Zhang, J., Yi, R., Wang, Y., and Wang, C. (2023, January 17\u201324). Multimodal Industrial Anomaly Detection via Hybrid Fusion. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00776"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Breunig, M.M., Kriegel, H.P., Ng, R.T., and Sander, J. (2000, January 14\u201318). LOF: Identifying density-based local outliers. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, Dallas, TX, USA.","DOI":"10.1145\/342009.335388"},{"key":"ref_7","unstructured":"Tang, J., Chen, Z., Fu, A.W.C., and Cheung, D.W. (2002, January 6\u20138). Enhancing effectiveness of outlier detections for low density patterns. Proceedings of the Advances in Knowledge Discovery and Data Mining: 6th Pacific-Asia Conference, PAKDD 2002, Taipei, Taiwan. Proceedings 6."},{"key":"ref_8","unstructured":"Zong, B., Song, Q., Min, M.R., Cheng, W., Lumezanu, C., Cho, D., and Chen, H. (May, January 30). Deep autoencoding gaussian mixture model for unsupervised anomaly detection. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the EM algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"J. R. Stat. Soc. Ser. B (Methodol.)"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1109\/LRA.2018.2801475","article-title":"A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder","volume":"3","author":"Park","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_12","unstructured":"Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Su, Y., Zhao, Y., Niu, C., Liu, R., Sun, W., and Pei, D. (2019, January 4\u20138). Robust anomaly detection for multivariate time series through stochastic recurrent neural network. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.","DOI":"10.1145\/3292500.3330672"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.","DOI":"10.3115\/v1\/D14-1179"},{"key":"ref_15","unstructured":"Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. (2022). TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1145\/3422622","article-title":"Generative adversarial networks","volume":"63","author":"Goodfellow","year":"2020","journal-title":"Commun. ACM"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Schlegl, T., Seeb\u00f6ck, P., Waldstein, S.M., Schmidt-Erfurth, U., and Langs, G. (2017, January 25\u201330). Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. Proceedings of the International Conference on Information Processing in Medical Imaging, Boone, NC, USA.","DOI":"10.1007\/978-3-319-59050-9_12"},{"key":"ref_18","unstructured":"Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv."},{"key":"ref_19","unstructured":"Zenati, H., Foo, C.S., Lecouat, B., Manek, G., and Chandrasekhar, V.R. (2018). Efficient gan-based anomaly detection. arXiv."},{"key":"ref_20","unstructured":"Donahue, J., Kr\u00e4henb\u00fchl, P., and Darrell, T. (2016). Adversarial feature learning. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.media.2019.01.010","article-title":"f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks","volume":"54","author":"Schlegl","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhou, B., Liu, S., Hooi, B., Cheng, X., and Ye, J. (2019, January 10\u201316). BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series. Proceedings of the IJCAI, Macao, China.","DOI":"10.24963\/ijcai.2019\/616"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Chen, X., Deng, L., Huang, F., Zhang, C., Zhang, Z., Zhao, Y., and Zheng, K. (2021, January 19\u201322). Daemon: Unsupervised anomaly detection and interpretation for multivariate time series. Proceedings of the 2021 IEEE 37th International Conference on Data Engineering (ICDE), Chania, Greece.","DOI":"10.1109\/ICDE51399.2021.00228"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, D., Chen, D., Jin, B., Shi, L., Goh, J., and Ng, S.K. (2019, January 17\u201319). MAD-GAN: Multivariate anomaly detection for time series data with generative adversarial networks. Proceedings of the Artificial Neural Networks and Machine Learning\u2013ICANN 2019: Text and Time Series: 28th International Conference on Artificial Neural Networks, Munich, Germany. Proceedings, Part IV.","DOI":"10.1007\/978-3-030-30490-4_56"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Geiger, A., Liu, D., Alnegheimish, S., Cuesta-Infante, A., and Veeramachaneni, K. (2020, January 10\u201313). Tadgan: Time series anomaly detection using generative adversarial networks. Proceedings of the 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA.","DOI":"10.1109\/BigData50022.2020.9378139"},{"key":"ref_26","unstructured":"Box, G.E., Jenkins, G.M., Reinsel, G.C., and Ljung, G.M. (2015). Time Series Analysis: Forecasting and Control, John Wiley & Sons."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hundman, K., Constantinou, V., Laporte, C., Colwell, I., and Soderstrom, T. (2018, January 19\u201323). Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3219845"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1991","DOI":"10.1109\/ACCESS.2018.2886457","article-title":"DeepAnT: A deep learning approach for unsupervised anomaly detection in time series","volume":"7","author":"Munir","year":"2018","journal-title":"IEEE Access"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Xie, J., Cui, Y., Huang, F., Liu, C., and Zheng, K. (2022, January 17\u201321). MARINA: An MLP-Attention Model for Multivariate Time-Series Analysis. Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA.","DOI":"10.1145\/3511808.3557386"},{"key":"ref_30","first-page":"423","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Ahuja","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1109\/6046.807953","article-title":"Multimodal integration\u2014A statistical view","volume":"1","author":"Wu","year":"1999","journal-title":"IEEE Trans. Multimed."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"63373","DOI":"10.1109\/ACCESS.2019.2916887","article-title":"Deep multimodal representation learning: A survey","volume":"7","author":"Guo","year":"2019","journal-title":"IEEE Access"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Bai, H., Zhang, J., Zhang, Y., Xu, S., Lin, Z., Timofte, R., and Van Gool, L. (2023, January 17\u201324). Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.00572"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhao, J., Zhao, Y., and Li, J. (2021, January 20\u201324). M3tr: Multi-modal multi-label recognition with transformer. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.","DOI":"10.1145\/3474085.3475191"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ju, X., Zhang, D., Li, J., and Zhou, G. (2020, January 12\u201316). Transformer-based label set generation for multi-modal multi-label emotion detection. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.","DOI":"10.1145\/3394171.3413577"},{"key":"ref_36","first-page":"3","article-title":"Autoencoders, minimum description length and Helmholtz free energy","volume":"6","author":"Hinton","year":"1993","journal-title":"Adv. Neural Inf. Process. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/2\/637\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T15:18:53Z","timestamp":1731079133000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/2\/637"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,19]]},"references-count":36,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["s24020637"],"URL":"https:\/\/doi.org\/10.3390\/s24020637","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2024,1,19]]}}}