{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,11,27]],"date-time":"2024-11-27T17:11:06Z","timestamp":1732727466264,"version":"3.29.0"},"reference-count":38,"publisher":"SAGE Publications","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IDA"],"published-print":{"date-parts":[[2024,9,19]]},"abstract":"For the aspect-based sentiment analysis task, traditional works are only for text modality. However, in social media scenarios, texts often contain abbreviations, clerical errors, or grammatical errors, which invalidate traditional methods. In this study, the cross-model hierarchical interactive fusion network incorporating an end-to-end approach is proposed to address this challenge. In the network, a feature attention module and a feature fusion module are proposed to obtain the multimodal interaction feature between the image modality and the text modality. Through the attention mechanism and gated fusion mechanism, these two modules realize the auxiliary function of image in the text-based aspect-based sentiment analysis task. Meanwhile, a boundary auxiliary module is used to explore the dependencies between two core subtasks of the aspect-based sentiment analysis. Experimental results on two publicly available multi-modal aspect-based sentiment datasets validate the effectiveness of the proposed approach.<\/jats:p>","DOI":"10.3233\/ida-230305","type":"journal-article","created":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T16:29:22Z","timestamp":1708705762000},"page":"1293-1308","source":"Crossref","is-referenced-by-count":0,"title":["A cross-model hierarchical interactive fusion network for end-to-end multimodal aspect-based sentiment analysis"],"prefix":"10.1177","volume":"28","author":[{"given":"Qing","family":"Zhong","sequence":"first","affiliation":[]},{"given":"Xinhui","family":"Shao","sequence":"additional","affiliation":[]}],"member":"179","reference":[{"key":"10.3233\/IDA-230305_ref1","doi-asserted-by":"crossref","first-page":"28750","DOI":"10.1109\/ACCESS.2022.3157712","article-title":"MEDT: Using multimodal encoding-decoding network as in transformer for multimodal sentiment analysis","volume":"10","author":"Qi","year":"2022","journal-title":"IEEE Access"},{"key":"10.3233\/IDA-230305_ref2","doi-asserted-by":"crossref","unstructured":"G. Chandrasekaran, T.N. Nguyen and D.J. Hemanth, Multimodal sentimental analysis for social media applications: A comprehensive review, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 11 (2021).","DOI":"10.1002\/widm.1415"},{"key":"10.3233\/IDA-230305_ref3","doi-asserted-by":"crossref","unstructured":"Q.C. Li, A. Stefani, G. Toto et al., Towards Multimodal Sentiment Analysis Inspired by the Quantum Theoretical Framework, in: 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), 2020. pp. 177\u2013180.","DOI":"10.1109\/MIPR49039.2020.00044"},{"key":"10.3233\/IDA-230305_ref4","doi-asserted-by":"crossref","unstructured":"N. Xu, W. Mao and G. Chen, A Co-Memory Network for Multimodal Sentiment Analysis, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018. pp. 177\u2013180.","DOI":"10.1145\/3209978.3210093"},{"key":"10.3233\/IDA-230305_ref6","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1109\/MSP.2021.3106895","article-title":"Emotion recognition from multiple modalities: Fundamentals and methodologies","volume":"38","author":"Zhao","year":"2021","journal-title":"IEEE Signal Processing Magazine"},{"key":"10.3233\/IDA-230305_ref7","doi-asserted-by":"crossref","first-page":"1966","DOI":"10.1109\/TAFFC.2022.3171091","article-title":"Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis","volume":"14","author":"Y","year":"2023","journal-title":"IEEE Transactions on Affective Computing"},{"key":"10.3233\/IDA-230305_ref8","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume":"1","author":"Devlin","year":"2019","journal-title":"North American Chapter of the Association for Computational Linguistics"},{"key":"10.3233\/IDA-230305_ref9","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.neucom.2020.05.047","article-title":"Deep memory network with Bi-LSTM for personalized context-aware citation recommendation","volume":"410","author":"Wang","year":"2020","journal-title":"Neurocomputing"},{"key":"10.3233\/IDA-230305_ref10","doi-asserted-by":"crossref","unstructured":"G.M. Chen, Y.H. Tian and Y. Song, Joint Aspect Extraction and Sentiment Analysis with Directional Graph Convolutional Networks, in: International Conference on Computational Linguistics, 2020. pp. 272\u2013279.","DOI":"10.18653\/v1\/2020.coling-main.24"},{"key":"10.3233\/IDA-230305_ref11","doi-asserted-by":"crossref","first-page":"664","DOI":"10.26599\/TST.2021.9010055","article-title":"Cross-modal complementary network with hierarchical fusion for multimodal sentiment classification","volume":"27","author":"Peng","year":"2022","journal-title":"Tsinghua Science and Technology"},{"key":"10.3233\/IDA-230305_ref12","doi-asserted-by":"crossref","unstructured":"N. Xu, W.J. Mao and G.D. Chen, Multi-interactive memory network for aspect based multimodal sentiment analysis, in: AAAI Conference on Artificial Intelligence, Vol. 46, 2019. pp. 371\u2013378.","DOI":"10.1609\/aaai.v33i01.3301371"},{"key":"10.3233\/IDA-230305_ref13","doi-asserted-by":"crossref","unstructured":"A. Kumar and J. Vepa, Gated Mechanism for Attention Based Multi Modal Sentiment Analysis, in: ICASSP 2020\u20132020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. pp. 4477\u20134481.","DOI":"10.1109\/ICASSP40776.2020.9053012"},{"key":"10.3233\/IDA-230305_ref14","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1109\/MIS.2013.34","article-title":"YouTube Movie Reviews: Sentiment Analysis in an Audio-Visual Context","volume":"28","author":"W\u00f6llmer","year":"2013","journal-title":"IEEE Intelligent Systems"},{"key":"10.3233\/IDA-230305_ref15","unstructured":"M. Mitchell, J. Aguilar, T. Wilson et al., Open Domain Targeted Sentiment, in: Conference on Empirical Methods in Natural Language Processing, 2013. pp. 1643\u20131654."},{"key":"10.3233\/IDA-230305_ref17","doi-asserted-by":"crossref","first-page":"15333","DOI":"10.1007\/s11042-022-12487-x","article-title":"Sentence constituent-aware attention mechanism for end-to-end aspect-based sentiment analysis","volume":"81","author":"Lu","year":"2020","journal-title":"Multimedia Tools and Applications"},{"key":"10.3233\/IDA-230305_ref18","doi-asserted-by":"crossref","unstructured":"X. Li, L.D. Bing, P.G. Li et al., A unified model for opinion target extraction and target sentiment prediction, in: AAAI Conference on Artificial Intelligence, Vol. 824, 2019. pp. 6714\u20136721.","DOI":"10.1609\/aaai.v33i01.33016714"},{"key":"10.3233\/IDA-230305_ref19","doi-asserted-by":"crossref","unstructured":"M.S. Zhang, Y. Zhang and D.T. Vo, Neural Networks for Open Domain Targeted Sentiment, in: Conference on Empirical Methods in Natural Language Processing, 2015. pp. 612\u2013621.","DOI":"10.18653\/v1\/D15-1073"},{"key":"10.3233\/IDA-230305_ref20","doi-asserted-by":"crossref","unstructured":"D.H. Ma, S.J. Li and H.F. Wang, Joint Learning for Targeted Sentiment Analysis, in: Conference on Empirical Methods in Natural Language Processing, 2018. pp. 4737\u20134742.","DOI":"10.18653\/v1\/D18-1504"},{"key":"10.3233\/IDA-230305_ref21","doi-asserted-by":"crossref","unstructured":"X. Li, L.D. Bing, W.X. Zhang et al., Exploiting BERT for End-to-End Aspect-based Sentiment Analysis, in: Conference on Empirical Methods in Natural Language Processing, 2019. pp. 34\u201341.","DOI":"10.18653\/v1\/D19-5505"},{"key":"10.3233\/IDA-230305_ref22","doi-asserted-by":"crossref","unstructured":"J. Pennington, R. Socher and C. Manning, GloVe: Global Vectors for Word Representation, in: Conference on Empirical Methods in Natural Language Processing, 2014. pp. 1532\u20131543.","DOI":"10.3115\/v1\/D14-1162"},{"key":"10.3233\/IDA-230305_ref23","doi-asserted-by":"crossref","unstructured":"D. Borth, R. Ji, T. Chen et al., Large-scale visual sentiment ontology and detectors using adjective noun pairs, in: Proceedings of the 21st ACM International Conference on Multimedia, 2013. pp. 223\u2013232.","DOI":"10.1145\/2502081.2502282"},{"key":"10.3233\/IDA-230305_ref24","doi-asserted-by":"crossref","first-page":"108107","DOI":"10.1016\/j.knosys.2021.108107","article-title":"Gated attention fusion network for multimodal sentiment classification","volume":"240","author":"Du","year":"2013","journal-title":"Knowl. Based Syst"},{"key":"10.3233\/IDA-230305_ref25","doi-asserted-by":"crossref","first-page":"41","DOI":"10.3390\/a9020041","article-title":"Visual and textual sentiment analysis of a microblog using deep convolutional neural networks","volume":"9","author":"Yu","year":"2016","journal-title":"Algorithms"},{"key":"10.3233\/IDA-230305_ref28","doi-asserted-by":"crossref","unstructured":"G.R. Wang, K.Z. Wang and L. Lin, Adaptively Connected Neural Networks, in: 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. pp. 1781\u20131790.","DOI":"10.1109\/CVPR.2019.00188"},{"key":"10.3233\/IDA-230305_ref29","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1109\/JSTSP.2017.2764438","article-title":"End-to-end multimodal emotion recognition using deep neural networks","volume":"11","author":"Tzirakis","year":"2019","journal-title":"IEEE Journal of Selected Topics in Signal Processing"},{"key":"10.3233\/IDA-230305_ref30","unstructured":"M. Hoang, O.A. Bihorac and J. Rouces, Aspect-Based Sentiment Analysis using BERT, in: Nordic Conference of Computational Linguistics, 2019. pp. 187\u2013196."},{"key":"10.3233\/IDA-230305_ref31","doi-asserted-by":"crossref","unstructured":"Y. Wang, Q. Chen and W. Wang, Multi-task BERT for Aspect-based Sentiment Analysis, in: 2021 IEEE International Conference on Smart Computing (SMARTCOMP), 2021. pp. 383\u2013385.","DOI":"10.1109\/SMARTCOMP52413.2021.00077"},{"key":"10.3233\/IDA-230305_ref32","doi-asserted-by":"crossref","unstructured":"H. Monkaresi, M.S. Hussain and R.A. Calvo, Classification of affects using head movement, skin color features and physiological signals, in: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2012. pp.\u00a02664\u20132669.","DOI":"10.1109\/ICSMC.2012.6378149"},{"key":"10.3233\/IDA-230305_ref33","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/978-3-319-25207-0_14","article-title":"Convolutional neural networks for multimedia sentiment analysis","volume":"9362","author":"Cai","year":"2012","journal-title":"Natural Language Processing and Chinese Computing"},{"key":"10.3233\/IDA-230305_ref34","doi-asserted-by":"crossref","first-page":"53","DOI":"10.5772\/54002","article-title":"Towards efficient multi-modal emotion recognition","volume":"10","author":"Dobri\u0161ek","year":"2013","journal-title":"International Journal of Advanced Robotic Systems"},{"key":"10.3233\/IDA-230305_ref35","doi-asserted-by":"crossref","unstructured":"B. Siddiquie, D. Chisholm and A. Divakaran, Exploiting Multimodal Affect and Semantics to Identify Politically Persuasive Web Videos, in: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015.","DOI":"10.1145\/2818346.2820732"},{"key":"10.3233\/IDA-230305_ref36","doi-asserted-by":"crossref","first-page":"142","DOI":"10.1109\/TMM.2011.2171334","article-title":"Error weighted semi-coupled hidden markov model for audio-visual emotion recognition","volume":"14","author":"Lin","year":"2012","journal-title":"IEEE Transactions on Multimedia"},{"key":"10.3233\/IDA-230305_ref38","doi-asserted-by":"crossref","first-page":"136843","DOI":"10.1109\/ACCESS.2020.3011977","article-title":"Enhanced video analytics for sentiment analysis based on fusing textual, auditory and visual information","volume":"8","author":"Al-Azani","year":"2020","journal-title":"IEEE Access"},{"key":"10.3233\/IDA-230305_ref39","doi-asserted-by":"crossref","unstructured":"Q.T. Truong and H.W. Lauw, VistaNet: visual aspect attention network for multimodal sentiment analysis, in: AAAI Conference on Artificial Intelligence, Vol. 38, 2019. pp. 305\u2013312.","DOI":"10.1609\/aaai.v33i01.3301305"},{"key":"10.3233\/IDA-230305_ref40","doi-asserted-by":"crossref","unstructured":"N. Xu and W. Mao, MultiSentiNet: A Deep Semantic Network for Multimodal Sentiment Analysis, in: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 2017.","DOI":"10.1145\/3132847.3133142"},{"key":"10.3233\/IDA-230305_ref43","doi-asserted-by":"crossref","unstructured":"J. Yu and J. Jiang, Adapting bert for target-oriented multimodal sentiment classification, in: International Joint Conference on Artificial Intelligence, 2019. pp. 5408\u20135414.","DOI":"10.24963\/ijcai.2019\/751"},{"key":"10.3233\/IDA-230305_ref44","unstructured":"D. Lu, L. Neves, V. Carvalho et al., Adaptive co-attention network for named entity recognition in tweets, in: AAAI Conference on Artificial Intelligence, 2018. pp. 5674\u20135681."},{"key":"10.3233\/IDA-230305_ref45","first-page":"1990","article-title":"Visual attention model for name tagging in multimodal social media","volume":"1","author":"Ba","year":"2018","journal-title":"Annual Meeting of the Association for Computational Linguistics"}],"container-title":["Intelligent Data Analysis"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/IDA-230305","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,26]],"date-time":"2024-11-26T13:51:44Z","timestamp":1732629104000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.medra.org\/servlet\/aliasResolver?alias=iospress&doi=10.3233\/IDA-230305"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,19]]},"references-count":38,"journal-issue":{"issue":"5"},"URL":"https:\/\/doi.org\/10.3233\/ida-230305","relation":{},"ISSN":["1088-467X","1571-4128"],"issn-type":[{"type":"print","value":"1088-467X"},{"type":"electronic","value":"1571-4128"}],"subject":[],"published":{"date-parts":[[2024,9,19]]}}}