{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T16:55:59Z","timestamp":1726851359459},"reference-count":0,"publisher":"Association for the Advancement of Artificial Intelligence (AAAI)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AAAI"],"abstract":"We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.<\/jats:p>","DOI":"10.1609\/aaai.v35i4.16431","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T18:16:28Z","timestamp":1662660988000},"page":"3208-3216","source":"Crossref","is-referenced-by-count":113,"title":["ERNIE-ViL: Knowledge Enhanced Vision-Language Representations through Scene Graphs"],"prefix":"10.1609","volume":"35","author":[{"given":"Fei","family":"Yu","sequence":"first","affiliation":[]},{"given":"Jiji","family":"Tang","sequence":"additional","affiliation":[]},{"given":"Weichong","family":"Yin","sequence":"additional","affiliation":[]},{"given":"Yu","family":"Sun","sequence":"additional","affiliation":[]},{"given":"Hao","family":"Tian","sequence":"additional","affiliation":[]},{"given":"Hua","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Haifeng","family":"Wang","sequence":"additional","affiliation":[]}],"member":"9382","published-online":{"date-parts":[[2021,5,18]]},"container-title":["Proceedings of the AAAI Conference on Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/download\/16431\/16238","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/download\/16431\/16238","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T18:16:28Z","timestamp":1662660988000},"score":1,"resource":{"primary":{"URL":"https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/view\/16431"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,18]]},"references-count":0,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,5,28]]}},"URL":"https:\/\/doi.org\/10.1609\/aaai.v35i4.16431","relation":{},"ISSN":["2374-3468","2159-5399"],"issn-type":[{"value":"2374-3468","type":"electronic"},{"value":"2159-5399","type":"print"}],"subject":[],"published":{"date-parts":[[2021,5,18]]}}}