Syntax Tree Constrained Graph Network for Visual Question Answering

Abstract

Visual Question Answering (VQA) aims to automatically answer natural language questions related to given image content. Existing VQA methods integrate vision modeling and language understanding to explore the deep semantics of the question. However, these methods ignore the significant syntax information of the question, which plays a vital role in understanding the essential semantics of the question and guiding the visual feature refinement. To fill the gap, we suggested a novel Syntax Tree Constrained Graph Network (STCGN) for VQA based on entity message passing and syntax tree. This model is able to extract a syntax tree from questions and obtain more precise syntax information. Specifically, we parse questions and obtain the question syntax tree using the Stanford syntax parsing tool. From the word level and phrase level, syntactic phrase features and question features are extracted using a hierarchical tree convolutional network. We then design a message-passing mechanism for phrase-aware visual entities and capture entity features according to a given visual context. Extensive experiments on VQA2.0 datasets demonstrate the superiority of our proposed model.

X. Su and Q. Zhang—The first two authors contribute equally to this work.

A Experimental Settings

1.1 A.1 Datasets

The dataset consists of real images from MSCOCO [13] with the same training, test, and validation set separation, with more than 80k images and 444k questions, 40k images and 214k questions, 80k images, and 448k questions. For each image, there are an average of at least three questions. The questions fall into three categories based on the type of response: yes/no, number, and others. Each pair collected 10 answers from human annotators, choosing the most frequent answer as the correct answer. This data set contains both open-ended and multiple-choice questions. In this paper, we focus on open-ended questions, taking answers that appear more than 9 times in the training set as candidate answers, and generating 3129 candidate answers.

1.2 A.2 Baselines

We use the following state-of-the-art methods as baselines to evaluate the performance of our STCGN: (1) LSTM-VGG [2] uses LSTM and VGG to extract text and image features respectively, and realizes the feature fusion through the element cross product form. (2) SAN [18] designs a stacked attention mechanism that computes the most relevant areas of the question step by step to deduce the final answer. (3) DMN [10] constructs four network modules of input, question, episodic memory, and answer. Through the cyclic work of these modules, an iterative attention process is generated. (4) MUTAN [3] proposed a tensor-based Tracker decomposition method, which uses low-rank matrix decomposition to solve the problem of the large number of parameters in traditional bilinear models. (5) BUTD [1] proposed a combination of bottom-up and top-down attention mechanisms. Through the top-down attention mechanism, the correlation between the question and each region is calculated, so as to obtain more accurate picture features related to the question. (6) BAN [9] considers the bilinear interaction between two input channels is considered and the visual and textual information is fused by calculating the bilinear attention distribution. (7) LCGN [7] designs a question-aware messaging mechanism and uses question features to guide the refinement of entity features through multiple iteration steps, and realizes the integration of entity features and context information. (8) Murel [4] enriches the interaction between the problem region and the image region by vector representation and a sequence of units composed of multiple MuRel units. (9) ReGAT [12] extracted explicit relation and implicit relation in each image and constructed explicit relation graph and implicit relation graph. At the same time, the graph attention network of question awareness is used to integrate the information of different relation spaces. (10)UFSCAN [23] proposed a multimodal feature-wise attention mechanism and modeled feature-wise co-attention and spatial co-attention between image and question modalities simultaneously. (11)MMMAN [11] proposes a multi-level mesh mutual attention model to utilize low-dimensional and high-dimensional question information at different levels, providing more feature information for modal interactions.

1.3 A.3 Settings

In the experiments, Adamx was selected as the optimizer, and the initial learning rate was set at 0.001. With the increase in learning rounds, the learning rate gradually increased from 0.001 to 0.004. When the model accuracy reached its peak, the learning rate began to decay. During training, the decay rate of the learning rate was set to 0.5, and the model training rounds were 30 rounds. Regarding text, the word representation dimension of the question was 300, and the part of the speech embedding dimension was 128. In the tree hierarchical convolution module, word-level convolution uses a convolution kernel with the size of 3*428 for text convolution operation and the maximum syntax subtree length set to 4, and phrase-level convolution uses multi-head attention mechanism with the number of 8. To extract question features, we use a bidirectional GRU network with a hidden layer dimension of 1024. For picture features, we set the dimension as 2048, each picture contains 10–100 visual entity areas, the total time step of message passing T is set as 4, and the dimension of scene context-aware entity feature is set as 1024. In our experiment, the VQA score provided by the official VQA competition is used as the evaluation metrics, shown as follows:

$$\begin{aligned} acc({\textbf {ans}}) = min(\frac{\#human \ provides \ {\textbf {ans}}}{3},1) \end{aligned}$$

(15)