1 Introduction

In recent years, researchers working in the field of computer vision have begun to pay increasing attention to scene understanding tasks (Zheng et al. 2015; Lu et al. 2016; Zhang et al. 2017; Zhao et al. 2020; Lin et al. 2020). Since human beings are often central to real-world scenes, Human–Object Interaction (HOI) detection has become a fundamental problem in scene understanding. HOI detection involves not only identifying the classes and locations of objects in the images, but also the interactions (verbs) between each human–object pair. As shown in Fig. 1, an interaction between a human–object pair can be represented by a triplet \({<}{} \textit{person verb object}{>}\), herein referred to as one HOI category. One human–object pair may comprise multiple triplets, e.g. \({<}{} \textit{person fly airplane}{>}\) and \({<}{} \textit{person ride airplane}{>}\).

Fig. 1
figure 1

Examples reflecting the verb polysemy problem in HOI detection. In terms of describing the HOIs, a and b present HOI examples of “play”. “feet” are more important in (a) while “hands” are more important in (b). c and d illustrate HOI examples of “hold”. The human–object pairs in (c) and (d) are characterized by dramatically different human–object spatial features, i.e. the relative location between two bounding boxes. e and f illustrate HOI examples of “fly”. The “person” in (e) exhibits discriminative pose features while the “person” in (f) does not

Full size image

The HOI detection task is notably challenging (Chao et al. 2018; Gao et al. 2018). One major reason is that verbs can be polysemic. As illustrated in Fig. 1, a verb may convey substantially different semantic meanings and visual characteristics with respect to different objects, as these objects may have diverse functions and attributes. One pair of examples can be found in Fig. 1a, b. Here, the “feet” are the more discriminative parts of the human figure for \({<}{} \textit{person play soccer}\_\textit{ball}{>}\) while “hands” are more important for describing \({<}{} \textit{person play frisbee}{>}\). A second pair of examples is presented in Fig. 1c, d. The human–object pairs in (c) and (d), despite being tagged with the same verb, present dramatically different human–object spatial features. Another more serious consideration is that the importance of the same type of visual feature may vary dramatically as the objects of interest change. For example, the human pose plays a vital role in describing \({<}{} \textit{person fly kite}{>}\) in Fig. 1e; by contrast, the human pose is invisible and therefore useless for characterizing \({<}{} \textit{person fly airplane}{>}\) in Fig. 1f. Verb polysemy therefore presents a significant challenge in the HOI detection.

The problem of verb polysemy is relatively underexplored, and sometimes even ignored, in existing works (Xu et al. 2019; Li et al. 2019b; Liao et al. 2020; Wang et al. 2020). Most contemporary approaches tend to assume that the same verb will have similar visual characteristics across different HOI categories, and accordingly opt to design object-shared verb classifiers. When the verb classifier is shared among all objects, each verb obtains more training samples, thereby promoting the robustness of the classification for HOI categories with a small sample size. However, due to the polysemic nature of the verbs, a dramatic semantic gap may exist between instances of the same verb across different HOI categories. Chao et al. (2018) constructed object-specific verb classifiers for each HOI category, which are able to overcome the polysemy problem for HOI categories that have sufficient training samples. However, this approach lacks few- and zero-shot learning abilities for HOI categories where only small amounts of training data are available.

Fig. 2
figure 2

Visual features of each human–object pair are duplicated multiple times so that polysemy-aware visual features can be obtained under the guidance of language priors. Each polysemy-aware feature is sent to a specific verb classifier, which can be any type of the three verb classifiers mentioned in this paper. To reduce the number of duplicated human–object pairs, meaningless HOI categories (e.g. \({<}{} \textit{person eat book}{>}\) and \({<}{} \textit{person ride book}{>}\)) are ignored. Meaningful and common HOI categories (e.g. \({<}{} \textit{person hold book}{>}\) and \({<}{} \textit{person read book}{>}\)) are available in each popular HOI detection database. Multiple verbs are “checked” in this figure as HOI detection performs multi-label verb classification for each human–object pair

Full size image

In this paper, we propose a novel Polysemy Deciphering Network (PD-Net) to address the challenging verb polysemy problem. As illustrated in Fig. 2, PD-Net transforms the multi-label verb classifications for each human–object pair into a set of binary classification problems. Here, each binary classifier is used for the verification of one verb category. The classifiers share the majority of their parameters; the main difference lies in the input features. Next, we decode the verb polysemy in the following three ways.

First, we enable features sent for each binary classifier to be polysemy-aware using two novel modules, namely Language Prior-guided Channel Attention (LPCA) and Language Prior-based Feature Augmentation (LPFA). The language priors are two word embeddings made up of one verb and one object. The object class is predicted by one object detector; the verb is the one to be determined by one specific binary verb classifier. For its part, LPCA is applied to both the human and object appearance features. The two appearance features are usually redundant, as only part of their information is involved for one specific HOI category (see Fig. 1). Therefore, LPCA is used to highlight important elements in the appearance features for each binary classifier. Moreover, both human–object spatial and human pose features are often vague and can vary dramatically for the same verb, as shown in Fig. 1a–d; we therefore propose LPFA, which concatenates the two features with language priors, respectively. In this way, the classifiers can receive hints to reduce the intra-class variation of the same verb for the pose and spatial features.

We further design a novel Polysemy-Aware Modal Fusion module (PAMF), which produces attention scores based on the above language priors in order to dynamically fuse multiple feature types. The language priors provide hints regarding the importance of the features for each HOI category. As can be seen in Fig. 1, the human pose feature is discriminative when the language prior is “fly kite” (Fig. 1e), but is less useful when the language prior is “fly airplane” (Fig. 1f). Therefore, our proposed PAMF deciphers the verb polysemy problem by highlighting the features that are more important for each HOI category.

Moreover, as mentioned above, both object-shared and object-specific verb classifiers have limitations. We therefore propose a novel clustering-based object-specific verb classifier, which combines the advantages of object-shared and object-specific classifiers. The main motivation is to ensure that semantically similar HOI categories containing the same verb, e.g. \({<}{} \textit{person hold cow}{>}\) and \({<}{} \textit{person hold elephant}{>}\) can share the same verb classifier. HOIs that are semantically very different (e.g. \({<}{} \textit{person hold book}{>}\) and \({<}\)person hold backpack\({>}\)) are identified using another verb classifier. In this way, the verb polysemy problem is mitigated. Meanwhile, clustering-based object-specific classifiers has the capacity to handle the few- and zero-shot learning problems that arise in HOI detection, since we merge the training data of semantically similar HOI categories.

To the best of our knowledge, our proposed PD-Net is the first approach to explicitly handle the verb polysemy problem in HOI detection. More impressively, our experimental results on three databases demonstrate that our approach consistently outperforms state-of-the-art methods by considerable margins. A preliminary version of this paper has been published in (Zhong et al. 2020). Compared with the conference version, this version further proposes a novel Language Prior-guided Channel Attention module, simplifies the architecture of PD-Net by using Clustering-based Object-Specific classifiers, builds a new database (named HOI-VP) to facilitate the research on the verb polysemy problem, and includes further experimental investigations.

The remainder of this paper is organized as follows. Section 2 briefly reviews related works. The details of the proposed components of PD-Net are described in Sect. 3. The databases and implementation details are introduced in Sect. 4, while the experimental results are presented in Sect. 5. Finally, we conclude the paper in Sect. 6.

2 Related Works

2.1 Human–Object Interaction Detection

HOI detection performs multi-label verb classification for each human–object pair, meaning that the interaction between the same human–object pair may be described using multiple verbs. Depending on the order of verb classification and target object association, existing HOI detection approaches can be divided into two categories. The first category of methods infer the verb actions being performed by one person, then associate each verb with a single object in the image. Multiple target object association approaches have been proposed. For example, Shen et al. (2018) proposed an approach based on the value of object detection scores, while Gkioxari et al. (2018) fitted a distribution density function of the target object locations based on the human appearance feature. Moreover, Qi et al. (2018) adopted a graph parsing network to associate the target objects. Liao et al. (2020) and Wang et al. (2020) first defined interaction points for HOI detection; next, they locate the interaction points and associate each point with one human–object pair.

The second category of methods first pair each human instance with all object instances as candidate human–object pairs, then recognize the verb for each candidate pair (Gupta et al. 2019). Many types of features have been employed to promote the verb classification performance. For example, Wan et al. (2019) employed both human parts and pose-aware features for verb classification, while Xu et al. (2020) exploited human gaze and intention to assist HOI detection. Furthermore, Wang et al. (2019a) extracted context-aware human and object appearance features to promote HOI detection performance. Li et al. (2020a) utilized 3D pose models and 3D object location to assist HOI detection. Moreover, Li et al. (2020b) annotated large amounts of part-level human–object interactions and trained a PaStaNet, which is helpful for HOI detection models to make use of fine-grained human part features. A large number of novel model architectures for HOI detection have been also developed. For example, Li et al. (2019b) introduced a Transferable Interactiveness Network that suppresses candidate pairs without interactions. Peyre et al. (2019) constructed a multi-stream model that projects visual features and word embeddings to a joint space, which is helpful for unseen HOI category detection. Xu et al. (2019) constructed a graph neural network to promote the quality of word embeddings by utilizing the correlation between semantically similar verbs, while Zhou et al. (2020) proposed a cascade architecture that facilitates coarse-to-fine HOI detection.

Besides, HOI recognition (Chao et al. 2015) is one similar task to HOI detection. Briefly, HOI recognition methods predict all possible HOI categories in an image but they do not detect the location of involved human–object pairs. For example, Kato et al. (2018) proposed to compose classifiers for unseen verb-noun pairs by leveraging an external knowledge graph and graph convolutional networks.

2.2 The Polysemy Problem

This problem is very common in our daily life. There have been some researches exploring the polysemy problem, e.g. natural language processing Ma et al. 2020; Huang et al. 2012; Oomoto et al. 2017) and recommendation systems (Liu et al. 2019). For example, the polysemy problem in natural language processing is mainly due to different usages of a word in grammar, e.g., functioned as a verb or a noun (Ma et al. 2020). Besides, each node in a recommendation system could have multiple facets because of the different links with its neighbour nodes, which brings in the node polysemy problem (Liu et al. 2019).

In this paper, we address the verb polysemy problem in HOI detection, i.e., a verb may convey substantially different visual characteristics when associated with different objects. There are also some related areas that may face the same verb polysemy problem, e.g. Action Recognition (Simonyan and Zisserman 2014; Tran et al. 2015; Damen et al. 2018) and Visual Relationship Detection (Lu et al. 2016; Krishna et al. 2017; Kuznetsova et al. 2020; Ji et al. 2020).

Action recognition methods aim to recognize human actions from an image or a video. In particular, the EPIC-KITCHENS Dataset (Damen et al. 2018) is a new large-scale egocentric video benchmark for action recognition tasks. Besides recognizing the human action from the video, the task on this dataset also involves identifying the interacted object category. It includes 125 verb and 352 noun categories. There are also many verbs suffering from the polysemy problem, e.g. “hold”, “open” and “close”.

Visual relationship detection involves detecting and localizing pairs of objects in an image and also classifying the predicate or interaction between each subject-object pair. Different from HOI detection, the subject for each subject-object pair in visual relationship detection can be any object category besides human. Polysemy verbs, e.g. “carry”, “ride”, “hold” are also common relationships in visual relationship detection. Therefore, visual relationship detection also suffers from the polysemy problem.

Furthermore, as the vast majority of HOI detection methods are based on single images (Gao et al. 2018; Xu et al. 2019; Li et al. 2019b; Gupta et al. 2019; Peyre et al. 2019; Qi et al. 2018; Liao et al. 2020; Wang et al. 2020; Ulutan et al. 2020), we only consider the verb polysemy problem for image-based HOI detection. However, image-based methods ignore the temporal information, which also provides rich cues to decipher the verb polysemy problem. Therefore, we expect more works on video-based HOI detection in the future.

2.3 The Exploitation of Language Priors

Language priors have also been successfully utilized in many computer vision-related fields, including Scene Graph Generation (Lu et al. 2016; Zhang et al. 2017; Gu et al. 2019; Wang et al. 2019b), Image Captioning (Zhou et al. 2019; Yao et al. 2019), and Visual Question Answering (Zhou et al. 2019; Gao et al. 2019; Marino et al. 2019). Moreover, several works (Xu et al. 2019; Peyre et al. 2019) have adopted language priors for HOI detection. All of these approaches project visual features and word embeddings to a joint space, which improves HOI detection by exploiting the semantic relationship between similar verbs or HOI categories (e.g. “drink” and “sip” or “ride horse” and “ride cow”). However, these works do not employ language priors to solve the challenging verb polysemy problem. Compared with the above methods, PD-Net aims to solve the verb polysemy problem by using three novel language prior-based components: Language Prior-guided Channel Attention, Language Prior-based Feature Augmentation, and Polysemy-Aware Modal Fusion.

Fig. 3
figure 3

Overview of the Polysemy Deciphering Network. For the sake of simplicity, only one binary CSP classifier (for “hold”) is illustrated here. PD-Net takes four feature streams as input: the human appearance stream (H stream), the object appearance stream (O stream), the human–object spatial stream (S stream), and the human pose stream (P stream). These four feature streams are first processed by either LPCA or LPFA to be polysemy-aware. They are then sent to the H, O, S, and P blocks for binary classification, respectively. Subsequently, the classification scores from the four feature streams are fused using the attention scores produced by \(\mathbf{PAMF}\). Here, \(\otimes \) and \(\oplus \) denote the element-wise multiplication and addition operations, respectively

Full size image

2.4 Attention Models

Attention mechanisms are becoming a popular component in computer vision tasks, including Image Captioning (Chen et al. 2017; Xu et al. 2015; You et al. 2016), Action Recognition (Girdhar and Ramanan 2017; Meng et al. 2019), and Pose Estimation (Li et al. 2019a; Ye et al. 2016). Existing studies on attention mechanisms can be roughly divided into three categories: namely, hard regional attention (Jaderberg et al. 2015; Li et al. 2018; Wang et al. 2021), soft spatial attention (Wang et al. 2017; Pereira et al. 2019; Zhu et al. 2018), and channel attention (Pereira et al. 2019; Hu et al. 2018; Ding et al. 2020). Hard regional attention methods typically predict regions of interest (ROIs) first, and then only utilize features in ROIs for subsequent tasks. In comparison, soft spatial attention and channel attention (CA) models use soft weights to highlight important features in the spatial and channel dimensions, respectively. There have also been existing works that adopted attention models to assist with HOI detection tasks. For example, Gao et al. (2018) and Wang et al. (2019a) employed an attention mechanism to enhance the human and object features by aggregating contextual information. Wan et al. (2019) proposed the PMFNet model, which adopts human pose and spatial features as cues to infer the importance of each human part. Ulutan et al. (2020) used cues derived from a human–object spatial configuration to highlight the important elements in appearance features.

To the best of our knowledge, few works thus far have made use of the attention mechanism to solve the verb polysemy problem in HOI detection. Moreover, existing attention models for HOI detection usually employ visual features (e.g. the appearance and pose features) as cues. By contrast, our proposed Language Prior-guided Channel Attention and Polysemy-Aware Modal Fusion adopt language priors as cues; these priors have clear semantic meanings, and are therefore well-suited to resolving the verb polysemy problem.

3 Method

We first formulate a basic HOI detection scheme that is adopted by most existing works (Gupta et al. 2019; Li et al. 2019b; Wan et al. 2019; Zhou and Chi 2019; Li et al. 2020a). Then, we explain the verb polysemy problem and formulate the PD-Net framework. Finally, we describe each of the key components in PD-Net.

3.1 Problem Formulation

Given an image I, human and object proposals are generated using Faster R-CNN (Ren et al. 2015). Each human proposal h and each object proposal o are paired as a candidate for verb classification. HOI detection models (Gupta et al. 2019; Li et al. 2019b; Wan et al. 2019; Zhou and Chi 2019; Li et al. 2020a) then produce a set of verb classification scores for each candidate pair (h, o). The classification scores for verb v can be represented as follows:

$$\begin{aligned} {\mathcal {S}}_{(h,o,v)} = \sigma \left( \sum \limits _{i} {T_{(i,v)}(G_{i}(I,h,o))}\right) , \end{aligned}$$
(1)

where i is a subscript that stands for the i-th feature type. Function \(G_{i}(\cdot )\) denotes the model that produces the i-th type of features (Gupta et al. 2019; Li et al. 2019b; Wan et al. 2019; Zhou and Chi 2019; Li et al. 2020a). \(T_{(i,v)}(\cdot )\) represents the classifier for the verb v using features produced by \(G_{i}(\cdot )\). \(\sigma (\cdot )\) denotes the sigmoid activation function.

As described in Sect. 1, existing works (Gupta et al. 2019; Li et al. 2019b; Wan et al. 2019; Zhou and Chi 2019; Li et al. 2020a) suffer from the verb polysemy problem. First, owing to the large intra-class appearance variance for one verb, it is challenging for function \(G_{i}(\cdot )\) to learn discriminative visual features for a polysemic verb. Second, the importance of the same feature type may vary dramatically as the objects of interest change. Third, \(T_{(i,v)}(\cdot )\) is often shared by different HOI categories with the same verb, which makes it difficult for \(T_{(i,v)}(\cdot )\) to capture important visual cues to identify one polysemic verb. Therefore, Eq. (1) cannot well address the verb polysemy problem.

Accordingly, in this paper, we propose PD-Net to address the above three problems. Given one human–object pair (h, o), the classification scores for verb v predicted by PD-Net can be represented as follows:

$$\begin{aligned} {\mathcal {S}}^{\mathbf {PD}}_{(h,o,v)} = \sigma \left( \sum \limits _{i} {a_{(i,o,v)}T_{(i,o,v)}(G_{i}(I,h,o, w_{o}, w_{v}))}\right) , \nonumber \\ \end{aligned}$$
(2)

where \(w_{o}\) and \(w_{v}\) represent the word embeddings for the object and verb categories, respectively. \(G_{i}(\cdot )\) can leverage word embeddings of one verb-object pair (HOI category) to generate polysemy-aware features. \(a_{(i,o,v)}\) denotes attention scores produced by the Polysemy-Aware Modal Fusion module. \(T_{(i,o,v)}\) represents the clustering-based object-specific classifiers that are shared by semantically similar HOI categories with the same verb.

In the following, we introduce the framework of PD-Net based on Eq. (2).

3.2 Overview of PD-Net

The framework of PD-Net is illustrated in Fig. 3. Similar to existing works (Li et al. 2019b; Gupta et al. 2019; Wan et al. 2019; Zhou and Chi 2019; Li et al. 2020a), four types of visual features are adopted, i.e., human appearance, object appearance, human–object spatial, and human pose features. We construct these four types of features for PD-Net following (Gupta et al. 2019). In particular, the human and object appearance features are \(K_{A}\)-dimensional vectors that are extracted from Faster R-CNN model using human and object bounding boxes, respectively. The human–object spatial feature is a 42-dimensional vector encoded using the bounding box coordinates of one human–object pair. Moreover, we use a pose estimation model (Fang et al. 2017) to obtain the coordinates of 17 keypoints for each human instance. Subsequently, following (Gupta et al. 2019), the human keypoints and the bounding box coordinates of object proposal are then encoded into a 272-dimensional pose feature vector.

As outlined in Fig. 2, we transform the multi-label verb classification into a set of binary classification problems. Each of the binary classifiers is used for one verb category verification and only processes features that use this verb as language priors. Moreover, each binary classifier includes a set of H, O, S, and P blocks; Apart from the final layer which is used for verb prediction, the parameters of the other layers in each respective block are shared across different binary classifiers. Therefore, their overall model size is comparable to an ordinary multi-label classifier. The binary classifiers mainly differ in terms of their input features and the way in which they combine predictions from the four feature streams.

In the following, we propose four novel components to handle the verb polysemy problem in HOI detection. First, we introduce the Language Prior-guided Channel Attention and Language Prior-based Feature Augmentation modules, which facilitate the four types of features being polysemy-aware. Second, we design the Polysemy-Aware Modal Fusion module that adaptively fuses the prediction scores produced by the four feature streams and obtains the final prediction score for each binary classifier. Finally, we propose a Clustering-based Object-Specific classification scheme that strikes a balance between resolving the verb polysemy problem and reducing the number of binary classifiers in PD-Net.

3.3 Polysemy-Aware Feature Generation

We here introduce two novel components, i.e. Language Prior-guided Channel Attention and Language Prior-based Feature Augmentation, that generate polysemy-aware features. The two components are denoted as \(G_{i}(\cdot )\) in Eq. (2). The language prior we used in this paper is the concatenated word embedding of two words: one word denotes the verb to be identified (\(w_{v}\) in Eq. (2)), while the other one is the detected object in the human–object pair (\(w_{o}\) in Eq. (2)). The word embeddings are generated using the word2vec tool, which was trained on the Google News dataset (Mikolov et al. 2013). The dimension of the language prior is 600.

Fig. 4
figure 4

Model structure of LPCA. \(F_{A}\) denotes the human or object appearance feature. \(P(\cdot )\) and \(D(\cdot )\) are realized by successive fully-connected layers. \(\otimes \) denotes the element-wise multiplication operation

Full size image

3.3.1 Language Prior-guided Channel Attention

Both the human and object appearance features are usually redundant as only part of their information is involved for a specific HOI category. One example can be found in Fig. 1a, b, where the human body parts most relevant to “play soccer_ball” and “play frisbee” are the “feet” and “hands”, respectively. We therefore propose Language Prior-guided Channel Attention (LPCA) to highlight the important elements in human and object appearance features, based on the channel attention scheme and the guidance of language priors. LPCA is realized through two steps, which are outlined below.

First, we infer the important elements in the human or object appearance feature \(F_{A}\) using language priors. \(F_{A}\) is extracted from the Faster R-CNN model using a bounding box. As can be seen from Fig. 4, the language prior is projected to a \(K_{A}\)-dimensional vector (denoted as \(L_{A}\)) via two successive fully-connected layers. The dimension of the first one is set to \(\frac{K_{A}}{2}\); \(K_{A}\) is equal to the dimension of \(F_{A}\). Similar to (Peyre et al. 2019), \(L_{A}\) is normalized via its L2 norm. To drive \(L_{A}\) to pay attention to important elements in \(F_{A}\) regarding to the verb-object pair, we perform element-wise multiplication between \(L_{A}\) and \(F_{A}\), as follows:

$$\begin{aligned} L_{B} = F_{A} \otimes L_{A}, \end{aligned}$$
(3)

and further compute the summation of elements in \(L_{B}\):

$$\begin{aligned} {\mathcal {S}}_{au} = \sigma (Sum(L_{B})), \end{aligned}$$
(4)

where \(\sigma \) denotes the sigmoid activation function and \(Sum(\cdot )\) denotes the summation of all elements in one vector. During the training stage, we minimize the binary cross-entropy loss between \({\mathcal {S}}_{au}\) and the binary label for the verb to verify. During inference, the operation in Eq. (4) can be ignored. By optimizing the fully-connected layers via the verb verification goal, the value of elements in \(L_{A}\) can reflect the importance of the corresponding elements in \(F_{A}\).

It is worth noting that the quality of both \(L_{A}\) and \(L_{B}\) can be affected by the discrepancy between visual features and word embeddings, since the word embeddings are not specifically designed for computer vision tasks (Xu et al. 2019). Therefore, directly using \(L_{B}\) as representation for verb verification may be suboptimal. Consequently, to handle this problem, we propose the following strategy to further enhance the quality of the representations.

Second, we obtain attention scores based on \(L_{B}\) via a plain channel attention module:

$$\begin{aligned} C_{att} = \sigma (D(L_{B})), \end{aligned}$$
(5)

where \( C_{att}\) stands for the final channel attention scores for \(F_{A}\). \(D(\cdot )\) is realized by two successive fully-connected layers. By imposing a supervision on \({\mathcal {S}}_{au}\), elements in \(L_{B}\) that are relevant to the verb to verify have large values, which facilitates the learning of an effective channel attention. For its part, the plain CA module then makes use of the correlation between elements in \(L_{B}\) to promote the quality of attention scores. Finally, the polysemy-aware human or object appearance features can be obtained via the following equation:

$$\begin{aligned} {\tilde{F}}_{A} = F_{A} \otimes C_{att}, \end{aligned}$$
(6)

Existing channel attention models, e.g. the SE network (Hu et al. 2018), usually utilize appearance features to generate channel attention scores for the features themselves. Besides, the VSGNet model (Ulutan et al. 2020) adopted the human–object spatial features as cues to infer channel attention scores for the appearance feature. Compared with the above two works, LPCA contains clearer semantic information for each HOI. First, the input of LPCA includes language priors, which have clear semantic meaning. Second, LPCA imposes an auxiliary binary cross-entropy loss between \(S_{au}\) and the verb to verify. This extra loss enables the visual features for different HOI categories with the same verb to be polysemy-aware according to their specific language priors. Moreover, language priors are adopted in (Peyre et al. 2019) to construct verb classifiers. In their method, human and object features are fixed. In comparison, we utilize language priors to generate polysemy-aware features, which are adjustable for each verb-object pair to identify.

Finally, LPCA can also be regarded as cross-modal attention (Lu et al. 2019) or a conditioning module (Perez et al. 2018). The differences between LPCA and existing cross-modal attention methods (e.g. ViLBERT (Lu et al. 2019)) and conditioning modules (e.g. FiLM (Perez et al. 2018)) are illustrated as follows. First, ViLBERT utilizes a set of image region features and a sequence of word embeddings as input. Then, it conducts cross modal attention based on the standard transformer blocks (Vaswani et al. 2017), which compute query, key, and value matrices to produce the attention-pooled features. In comparison, LPCA only uses one visual feature vector and one text segment (the concatenation of the word embeddings of the verb and object category) as the input. The Hadamard product between the features \(F_{A}\) and \(L_{A}\) is utilized to generate the channel attention scores. Second, FiLM (Perez et al. 2018) carries out a feature-wise affine transformation on a deep model’s intermediate features, directly conditioned on the language prior. In comparison, in order to address the discrepancy between visual features and word embeddings, LPCA generates channel attention scores based on the correlation between visual feature and word embeddings.

3.3.2 Language Prior-Based Feature Augmentation

Language Prior-based Feature Augmentation (LPFA) is applied to human–object spatial and human pose features. As illustrated in Fig. 1c–f, the spatial and pose features are often vague and vary dramatically for the same verb, meaning that they contain insufficient information. Therefore, we propose LPFA to augment the pose and spatial features. More specifically, we concatenate each of the two features with the 600-dimensional language prior As a result of this concatenation, the classifiers receive hints that can aid in reducing the intra-class variation of the same verb for the pose and spatial features.

3.4 Polysemy-Aware Modal Fusion

As illustrated in Fig. 3, the four feature streams are sent to the H, O, S and P blocks, respectively. The H and O blocks are constructed using two successive fully-connected layers, while the S and P blocks are constructed using three successive fully-connected layers. In the interests of simplicity, the dimension of each hidden fully-connected layer is set as the dimension of its input feature vector. The output dimension of these four blocks is set to \(K_{C}\), which is the number of binary classifiers in PD-Net.

As discussed in Sect. 1, one major challenge posed by the verb polysemy problem is that the relative importance of each of the four feature streams to the identification of the same verb may vary dramatically as the objects change. As shown in Fig. 1e, f, the human appearance and pose features are most important for detecting \({<}{} \textit{person fly kite}{>}\); by contrast, these features are almost invisible and therefore less useful for detecting \({<}{} \textit{person fly airplane}{>}\). Therefore, we propose Polysemy-Aware Modal Fusion (PAMF) to generate attention scores that dynamically fuse the predictions of the four feature streams. To the best of our knowledge, PAMF is the first attention module that effectively uses language prior to address the verb polysemy problem in HOI detection. In more detail, we use the same 600-dimensional word embedding (e.g. “hold book”) that is implemented in LPCA and LPFA. The language prior is fed into two successive fully-connected layers, the dimensions of which are 48 and 4, respectively. The first fully-connected layer is followed by a ReLU layer, while the second one is followed by a sigmoid activation function. The output of PAMF is used as attention scores for the four feature streams. In this way, the important feature streams with respect to each HOI category is highlighted, while those that are less important are suppressed. We can see that PAMF is a weight-light module that can be effectively optimized even with limited training data. Moreover, we use the pre-trained word embeddings (Mikolov et al. 2013) as input for PAMF. These word embeddings have semantical relationships with each other as prior, which further reduces the optimization difficulty of PAMF.

Therefore, Eq. (2) can be rewritten as follows:

$$\begin{aligned} {\mathcal {S}}^{\mathbf {PD}}_{(h,o,v)} = \sigma \left( \sum \limits _{i \in \{ {\mathbf {H}}, {\mathbf {O}}, {\mathbf {S}}, {\mathbf {P}}\} } {a_{(i,o,v)}s_{(i,o,v)}}\right) , \end{aligned}$$
(7)

where i denotes one feature stream, while \(a_{(i,o,v)}\) is the attention score generated by PAMF for the i-th feature stream. \(s_{(i,o,v)}\) is the output for verb v generated by the i-th feature stream.

3.5 Clustering-Based Object Specific Verb Classifiers

Although the above proposed components, i.e. LPCA, LPFA, and PAMF, can help object-shared verb classifiers to relieve the verb polysemy problem, the defects of object-shared verb classifiers still remains. The essential reason is that HOIs with different objects share the same verb classifier. Under ideal circumstances, object-specific verb classifiers can overcome this problem if sufficient training data exists for each HOI category. However, if we assume that the number of object categories is |O| and the number of verb categories is |V|, the total number of their combinations will therefore be \(|O|\times |V|\), which is usually very large even if meaningless HOI categories are removed. Therefore, it is too difficult to obtain sufficient train samples for each HOI category. Moreover, due to the class imbalance problem for HOI categories, the object-specific classifiers lack few- and zero-shot learning abilities for HOI categories which have small amount of training data. Therefore, both types of verb classifiers have limitations.

In this subsection, we introduce a novel verb classifier, named Clustering-based object-SPecific (CSP) verb classifiers, which are denoted as \(T_{(i,o,v)}\) in Eq. (2). CSP classifiers can strike a balance between overcoming the verb polysemy problem and handling the zero- or few-shot learning problems. The main motivation behind CSP classifiers is that some HOIs tagged with the same verb are both semantically and visually similar, e.g. \({<}{} \textit{person hold sheep}{>}\), \({<}{} \textit{person hold horse}{>}\), and \({<}{} \textit{person hold cow}{>}\); therefore, they can share the same verb classifier, meaning that the number of SP classifiers is consequently reduced. In more detail, we first obtain all meaningful and common HOI categories for each verb, which are available in popular databases such as HICO-DET (Chao et al. 2018) and V-COCO (Gupta and Malik 2015). The number of meaningful HOI categories including the verb v is indicated by \({O}_{v}\). We then use the K-means method (MacQueen et al. 1967) to cluster the HOI categories with the same verb v into \({C}_{v}\) clusters according to the cosine distance between the word embeddings of the objects. We empirically set the \({C}_{v}\) for each verb as a rounded number of the square root of \({O}_{v}\).

We provide visualization of clustering results for some polysemic verbs in the supplementary file. During both training and inference, only one CSP classifier is adopted to predict whether the verb is positive for one verb-object pair. The adopted CSP classifier is determined by the object category in the verb-object pair. This clustering strategy is also capable of handling the few- and zero-shot HOI detection problems (Bansal et al. 2020). For example, during testing, a new HOI category \({<}{} \textit{person hold elephant}{>}\) can share the same classifier with other HOI categories that have similar semantic meanings (e.g. \({<}{} \textit{person hold horse}{>}\)).

Besides our automatic way to cluster semantically similar HOI categories, one alternative is to find all possible semantic meanings for a verb from an English dictionary. A dictionary usually elaborates different semantic meanings for each verb. Therefore, the number of clusters for a verb can be determined. Then, one may manually associate each HOI category with one semantic meaning.

One recent work (Bansal et al. 2020) also utilized clustering methods to achieve zero-shot HOI detection. There are two differences between the clustering strategies in this work and (Bansal et al. 2020). First, we utilize clustering to build new verb classifiers. In comparison, the clustering strategy in (Bansal et al. 2020) is used to generate new training data. Second, we cluster the available HOI categories in one database for each respective verb. In comparison, clustering is conducted only once in (Bansal et al. 2020). Our idea is that some HOI categories are meaningless. By clustering the available HOI categories for each verb, we can obtain more meaningful and fine-grained clustering result.

3.6 Training and Testing

3.6.1 Training

PD-Net can be conceptualized as a multi-task network. Its loss for the verification of the verb v in one HOI category (h, v, o) can be represented as follows:

$$\begin{aligned} {\mathcal {L}}_{(h,o,v)} = {\mathcal {L}}_{BCE}({\mathcal {S}}^{\mathbf {PD}}_{(h,o,v)}, {l}_{v}) + \sum \limits _{i \in \{ {\mathbf {H}}, {\mathbf {O}}\} } {{\mathcal {L}}_{BCE}({\mathcal {S}}^{i}_{au}},{l}_{v}), \end{aligned}$$
(8)

where \({\mathcal {L}}_{BCE}\) represents binary cross-entropy loss, while \({l}_{v}\) denotes a binary label (\({l}_{v} \in \) \(\{ 0,1 \}\)) for one verb to verify. Moreover, \({\mathcal {S}}^{{\mathbf {H}}}_{au}\) and \({\mathcal {S}}^{{\mathbf {O}}}_{au}\) denote the output of Eq. (4) for the human and object appearance features, respectively.

3.6.2 Testing

During testing, we use the same method as that utilized in the training stage to obtain the language priors. Here, the object category in the prior is predicted using Faster R-CNN (rather than the ground-truth); the verb category in the prior varies for each binary classifier of the verb. Following (Li et al. 2019b, 2020a; Ulutan et al. 2020; Wan et al. 2019), we also construct an Interactiveness Network (INet) capable of suppressing pairs without interaction. Finally, the prediction score for one HOI category (h, v, o) is represented as follows:

$$\begin{aligned} {\mathcal {S}}^{\mathbf {HOI}}_{(h,o,v)} = {\mathcal {S}}_h \times {\mathcal {S}}_o \times {\mathcal {S}}^{\mathbf {PD}}_{(h,o,v)} \times {\mathcal {S}}^{{\mathbf {I}}}_{(h,o)}, \end{aligned}$$
(9)

where \({\mathcal {S}}_h\) and \({\mathcal {S}}_o\) are the detection scores of human and object proposals, respectively, while \({\mathcal {S}}^{{\mathbf {I}}}_{(h,o)}\) denotes the prediction score generated by the pre-trained INet. In the experimental section below, we demonstrate that INet slightly promotes the performance of PD-Net.

Table 1 The number of associated objects and instances for each verb in the HOI-VP dataset
Full size table

4 Experimental Setup

4.1 Datasets

HICO-DET (Chao et al. 2018) is a large-scale dataset for HOI detection, containing a total of 47,776 images; of these, 38,118 images are assigned to the training set, while the remaining 9568 images are used as the testing set. There are 117 verb categories, 80 object categories, and 600 common HOI categories overall; moreover, these 600 HOI categories are divided into 138 rare and 462 non-rare categories. Each rare HOI category contains less than 10 training samples. Each verb is included in an average of five HOI categories.

V-COCO (Gupta and Malik 2015) is a subset of MS-COCO (Lin et al. 2014) and contains 2533, 2867, and 4946 images used for training, validation and testing, respectively. There are 24 verb categories and 259 HOI categories in total. Each verb is included in 10 HOI categories on average.

HOI-VerbPolysemy (HOI-VP) is a new database constructed in this paper. To the best of our knowledge, this is the first database to be designed explicitly for the verb polysemy problem in HOI detection. In more detail, it consists of 15 common verbs (predicates) that have rich and diverse semantic meanings. It also contains 517 common objects in real-world scenarios. Each verb is included in an average of 55 HOI categories, as detailed in Table 1. In particular, “in” and “on” are two highly common predicates that are also polysemic in visual relationship detection tasks (Lu et al. 2016; Krishna et al. 2017; Kuznetsova et al. 2020; Ji et al. 2020) and are thus both included in the HOI-VP database. There are 21,928 and 7262 images used for training and testing, respectively. All images are collected from the VG database (Krishna et al. 2017), while the corresponding annotations are provided by the HCVRD database (Zhuang et al. 2017). In the HOI-VP dataset, we only use images that were labelled with the 15 predicates listed in Table 1. These images and their labels are collected based on the HCVRD dataset. Therefore, the images of HOI-VP can be considered as a subset of HCVRD.

Table 2 Performance comparisons between common object detectors on COCO (Lin et al. 2014)
Full size table

It is worth noting here that the annotations in HCVRD contain noise. For example, the same verb may be annotated with different words, e.g., “hold”, “holds”, and “holding”, while a similar problem exists for the objects, e.g., “camera”, “digital camera”, and “video camera”. We therefore merge different annotations for the same verb or object categories, respectively. In the following, we take “hold” as example to explain the correction of annotations. We search for the highly relevant labels with the key word “hold”. Then, we manually check the images with these labels, and make sure that these labels indeed have the same semantic meaning. Finally, we merge the labels of the same semantic meaning with “hold”. The annotation noise for each object category is mainly from its fine-grained attributes. For example, the object “shirt” may be labelled as “black shirt”, “blue shirt”, and “stripe shirt”. The merging steps for object labels are the same as those for the verbs. Some sample images from HOI-VP are illustrated in Fig. 9. This database will be made publicly available to expedite research into the verb polysemy problem.

4.2 Evaluation Metrics

According to the official protocols (Chao et al. 2018; Gupta and Malik 2015), mean average precision (mAP) is used as the evaluation metric for HOI detection on both HICO-DET and V-COCO datasets. A positive human–object pair must meet the following requirements: first, the predicted HOI category must be the same type as the ground truth; second, both the human and object proposals must have an Intersection over Union (IoU) with the ground truth proposals of more than 0.5. Moreover, there are two mAP modes in HICO-DET, namely the Default (DT) mode and the Known-Object (KO) mode. In the DT mode, we calculate the average precision (AP) for each HOI category in all testing images. In the KO mode, the object categories in all images are known; therefore, we need only to compute the AP for each HOI category from images containing the interested object. For example, we evaluate the AP of \({<}{} \textit{person ride horse}{>}\) using only those testing images that contain a “horse”. Since the images that contain the object category of interest are known, the KO mode is better able to reflect the verb classification ability. For V-COCO, the role mAP (Gupta and Malik 2015) (\(AP_{role}\)) is used for evaluation.

For the HOI-VP database, we use an evaluation protocol similar to that of HICO-DET. As there are as many as 517 object categories in HOI-VP, object detection becomes a challenging task. Accordingly, to reduce the impact of object detection errors, the ground-truth bounding boxes and categories for both human and object instances are provided. This strategy is similar to the Predicate Classification (PREDCLS) protocol, which has been widely adopted in scene graph generation tasks (Zellers et al. 2018; Lin et al. 2020). It facilitates a clean comparison of verb classification ability between different HOI detection models.

4.3 Implementation Details

To facilitate fair comparison with existing works, we consider two popular object detection models for PD-Net. The first of these, Faster R-CNN (Ren et al. 2015) with ResNet-50-FPN (Lin et al. 2017) backbone, attaches a Feature Pyramid Network (FPN) to ResNet-50 (He et al. 2016) and generates object proposals from the FPN. Based on these proposals, instance appearance features are extracted from the ResNet-50 model. The second model is Faster R-CNN with ResNet-152 backbone (He et al. 2016). Here, both instance proposals and appearance features are obtained from the ResNet-152 model. The above two object detectors are trained on the COCO database (Lin et al. 2014). As shown in Table 2, the two object detectors achieve comparable detection performance. Moreover, to facilitate fair comparison with the majority of existing works (Ulutan et al. 2020; Gupta et al. 2019; Li et al. 2019b; Peyre et al. 2019; Qi et al. 2018), we fix the parameters of both object detectors. Following (Gupta et al. 2019), for both human and each object category, we first select the top 10 proposals according to the detection scores after non-maximum suppression. Moreover, the bounding boxes whose detection scores are lower than 0.01 are removed. The dimension of appearance features for both object detectors, i.e. \(K_{A}\), is 2,048.

Utilizing the same approach as existing works (Gao et al. 2018; Xu et al. 2019; Li et al. 2019b; Gupta et al. 2019; Peyre et al. 2019; Qi et al. 2018; Liao et al. 2020; Wang et al. 2020; Ulutan et al. 2020; Zhong et al. 2021), the HOI categories that appear in the training set are set as the meaningful and common HOI categories in each HOI database. The dimension of output layers of the H, O, S, and P blocks, i.e. \(K_{C}\), is set to 187, 45, and 83 on the HICO-DET, V-COCO, and HOI-VP databases, respectively; these figures are equal to the number of CSP classifiers on each respective database. We train PD-Net for 6 (10) epochs using Adam optimizer (Kingma et al. 2014) with a learning rate of 1e-3 (1e-4) on HICO-DET (V-COCO), while on HOI-VP, we train PD-Net for 12 epochs using a learning rate of 1e-3. During testing, we rank the HOI candidate pairs according to their detection scores (obtained via Eq. (9)) and calculate mAP for evaluation purposes.

5 Experimental Results and Discussion

5.1 Ablation Studies

Table 3 Ablation studies on each component of PD-Net. Full refers to evaluation on all 600 HOI categories in HICO-DET
Full size table

To demonstrate the effectiveness of each proposed component in PD-Net, we perform ablation studies on the HICO-DET database. In Table 3, the baseline is constructed by removing Language Prior-guided Channel Attention (LPCA), Language Prior-based Feature Augmentation (LPFA), and Polysemy-Aware Modal Fusion (PAMF) from PD-Net; we also replace Clustering-based object-SPecific (CSP) classifiers with object-SHared (SH) classifiers. The other settings for the baseline remain the same as in PD-Net. For both models, the Faster R-CNN with ResNet-152 backbone is used for object detection. Experimental results are summarized in Table 3. From these results, we can make the following observations.

5.1.1 Effectiveness of PAMF

Polysemy-Aware Modal Fusion is designed to decipher the verb polysemy by assigning larger weights to more important feature types for each HOI category. As shown in Table 3, PAMF promotes the performance of the baseline by 1.29% and 1.36% mAP in DT and KO modes, respectively.

5.1.2 Effectiveness of LPFA

Language Prior-based Feature Augmentation is used to provide hints for the classifier in order to reduce the intra-class variation of the pose and spatial features by augmenting them with language priors. When LPFA is incorporated, HOI detection performance is promoted by 0.52% and 0.21% mAP in DT and KO modes, respectively.

5.1.3 Effectiveness of LPCA

The appearance features are redundant for HOI detection. Language Prior-guided Channel Attention is proposed to generate polysemy-aware appearance features. As can be seen from Table 3, LPCA promotes the HOI detection performance by a clear margin of 1.33% and 0.21% in DT and KO modes, respectively.

5.1.4 Effectiveness of CSP Classifiers

Clustering-based object-SPecific classifiers can relieve the verb polysemy problem by assigning the same verb classifier to semantically similar HOI categories. As shown in Table 3, CSP classifiers improve the HOI detection performance by 1.06% and 2.13% mAP in DT and KO modes, respectively.

5.1.5 Drop-One-Out Study

We further perform a drop-one-out study in which each proposed component is removed individually. These experimental results further demonstrate that each component is indeed helpful to promote HOI detection performance.

Finally, when INet is integrated, the mAP of PD-Net in the DT mode is further promoted by 0.60%. However, the mAP in the KO mode does not improve. This is because INet can assist PD-Net by suppressing candidate pairs without interactions, which are usually caused by incorrect or redundant object proposals in the DT mode. However, the KO mode is comparatively less affected by object detection errors; therefore, PD-Net can achieve high performance without the assistance of INet in this mode. This experiment demonstrates that the strong performance of PD-Net is primarily a result of its excellent verb classification ability.

Table 4 Comparisons with one variant of the language prior
Full size table

5.2 Comparisons with Variants of PD-Net

5.2.1 Comparisons with Variants of the Language Prior

In this experiment, we remove the word embedding of the object category from the language prior so that only the word embedding of the verb category to identify is used as input for PAMF, LPFA, and LPCA. As shown in Table 4, without the word embedding of the object category, the performance of PD-Net drops by a large margin of 2.39% (1.87%) mAP in DT (KO) mode. These experimental results indicate that the word embedding of the object category in the language prior is an important hint to decipher the verb polysemy problem.

5.2.2 Comparisons with Variants of LPCA

In this experiment, we compare the performance of Language Prior-guided Channel Attention (LPCA) with five possible variants: namely, Plain Channel Attention (CA), ‘w/o \({\mathcal {S}}_{au}\)’, ‘w/o \({\mathcal {C}}_{att}\)’, ‘\(D([L_{A}, F_{A}])\)’ and FiLM (Perez et al. 2018). The other implementation details of PD-Net are kept the same for different variants. Plain CA means that we feed the appearance feature \(F_{A}\) directly into a plain CA module, i.e. \(D(\cdot )\) in Fig. 4, and obtain \({\tilde{F}}_{A}\). ‘w/o \(S_{au}\)’ involves removing the extra supervision signal \(S_{au}\) from LPCA, while ‘w/o \({C}_{att}\)’ means that we directly use \(L_{B}\) in Fig. 4 as the input of the \({\mathbf {H}}\) and \({\mathbf {O}}\) blocks in Fig. 3, without the further processing by the plain CA module. ‘\(D([L_{A}, F_{A}])\)’ means that we use the concatenation of \(L_{A}\) and \(F_{A}\) as the input for function \(D(\cdot )\) to generate channel attention scores in Eq. (5). FiLM means that we replace LPCA with a FiLM layer (Perez et al. 2018). Experimental results are tabulated in Table 5. In this table, ‘w/o LPCA’ is a baseline that removes the entire LPCA module from PD-Net. From these results, we can make the following observations.

First, the plain CA module alone slightly promotes the performance of PD-Net. One main reason for this is that the plain CA module has very little ability to identify important elements in the appearance features for each HOI category.

Table 5 Performance comparisons with variants for LPCA. \([\cdot ]\) represents feature concatenation operation
Full size table

Second, without supervision from \(S_{au}\), the performance of LPCA degrades dramatically. Compared to the plain CA module, this setting adopts language priors to provide cues regarding the channel-wise importance of \(F_{A}\) for each HOI category. However, it receives only implicit supervision from the binary score \({\mathcal {S}}^{\mathbf {PD}}_{(h,o,v)}\) in Eq. (9), which is too weak to optimize LPCA’s parameters. We therefore observe degraded performance after the extra supervision \({\mathcal {S}}_{au}\) is removed.

Third, ‘w/o \(C_{att}\)’ obtains better performance than both the ‘Plain CA’ and ‘w/o \(S_{au}\)’ settings. However, its performance is still lower than that of our proposed LPCA by 0.38% and 0.52% mAP in DT and KO modes, respectively. This may be because \(L_{A}\) is obtained via projection from the language prior. As word embeddings are not specifically designed for computer vision tasks, \(L_{A}\) may not always be reliable and the quality of \(L_{B}\) is affected (Xu et al. 2019). Therefore, further processing \(L_{B}\) using the plain CA module is helpful.

Fourth, LPCA outperforms \(D([L_{A}, F_{A}])\) by significant margins in both DT and KO modes. There are two main reasons for this. First, the concatenation operation significantly increases the model size of the channel attention module, which makes the model more difficult to train. Second, with the optimization on \({\mathcal {S}}_{au}\), \(L_{B}\) can provide more direct hints about important channels in \(F_{A}\) to the verb to verify than \([L_{A}, F_{A}]\).

Fifth, LPCA also outperforms FiLM (Perez et al. 2018) by significant margins in both DT and KO modes.This is because the feature-wise affine transformation in (Perez et al. 2018) is directly conditioned on the language prior; therefore, it can be affected by the semantic misalignments between visual features and word embeddings. In comparison, LPCA can better address the discrepancy because it produces channel attention scores conditioned on the correlation between language priors and visual features. Moreover, FiLM (Perez et al. 2018) is only supervised by the final classification loss of the model while LPCA is also optimized by an auxiliary supervision on \(S_{au}\).

In comparison, our proposed LPCA achieves the best performance for the following reasons. First, it adopts language priors to provide hints regarding the channel-wise importance of \(F_{A}\) for each HOI category. Second, it imposes direct supervision to the attention module, which helps to more effectively optimize the model parameters. Third, it refines the attention vector obtained from the language priors using a plain CA module, which enhances the quality of the channel attention vectors. The above experimental results and analysis demonstrate the effectiveness of LPCA.

Table 6 Performance comparisons between SH, SP, and CSP verb classifiers
Full size table

5.2.3 Comparisons with Variants of Verb Classifiers

To further demonstrate the advantages of clustering-based object-specific (CSP) classifiers, we compare their performance with that of object-SHared (SH) and object-SPecific (SP) verb classifiers. To facilitate fair comparison, other settings of PD-Net remain unchanged. Experimental results are tabulated in Table 6. It is shown that SH classifiers outperform SP classifiers by 1.42% (2.06%) mAP in DT (KO) mode for rare HOI categories. This is because SH classifiers enable these rare HOI categories to share verb classifiers with other HOI categories that have sufficient training data. By comparison, SP classifiers are better able to relieve the verb polysemy problem for the HOI categories that have sufficient training data. Therefore, the SP classifiers outperform SH classifiers by 0.23% (0.41%) mAP in DT (KO) mode for non-rare HOI categories.

Table 7 Performance comparisons on HICO-DET
Full size table

In comparison, CSP classifiers achieve superior performance on both rare and non-rare HOI categories. This is due to the same verb classifiers being assigned to semantically similar HOI categories, enabling HOI categories with few training samples to share verb classifiers with those HOI categories that have sufficient training data. Moreover, different verb classifiers are adopted for semantically different HOI categories, which is helpful to overcome the verb polysemy problem. Overall, CSP classifiers outperform SH and SP classifiers by 1.31% (2.03%) and 1.46% (2.19%) mAP in DT (KO) mode for the full HOI categories, respectively. The superior performance on rare HOI categories demonstrates the effectiveness of CSP classifiers in the few-shot learning ability. We also further justify the effectiveness of CSP classifiers in terms of zero-shot HOI detection in Section B of the supplementary file.

Table 8 Performance comparisons on V-COCO (Gupta and Malik 2015)
Full size table

5.3 Comparisons with State-of-the-Art Methods

We compare the performance of PD-Net with state-of-the-art methods on three databases, namely HICO-DET, V-COCO, and HOI-VP. Experimental results are summarized in Table 7, Table 8, and Table 10, respectively.

5.3.1 Performance Comparisons on HICO-DET

As shown in Table 7, PD-Net outperforms state-of-the-art methods by significant margins using both object detector backbones. It is worth noting that one most recent method, i.e. PPDM, adopts CenterNet with Hourglass-104 backbone (Zhou et al. 2019b) as the object detector. As shown in Table 2, this object detector significantly outperforms the two Faster R-CNN object detectors utilized in our model. To facilitate fair comparison, we mainly compare PPDM with PD-Net in the KO mode, as this mode is less affected by object detection results. As shown in Table 7, PD-Net outperforms PPDM in KO mode by significant margins of 2.28%, 5.05% and 1.60% mAP on the full, rare and non-rare HOI categories, respectively. Moreover, PD-Net also outperforms PPDM by 0.64% in the DT mode on the full HOI categories.

Moreover, as shown in Table 7 and Table 2, the object detector adopted by another recent work (Ulutan et al. 2020) is also much stronger than ours. But PD-Net still outperforms this model by large margins of 2.57% (22.37–19.80%), 1.56% (17.61–16.05%), and 2.88% (23.79–20.91%) mAP in the DT mode on the full, rare, and non-rare HOI categories, respectively.

Fig. 5
figure 5

The top 10 verbs that are most likely to suffer from the polysemy problem on HICO-DET

Full size image

Finally, with a similar multi-stream representation network and object detector backbone (ResNet-50-FPN), PD-Net outperforms one very recent model 2D-RN (Li et al. 2020a) by 3.03% (25.59%-22.56%) and 0.78% (20.76%-19.98%) in mAP on the full HOI categories in the KO and DT modes, respectively. Another advantage of PD-Net compared with 2D-RN is that PD-Net requires no extra human annotation. Besides, 3D human pose and 3D object locations are also utilized to improve 2D-RN during inference in (Li et al. 2020a). To facilitate fair comparisons, we only compare the performance of PD-Net with methods that utilize 2D human pose and 2D object locations during inference.

To further illustrate the advantage of PD-Net in deciphering the verb polysemy problem, we present the top 10 verbs (from the total 117 verbs in HICO-DET) ranked by the number of HOI categories in which each verb is included in Fig. 5. The largest number of HOI categories associated with the same verb (“hold”) is 61. As these verbs are more likely to be affected by the visual polysemy problem, we therefore compare the performance of PD-Net with one state-of-the-art method (Gupta et al. 2019) on these verbs. This method is chosen as it is very similar to our baseline. Results show that PD-Net achieves superior performance on all of these top 10 verbs.

Table 9 Per verb class AP (%) comparisons between PMFNet\(^{\dagger }\) and PD-Net\(^{\dagger }\) on V-COCO
Full size table

5.3.2 Performance Comparisons on V-COCO

To boost the performance on V-COCO, we add another appearance feature stream to both our baseline and PD-Net, following (Wan et al. 2019). There are consequently a total of five feature streams for experiments on V-COCO. This new stream extracts appearance features from union boxes composed of human–object pairs. We further apply LPCA to this feature stream in PD-Net. As shown in Table 8, PD-Net outperforms state-of-the-art methods by clear margins with both object detectors. In particular, PD-Net outperforms one of the most recently developed methods, i.e. VSGNet (Ulutan et al. 2020). As shown in Table 2, the object detector utilized by VSGNet is much stronger (Huang et al. 2017) than ours; nevertheless, PD-Net still outperforms VSGNet by clear margins, as indicated in Table 8.

Moreover, PD-Net outperforms another particularly strong model, named PMFNet (Wan et al. 2019) by 0.3% (52.3%-52.0%) in mAP. The excellent performance of PMFNet may benefit from the use of human part features. Therefore, we adopt the same five feature streams that include the human part features in PMFNet as input for PD-Net; this model is denoted as PD-Net\(^{\dag }\) in Table 8. The contributions in this paper remain unchanged. PD-Net\(^{\dag }\) outperforms PMFNet by a large margin of 1.3% (53.3%-52.0%) in mAP. Moreover, as shown in Table 9, we compare the performance between PD-Net\(^{\dagger }\) and PMFNet on each of the 24 verbs in V-COCO. Here, our method demonstrates superior performance on the vast majority of verb classes.

Table 10 Performance Comparisons on HOI-VP
Full size table
Fig. 6
figure 6

Attention scores produced by PAMF on four types of features. HOI categories in this figure share the same verb, i.e. “ride”. H, O, S, and P denote human appearance, object appearance, spatial feature, and human pose feature respectively

Full size image

5.3.3 Performance Comparisons on HOI-VP

We next compare the performance of PD-Net with some recent open-source methods, i.e. iCAN (Gao et al. 2018), TIN (Li et al. 2019b), No-Frills (Gupta et al. 2019), and PMFNet (Wan et al. 2019) on the new HOI-VP database. We also reproduce the method presented in (Peyre et al. 2019) that achieves high performance on the HICO-DET database. To facilitate fair comparison, we compare the performance of PD-Net with each of these methods using the same feature extraction backbone, respectively. As shown in Table 10, PD-Net consistently achieves the best performance out of all compared methods. In particular, PD-Net outperforms one recent powerful model PMFNet by a clear margin of 1.36% (63.66%-62.30%) in mAP. As the verbs (predicates) in the HOI-VP database are very common and polysemic in real world scenarios, experimental results on this database demonstrate the superiority of PD-Net to overcome the verb polysemy problem.

Fig. 7
figure 7

Visualization of PD-Net’s advantage in deciphering the verb polysemy problem on HICO-DET. We randomly select three verbs affected by the polysemy problem: “hold” (top row), “ride” (middle row), and “open” (bottom row). The green and red numbers denote the AP of our baseline and PD-Net respectively for the same HOI category (Color figure online)

Full size image
Fig. 8
figure 8

Visualization of PD-Net’s advantage in deciphering the verb polysemy problem on V-COCO. We randomly select three verbs affected by the polysemy problem: “carry” (top row), “ride” (middle row), and “hold” (bottom row)

Full size image
Fig. 9
figure 9

Visualization of PD-Net’s advantage in deciphering the verb polysemy problem on HOI-VP. We randomly select three verbs affected by the polysemy problem: “in” (top row), “play” (middle row), and “use” (bottom row)

Full size image

5.4 Qualitative Visualization Results

Figure 6 illustrates attention scores produced by PAMF for four types of features. HOI categories in this figure share the verb “ride”, but differ dramatically in semantic meanings. The “person” proposal in Fig. 6a is very small and severely occluded while the “airplane” proposal is very large; therefore, object appearance feature is much more important for verb classification than the human appearance feature. In Fig. 6b, both the spatial feature and the object appearance feature play important roles in determining the verb. Attention scores for Fig. 6c, d are similar, as \({<}{} \textit{person ride horse}{>}\) and \({<}{} \textit{person ride elephant}{>}\) are indeed close in semantics.

Figures 7, 8, and 9 provide more examples that demonstrate PD-Net’s advantages in deciphering the verb polysemy problem on HICO-DET, V-COCO, and HOI-VP, respectively. The performance gain by PD-Net compared with our baseline reaches 10.6%, 3.33%, and 48.1% in AP for the “open microwave”, “carry backpack”, and “play drum” category on the three datasets, respectively.

6 Conclusion

The verb polysemy problem is relatively underexplored and is sometimes even ignored in existing works for HOI detection. Accordingly, in this paper, we propose a novel model named PD-Net, which significantly mitigates the challenging verb polysemy problem. PD-Net includes four novel components: Language Prior-guided Channel Attention, Language Prior-based Feature Augmentation, Polysemy-Aware Modal Fusion, and Clustering-based Object Specific classifiers. Language Prior-guided Channel Attention and Language Prior-based Feature Augmentation are introduced to generate polysemy-aware visual features. Polysemy-Aware Modal Fusion highlights important feature types for each HOI category. The Clustering-based Object Specific classifiers not only relieve the verb polysemy problem, but also is capable of handling the zero- or few-shot learning problems. Exhaustive ablation studies are performed to demonstrate the effectiveness of these components. We further develop and present a new dataset, named HOI-VP, that is specifically designed to expedite the research on the verb polysemy problem for HOI detection. Finally, by decoding the verb polysemy, we achieve state-of-the-art methods on the three HOI detection benchmarks. In the future, we will study the verb polysemy problem in related tasks to HOI detection, e.g., visual relationship detection and action recognition.