CAPN: a Combine Attention Partial Network for glove detection

Object detection method based on deep learning

Object detection is one of the most classic problems in computer vision (Li et al., 2021; Kattenborn et al., 2021). In object detection networks (Cengil, Çinar & Yildirim, 2021), features are extracted using convolutional neural networks, which have been widely applied in fields such as classification (Wang & Chen, 2019; Shahverdy et al., 2020), object detection, and object segmentation. Currently, there are different methods for glove detection, with most methods using sensors to detect whether gloves are being worn (Barfidokht et al., 2019). In some dangerous situations, object detection is rarely used to identify workers wearing gloves. Object detection networks can identify workers wearing gloves to improve safety in dangerous areas. Currently, object detection networks can be divided into two types: two-stage object detection algorithms and one-stage object detection algorithms.

Transfer learning

Transfer learning is a subfield of machine learning (Hart et al., 2021) that involves retaining useful information from previously learned data to solve new problems faster and more effectively. Due to the lack of data in specific tasks, transfer learning is widely used in image classification and object detection experiments. Many studies have shown that using pre-trained convolutional neural networks (CNNs) for transfer learning produces better results than training from scratch. Therefore, an object detection framework for deep CNNs is important in the base network. For instance, GoogleLeNet (Szegedy et al., 2015), VGG Network (Simonyan & Zisserman, 2014), and ResNet (He et al., 2016) all utilize pre-training weights from the ImageNet dataset and demonstrate superior performance. To detect micro-aperture radar, an object detection method utilizing transfer learning is proposed, which extracts features from the three-channel sub-aperture (Wang et al., 2018) data obtained from the existing Synthetic Aperture Radar (SAR) target recognition dataset and the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset. The Singer Shot Detector (SSD) (Liu et al., 2016) algorithm is then used to detect the target, resulting in improved detection performance. This approach enables end-to-end trainable aircraft detection with a single deep CNN and limited training samples (Chen, Zhang & Ouyang, 2018), demonstrating high average precision and significant application potential for remote sensing target detection.

Attention mechanism

Attention mechanisms are often used in neural network structures to better extract key information while ignoring unimportant information. There are three common attention mechanisms: spatial attention, channel attention, and self-attention mechanisms. Point-Wise Spatial Network (PSANet) (Zhao et al., 2018) is proposed to reduce local neighborhood constraints. Each location on the feature map is connected to all other locations through an adaptive learned attention mask, and this approach achieved good results on the ADE20K (Zhou et al., 2017) and PASCAL VOC 2012 datasets. Semantic Segmentation Network with Spatial and Channel Attention (SCAttNet) (Li et al., 2020) is proposed for high-resolution remote sensing image segmentation, and it has achieved significant improvement. Squeeze-and-Excitation Networks (SE-Net) (Hu, Shen & Sun, 2018) are proposed to adaptively calibrate the feature responses of channel directions, forming a SENet architecture that efficiently generalizes across different datasets and further improves the feature extraction ability of CNNs. The self-attention module (Vaswani et al., 2017) is initially used in the field of natural language processing (Liu et al., 2021). It uses a fixed-length vector in the image field, and then applies keys, queries, and values to perform global modeling, thus improving the global information extraction ability. The Convolutional Block Attention Module (CBAM) (Woo et al., 2018) is proposed, which sequentially performs the attention mechanism along two independent dimensions of channel and space. Then, the attention map is multiplied by the input feature map for adaptive feature refinement. This article also makes reasonable use of this module to improve the feature extraction of gloves, which in turn improves recognition precision.

CAPN Framework

The feature extraction network module in the CAPN structure can effectively extract gloves during glove detection, and the improved CSP-Darknet53 (Wang et al., 2020) is used as the backbone network of CAPN to extract features from the input image. As shown in Fig. 1, the CBL and CSP modules are used in the feature extraction module. The CBL module is composed of ordinary convolution and Deep-wise (DW) (Howard et al., 2019) convolution. The DW convolution module is first applied in the MobileNet network structure, where the DW convolution performs convolution operations on each input channel separately and uses a 1 × 1 convolution on the output channel to integrate the results of all input channels. The CSP module, inspired by the residual module, constructs a large residual block to deepen the network, prevent gradient vanishing, and extract deeper feature information. The combination of CBL and CSP modules achieves higher precision and detection speed. Therefore, the efficient backbone network model speeds up calculation speed and reduces training parameters, making it possible to detect workers wearing gloves in real-time at the electric power operation site.

In the feature extraction network structure of CAPN, three CBL modules are used, in which the CBL module is a network structure composed of DW convolution and ordinary convolution. After each CBL module, a large residual CSP module is used. The overall design idea is to use DW convolution to independently perform a 1 × 1 convolution operation on the information channel of each input layer, the feature map of each feature channel will be obtained by using the ordinary convolution module to perform feature extraction again, and finally all the feature maps will be added. Owing to the DW convolution is to perform feature extraction on the information on each individual channel, the information across channels (Rezaee et al., 2021) is not considered, so the advantages of ordinary convolution are used to complement each other, so as to better extract the features of the image. This article use the CSP module that can deepen the network depth to extract information from the shallow feature network. This module first passes through a common convolution module, and then stacks residual modules. This residual module is composed of a convolution module, CBL module and a residual block. Finally, the output feature information is added to the CSP input feature information to form a large residual block. This module can extract the deep information of the image, which is divided into two parts. One part enters the residual block to improve the operation speed, and the other part is added to the feature map after feature extraction to fuse the information, while extracting deeper feature information, the information of the original feature map is preserved and fused, which effectively improves the extraction of image information. Convolution operations are used in the entire feature extraction network frequently, and Batch Normalization (Ioffe & Szegedy, 2015) is used after each convolution to make the overall data satisfy the distribution law with a mean value of 0 and a variance of 1, and use the Sigmoid Linear Unit (SiLu) activation function for unified processing.

The feature pyramid layer (Ghiasi, Lin & Le, 2019) in the CAPN fuses information from different levels in the backbone network. First, the output layer in the feature extraction network is up-sampled, and the resulting feature map after passing through the CAP module is combined with the second CBL feature layer in the backbone network. This feature map is then convoluted again and subjected to feature fusion with the first CBL feature layer in the backbone network after up-sampling and CAP module features. Second, after the obtained feature map passes through the CAP module, a downsampling operation is performed immediately, and it is fused with the convolutional features in the FPN network. Finally, after downsampling and the first CBL feature layer in the backbone network, the feature fusion is sent to the CAP module to obtain the prediction result. The CAP module is a model component used for image processing, consisting of the CSP module, channel attention module, and spatial attention module. The CSP module is a convolutional neural network module used to extract features from images. It divides the input feature map into two parts, processes them differently, and then merges them. This separation and merging method can effectively reduce the number of parameters. Both the channel attention module and the spatial attention module are attention mechanisms. The channel attention module is used to adaptively adjust the weights of different channels in the input feature map to better capture the important features of input data. The spatial attention module is used to adaptively adjust the weights of different positions in the input feature map to better capture the spatial information of input data. Through the combination of these modules, the CAP module can better extract deeper information from image data. It can improve the training efficiency and generalization performance of the model, leading to better results in image processing tasks. By using the FPN pyramid, the feature layers from different levels in the backbone network can be fused, effectively combining the deep and shallow information and retaining image information on the original feature layer, which improves the recognition precision of the network. The CAPN structure is better explained in Table 1.

CAP Attention mechanism

To address the challenge of diverse glove types and small glove sizes during glove detection at electric power work sites, a solution is proposed to extract deeper information from the input image for accurate learning of small targets such as gloves. CAP attention module is proposed to be added between convolutional layers to enhance the network’s ability to extract image features, as shown in Fig. 2. The module undergoes a convolution operation, followed by three small residual modules, and finally an attention mechanism module, which combines channel attention and spatial attention to selectively amplify informative features and suppress less informative ones.

The channel attention module has two branches: average pooling and global pooling (Chen et al., 2021). Each branch generates a (C, 1, 1) weight vector. Average pooling, which slides a window over the feature map, preserves background information and reduces overfitting (Ying, 2019). Maximum pooling extracts feature textures and reduces the influence of irrelevant information. The weight vectors from both branches are fully connected and processed by the ReLU activation function before being added. Finally, the resulting vector is mapped to the dimensional information of each channel. This process preserves the image background information and extracts texture features simultaneously. The calculation formula for this process is shown in Eq. (1): (1)${\displaystyle {\text{Chanel}}_c=\sigma \left(MLP\left(AvgPool\left(F\right)\right)+MLP\left(MaxPool\left(F\right)\right)\right)=\sigma \left(W_1\left(W_0\left(F_{\text{avg}}^c\right)\right)+W_1\left(W_0\left(F\left(c_{\text{Max}}\right)\right)\right)\right)}$ where σ is the Sigmoid activation function, MLP is the weighted layer of the fully connected layer, W₀ and W₁ correspond to the weight parameters of the fully connected layer respectively.

After multiplying the output result with the input feature vector, it is passed through a spatial attention module. The feature maps then undergo average pooling and maximum pooling, respectively, to obtain two weight vectors of size (1, H, W). The number of channels changes from C to 1, and the two weight vectors are stacked to form weight information of size (2, H, W) after convolution processing. This weight information is then multiplied with the original feature map, where each point on the original feature map is given a weight change. This process amplifies the values with larger weights in the original feature map and emphasizes the importance of information, as shown in Eq. (2): (2)${\displaystyle {\text{Spatial}}_c=\sigma \left(f^{7\times 7}\left(\text{concat}\left(AvgPool\left(F\right),MaxPool\left(F\right)\right)\right)\right)=\sigma \left(f^{7\times 7}\left(\left[F_{avg}^s;F_{max}^s\right]\right)\right)}$ where σ is the Sigmoid activation function, f^7×7 means that the size of the convolution kernel used in the convolution process is 7 × 7, concat is the channel splicing of the weight vector obtained after global pooling and average pooling.

Finally, after being activated by the Sigmoid function, a multiplication operation is performed with the original input feature to form a large residual edge output. The CAP module helps to represent important information in feature maps and improve detection precision.

Transfer learning

The CAPN model is employed in this article to detect glove objects, requiring a large number of training samples. However, the availability of such training samples is limited for the specific glove detection task, leading to the challenge of learning a good CAPN model. One reason for the lack of sufficient training samples is the high cost and difficulty of collecting and annotating glove images in real-world electric power scenarios.

In this article, a data augmentation method suitable for object detection in gloves is proposed. To address this issue, several data augmentation techniques, such as random cropping, rotation, and flipping, are explored to generate additional training samples and increase the diversity of the dataset. Transfer learning from other related tasks or domains, such as hand detection or object detection in general, is also considered to pretrain the CAPN and improve its initialization and feature representation. Alternatively, synthetic data generated by computer graphics or simulation tools may be used to augment the training set and increase the variability of the images.

Transfer learning is utilized in this study to extract useful knowledge from a source dataset and apply it to the target task. The original model is trained on the dataset of human hand features using CAPN. The learned model parameters W are then used as the starting point for training on glove data, without performing reverse gradient updates in the backbone network. The goal is to transfer knowledge from the source domain of human hands to the target domain of gloves.

To achieve this, the feature information of human hands is used to learn the feature information of gloves. In the network, X and Y are used as the input and output of the original network. Let S^m(x) be the feature expression of the mth layer in the pre-trained source network and T_w be the network to be learned in the target domain. Then $T_w^n\left(x\right)$ is the feature expression of the n-th layer in the target domain network, where w is the set of parameters to be learned, as shown in the following Equation (3): (3)${\displaystyle \parallel r_w\left(T_w^n\left(x\right)\right)-S^m\left(x\right){\parallel }_2^2}$ where r_w is a linear transformation of the input matrix, the m-th layer and n-th layer in the above formula perform weight transfer, which is the overall transfer learning goal.

Considering that not all layer features in the source domain are conducive to the target task, so a weight matrix, γ^m,n > 0 is proposed to represent the transferable index of the m-th layer in the source domain to the n-th layer in the target domain, the larger metric, the more migrations can be made, and the smaller metric, the less migrations, and parameters can be expressed as Equation (4): (4)${\displaystyle {\gamma }^{m,n}>0=g_{\theta }^{m,n}\left(S^m\left(x\right)\right)}$ where the θ parameter is the learning target. The weight of the network that can be migrated according to the index of the weight matrix. As shown in Fig. 3, this training process enables the efficient transfer of knowledge acquired from source datasets, thereby improving the learning ability of target tasks using relatively small datasets, reduces the negative impact of the small number of datasets significantly, and improved network extraction of gloves features.

Data set

The experimental data includes two self-made datasets: the hand dataset and the gloves dataset. The hand dataset comprises 8,000 natural-state images of human hands captured in various environments. Among these, 7,200 images are used for training, 720 images are used for validation, and 800 images are used for testing the source domain data model. The hand dataset is obtained from hand captures in various scenarios, which effectively ensures the state characteristics of hands in different postures. The gloves dataset is obtained from outdoor and indoor field shooting of electric power work sites, and contains 1,000 images. Of these, 900 images are used for training, 90 images are used for validation, and 100 images are used for testing the source domain data model. As obtaining datasets in power operation scenarios is difficult and requires consideration of privacy concerns, we created a small dataset consisting of 1,000 samples. The uniqueness of these datasets contributes to their significance and theoretical value in the research.

Pre-training

In this study, the hand recognition model is first pre-trained using a hand dataset, and the pre-trained weights are not used during the subsequent training on the hand dataset, resulting in a model capable of recognizing hands. The learned hand features are then transferred to the glove training process, as described in the transfer learning section. The platform, environment version, optimizer, and learning rate used for both training processes are consistent. Prior to network training, data augmentation (Bayer, Kaufhold & Reuter, 2022) operations are performed on the dataset to increase its size and diversity. Specifically, four images are randomly selected and stretching, scaling, and rotation operations are applied to a portion of each image, along with its corresponding label information. The augmented images are then recombined into a spliced image. This process enhanced the information of the detected objects and increased the batch size, thereby improving the performance of the model. The specific image enhancement operation is shown in Fig. 4. By utilizing this method, the dataset is expanded, and the background information of the detection targets is enriched, effectively improving the diversity of the dataset.

Evaluation criterion

The experiment uses precision rate (P), recall rate (R), F1-score (F1), average precision (AP), intersection over union (IoU), and log-average miss rate (MR-2) as the main evaluation metrics, which are defined in the following Eqs. (5), (6), (7) and (8): (5)${\displaystyle \text{Precision}=\frac{TP}{TP+FP}}$ (6)${\displaystyle F1=\frac{2\times P\times R}{P+R}}$ (7)${\displaystyle \text{Recall}=\frac{TP}{TP+FN}}$ (8)${\displaystyle \text{MR}\text{-}{2}_i=\frac{1}{\left|\text{gt}i\right|}\sum {j=1}^{\left|\text{gt}i\right|}log\left(max\left(1-\text{IoU}\left(d_j,\text{gt}i,k\right)\right)\right)}$ where the true positives (TP) refer to the cases where the predicted value matches the ground truth, and the predicted value is a positive sample. False positives (FP) refer to the cases where the predicted value differs from the ground truth, but the predicted value is a positive sample. False negatives (FN) refer to the cases where the predicted value differs from the ground truth, but the predicted value is a negative sample.

Precision (P) is the ratio of true positive samples to all predicted positive samples, while Recall (R) is the ratio of true positive samples to all true positive samples. The F1 score (F1) is a single metric that combines both precision and recall. Average precision (AP) is the area under the Precision-Recall curve and is a widely used performance metric for object detection. A higher AP indicates better performance. The log-average miss rate (MR-2) is a performance metric for object detection that measures the logarithmically averaged false positive rate at different levels of recall. |gti| denotes the number of ground truth annotations for the ith class in the dataset, gti, k denotes the kth ground truth bounding box for the ith class, d_j denotes the jth predicted bounding box, and IoU(d_j, gt_i,k) denotes the IoU between the jth predicted bounding box and the kth ground truth bounding box for the ith class.

Comparative experiments

To evaluate the effectiveness of the CAPN algorithm for detecting gloves in power operation sites, comparison experiments are conducted with several state-of-the-art object detection algorithms, including Faster-RCNN, SSD, YOLOV3, YOLOV4, YOLOV5, YOLOX, and YOLOV7. The experiments are designed to assess the performance of each algorithm in detecting workers wearing gloves, using standard metrics such as precision, recall, F1 score ,and MR-2. Currently, YOLO X is considered one of the top-performing object detection algorithms. It utilizes an Anchor-free approach, which predicts the coordinates of each prediction center point with only four parameters, reducing the number of parameters and improving the calculation speed of the target detection algorithm. To build on this approach, an anchor-free CAPN based on the predictive head approach is proposed, which can efficiently detect workers wearing gloves in real-time in power operation sites.

The training approach for the glove dataset in this study involves first training a hand weight model using the hand dataset, and then using this weight model as a pre-training weight for training the glove dataset. The training process involves two stages. During the first 50 epochs, the weights of the backbone network are frozen to speed up the training process and prevent weight destruction. At this stage, only the feature extraction network can update its weights, and the model training involves only fine-tuning the feature extraction network to reduce GPU memory usage. In the following 250 epochs, all network weights can be updated, and the entire model participates in the training process to further improve model performance.

Table 2 shows the model experimental results of seven classic networks compared with the proposed method on the glove data set. It can be concluded that the proposed CAPN has achieved good results in precision, recall, and average precision. Detecting workers wearing gloves is important in power operation sites because it can help prevent electrical accidents and ensure the safety of workers. By using the CAPN algorithm, the precision and efficiency of glove detection can be improved, which can ultimately contribute to a safer workplace.

Ablation experiments

To comprehensively evaluate the proposed CAPN in this article and understand the performance of each module, an ablation experiment is conducted. The goal of this experiment is to gradually split each part of the network and assess its impact on the overall performance of the model. By doing so, the contribution of each module to the network’s precision and recall can be fully evaluated.

Specifically, the ablation experiment is divided into four parts. The first part excluded the transfer learning and attention mechanism from the CAPN network. In the second part, the attention mechanism is added to the network but the transfer learning mechanism is not included. In the third part, the transfer learning mechanism is added to the network but the attention mechanism is not included. Finally, in the fourth part, both mechanisms are introduced. Table 3 displays the results of the ablation experiment, which show the performance of the network with and without each of the modules.

Based on the experimental results presented in Table 3, it is observed that the introduction of the attention mechanism module improved the network’s recognition precision for targets but reduced the recall of the targets. This is due to the attention mechanism’s deepening of the extraction of detailed features, which resulted in the loss of some contextual information. However, it effectively improved the precision of recognizing gloves with smaller targets.

Furthermore, during the transfer learning process in this study, it is observed that the introduction of attention mechanisms did not significantly improve the recall of the target, while the recall rate is greatly enhanced. The reason for this is that transfer learning commonly utilizes pre-trained models that are trained on large benchmark datasets. As the features of hands are similar to those of gloves in the electrical work scene, the weights from the hand dataset features are effectively utilized to enhance the network’s recall of gloves. This led to an improvement in recognition recall, even with a limited glove dataset of only 1,000 samples, effectively preventing missed inspections of gloves at power work sites.

After introducing both the attention mechanism and transfer learning, it is observed that there is an improvement in both precision and recall of the network. These two modules complemented each other and enhanced the precision and stability of target recognition, ensuring effective identification of gloves at the power work site. In conclusion, this article utilizes the combination of attention mechanism and transfer learning to improve the average precision and stability of target recognition in the electrical work scene.

Seed experiments

Using different random seeds to verify the stability of the network is a common practice in object detection networks. A random seed is an initial value used to generate a sequence of random numbers. In deep learning, random seeds are typically used for initializing model parameter. Therefore, different random seeds can lead to different model parameters and random operations, which can affect the training and performance of the model. In this study, different seed experiments usually refer to using different random seeds to initialize the model’s weight parameters under the same dataset and model settings, and using different standard deviation scaling methods to compare the performance differences between different seed experiments. The purpose of doing this is to test the stability and generalization ability of the model under different random initializations, and thus better evaluate the model’s performance. By using different random seeds, the performance of the model can be more comprehensively evaluated, and the impact of incidental results caused by a particular random seed on experimental conclusions can be reduced.

In this study, three different seeds are used for experimentation. The detailed results are shown in Table 4. For seed 1, the weights are initialized using the “Normal” method, which initializes the weights using a standard normal distribution. This initialization method helps the model to converge faster and fit the data better. During initialization, the mean is kept at 0, and standard deviation scaling of 0.01, 0.02, and 0.03 are used, respectively.

For seed 2, the weights are initialized using the “Xavier” method, which initializes the weights using a normal distribution with mean 0 and variance ${\sigma }^2=\frac{2}{n_{in}+n_{out}}$, where n_in and n_out are the input and output channel numbers of the weight matrix. The same standard deviation scaling of 0.01, 0.02, and 0.03 are used, respectively.

For seed 3, the weights are initialized using the “Kaiming” method, which is an orthogonal initialization method that initializes the weights based on an orthogonal matrix to help the model converge faster. The same standard deviation scaling of 0.01, 0.02, and 0.03 are used, respectively.

In summary, using different random seeds to verify the stability of the network is an effective experimental method that can help us more comprehensively evaluate the performance and stability of the model. In this article, it has been demonstrated by experiments that using the Normal weight initialization method with a standard deviation scaling of 0.01 can achieve the best results, with an average precision of 96.59%.

Detection results

The results of the algorithm are visualized by testing glove detection in a set of real-world power operation scenarios. As shown in Fig. 5, the test data samples are collected from real power operation scenarios and are different from the original dataset. The detection results demonstrate that the proposed algorithm has good robustness and can accurately identify whether the operator is wearing gloves correctly in real-world power operation scenarios. For example, in (a), the transfer learning method employed in this article transfers hand features to the recognition of similar target gloves, allowing the network to better extract the features of hands wearing gloves. Thus, even with a small sample size of the dataset, gloves can still be accurately identified under occlusion. In (b), gloves can be accurately identified under good light and normal power operation conditions, as the features are more distinct. In (c), due to the attention mechanism introduced in the network structure, which can extract channel and spatial feature information at a deeper level, gloves can be accurately identified even under small target conditions, such as insufficient indoor ambient light and long distance from the target. Experimental results show that this method can improve recall rate and effectively enhance the recognition of small targets. Overall, the proposed algorithm can detect gloves in real time in various complex scenarios, effectively ensuring the personal safety of workers in power operation scenes.