Hierarchical Contrast for Unsupervised Skeleton-Based Action Representation Learning

Authors

  • Jianfeng Dong College of Computer Science and Technology, Zhejiang Gongshang University Zhejiang Key Lab of E-Commerce
  • Shengkai Sun College of Computer Science and Technology, Zhejiang Gongshang University
  • Zhonglin Liu College of Computer Science and Technology, Zhejiang Gongshang University
  • Shujie Chen College of Computer Science and Technology, Zhejiang GongShang University Zhejiang Key Lab of E-Commerce
  • Baolong Liu College of Computer Science and Technology, Zhejiang Gongshang University Zhejiang Key Lab of E-Commerce
  • Xun Wang College of Computer Science and Technology, Zhejiang Gongshang University Zhejiang Key Lab of E-Commerce

DOI:

https://doi.org/10.1609/aaai.v37i1.25127

Keywords:

CV: Representation Learning for Vision, CV: Video Understanding & Activity Analysis, CV: Motion & Tracking, CV: Image and Video Retrieval

Abstract

This paper targets unsupervised skeleton-based action representation learning and proposes a new Hierarchical Contrast (HiCo) framework. Different from the existing contrastive-based solutions that typically represent an input skeleton sequence into instance-level features and perform contrast holistically, our proposed HiCo represents the input into multiple-level features and performs contrast in a hierarchical manner. Specifically, given a human skeleton sequence, we represent it into multiple feature vectors of different granularities from both temporal and spatial domains via sequence-to-sequence (S2S) encoders and unified downsampling modules. Besides, the hierarchical contrast is conducted in terms of four levels: instance level, domain level, clip level, and part level. Moreover, HiCo is orthogonal to the S2S encoder, which allows us to flexibly embrace state-of-the-art S2S encoders. Extensive experiments on four datasets, i.e., NTU-60, NTU-120, PKU-I and PKU-II, show that HiCo achieves a new state-of-the-art for unsupervised skeleton-based action representation learning in two downstream tasks including action recognition and retrieval, and its learned action representation is of good transferability. Besides, we also show that our framework is effective for semi-supervised skeleton-based action recognition. Our code is available at https://github.com/HuiGuanLab/HiCo.
AAAI-23 Proceedings Cover

Downloads

Published

2023-06-26

How to Cite

Dong, J., Sun, S., Liu, Z., Chen, S., Liu, B., & Wang, X. (2023). Hierarchical Contrast for Unsupervised Skeleton-Based Action Representation Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(1), 525-533. https://doi.org/10.1609/aaai.v37i1.25127

Issue

Vol. 37 No. 1: AAAI-23 Technical Tracks 1

Section

AAAI Technical Track on Computer Vision I