Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation
Shuaicheng Li (Sensetime Research),* Feng Zhang (Fudan University), Rui-Wei Zhao (Fudan University), Kunlin Yang (Sensetime Group Limited), Lingbo Liu (Hong Kong Polytechnic University), Rui Feng (Fudan University), Jun Hou (SenseTime Group Limited)
The 33rd British Machine Vision Conference
Abstract
It has been found that temporal action proposal generation, which aims to discover the temporal action instances within the range of the start and end frames in the untrimmed videos, can largely benefit from proper temporal and semantic context exploitation. The latest efforts were dedicated to considering the temporal context and similarity-based semantic context through self-attention modules. However, they still suffer from cluttered background information and limited contextual feature learning. In this paper, we propose a novel Pyramid Region-based Slot Attention (PRSlot) module to address these issues. Instead of using the similarity computation, our PRSlot module directly learns the local relations in an encoder-decoder manner and generates the representation of a local region enhanced based on the attention over input features called \textit{slot}. Specifically, upon the input snippet-level features, PRSlot module takes the target snippet as \textit{query}, its surrounding region as \textit{key} and then generates slot representations for each \textit{query-key} slot by aggregating the local snippet context with a parallel pyramid strategy. Based on PRSlot modules, we present a novel Pyramid Region-based Slot Attention Network termed PRSA-Net to learn a unified visual representation with rich temporal and semantic context for better proposal generation. Extensive experiments are conducted on two widely adopted THUMOS14 and ActivityNet-1.3 benchmarks. Our PRSA-Net outperforms other state-of-the-art methods. In particular, we improve the AR@100 from the previous best 50.67% to 56.12% for proposal generation and raise the mAP under 0.5 tIoU from 51.9\% to 58.7\% for action detection on THUMOS14.Video
Citation
@inproceedings{Li_2022_BMVC,
author = {Shuaicheng Li and Feng Zhang and Rui-Wei Zhao and Kunlin Yang and Lingbo Liu and Rui Feng and Jun Hou},
title = {Pyramid Region-based Slot Attention Network for Temporal Action Proposal Generation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0424.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by