Spatio-Temporal Learnable Proposals for End-to-End Video Object Detection
Khurram Azeem Hashmi (Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) GmbH),* Didier Stricker (DFKI), Muhammad Zeshan Afzal (Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) GmbH)
The 33rd British Machine Vision Conference
Abstract
This paper presents the novel idea of generating object proposals by leveraging temporal information for video object detection. The feature aggregation in modern region-based video object detectors heavily relies on learned proposals generated from a single frame RPN. This imminently introduces additional components like NMS and produces unreliable proposals on low-quality frames. To tackle these restrictions, we present SparseVOD, a novel video object detection pipeline that employs Sparse R-CNN to exploit temporal information. In particular, we introduce two modules in the dynamic head of Sparse R-CNN. First, the Temporal Feature Extraction module based on the Temporal RoI Align operation is added to extract the RoI proposal features. Second, motivated by sequence-level semantic aggregation, we incorporate the attention-guided Semantic Proposal Feature Aggregation module to enhance object feature representation before detection. The proposed SparseVOD effectively alleviates the overhead of complicated post-processing methods and makes the overall pipeline end-to-end trainable. Extensive experiments show that our method significantly improves the single-frame Sparse RCNN by 8%-9% in mAP. Furthermore, besides achieving state-of-the-art 80.3% mAP on the ImageNet VID dataset with ResNet-50 backbone, our SparseVOD outperforms existing proposal-based methods by a significant margin on increasing IoU thresholds (IoU > 0.5).Video
Citation
@inproceedings{Hashmi_2022_BMVC,
author = {Khurram Azeem Hashmi and Didier Stricker and Muhammad Zeshan Afzal},
title = {Spatio-Temporal Learnable Proposals for End-to-End Video Object Detection},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0018.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by