Track Targets by Dense Spatio-Temporal Position Encoding
Jinkun Cao (Carnegie Mellon University),* Hao Wu (Bytedance Inc.), Kris Kitani (Carnegie Mellon University)
The 33rd British Machine Vision Conference
Abstract
In this work, we propose a novel paradigm to encode the position of targets for target tracking in videos using transformers. The proposed paradigm, Dense Spatio-Temporal (DST) position encoding, encodes spatio-temporal position information in a pixel-wise dense fashion. The provided position encoding provides location information to associate targets across frames beyond appearance matching by comparing objects in two bounding boxes. Compared to the typical transformer positional encoding, our proposed encoding is applied to the 2D CNN features instead of the projected feature vectors to avoid losing positional information. Moreover, the designed DST encoding can represent the location of a single-frame object and the evolution of the location of the trajectory among frames uniformly. Integrated with the DST encoding, we build a transformer-based multi-object tracking model. The model takes a video clip as input and conducts the target association in the clip. It can also perform online inference by associating existing trajectories with objects from the new-coming frames. Experiments on video multi-object tracking (MOT) and multi-object tracking and segmentation (MOTS) datasets demonstrate the effectiveness of the proposed DST position encoding.Video
Citation
@inproceedings{Cao_2022_BMVC,
author = {Jinkun Cao and Hao Wu and Kris Kitani},
title = {Track Targets by Dense Spatio-Temporal Position Encoding},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0311.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by