Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation
Zhaochong An (ETH Zurich), Guolei Sun (ETH Zurich),* Zongwei WU (University of Wurzburg), Hao Tang (ETH Zurich), Luc Van Gool (ETH Zurich)
The 34th British Machine Vision Conference
Abstract
Modern approaches have proved the huge potential of addressing semantic segmentation as a mask classification task which is widely used in instance-level segmentation. This paradigm trains models by assigning part of object queries to ground truths via conventional one-to-one matching. However, we observe that the popular video semantic segmentation (VSS) dataset has limited categories per video, meaning less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. This inefficiency limits the full expressive potential of all queries.Thus, we present a novel solution THE-Mask for VSS, which introduces temporal-aware hierarchical object queries for the first time. Specifically, we propose to use a simple two-round matching mechanism to involve more queries matched with minimal cost during training while without any extra cost during inference. To support our more-to-one assignment, in terms of the matching results, we further design a hierarchical loss to train queries with their corresponding hierarchy of primary or secondary. Moreover, to effectively capture temporal information across frames, we propose a temporal aggregation decoder that fits seamlessly into the mask-classification paradigm for VSS. Utilizing temporal-sensitive multi-level queries, our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.Citation
@inproceedings{An_2023_BMVC,
author = {Zhaochong An and Guolei Sun and Zongwei WU and Hao Tang and Luc Van Gool},
title = {Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://bmvc2022.mpi-inf.mpg.de/BMVC2023/0542.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by