Dual Decision Improves Open-Set Panoptic Segmentation

Haiming Xu (The University of Adelaide),* Hao Chen (Huawei Noah's Ark Lab), Lingqiao Liu (University of Adelaide), Yufei Yin (University of Science and Technology of China)
The 33rd British Machine Vision Conference


Open-set panoptic segmentation (OPS) problem is a new research direction aiming to perform segmentation for both \known classes and \unknown classes, i.e., the objects (``things'') that are never annotated in the training set. The main challenges of OPS are twofold: (1) the infinite possibility of the \unknown object appearances makes it difficult to model them from a limited number of training data. (2) at training time, we are only provided with the ``void'' category, which essentially mixes the ``unknown thing'' and ``background'' classes. We empirically find that directly using ``void'' category to supervise \known class or ``background'' classifiers without screening will lead to an unsatisfied OPS result. In this paper, we propose a divide-and-conquer scheme to develop a dual decision process for OPS. We show that by properly combining a \known class discriminator with an additional class-agnostic object prediction head, the OPS performance can be significantly improved. Specifically, we first propose to create a classifier with only \known categories and let the ``void'' class proposals achieve low prediction probability from those categories. Then we distinguish the ``unknown things'' from the background by using the additional object prediction head. To further boost performance, we introduce ``unknown things'' pseudo-labels generated from up-to-date models to enrich the training set. Our extensive experimental evaluation shows that our approach significantly improves \unknown class panoptic quality, with more than 30\% relative improvements than the existing best-performed method.



author    = {Haiming Xu and Hao Chen and Lingqiao Liu and Yufei Yin},
title     = {Dual Decision Improves Open-Set Panoptic Segmentation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0190.pdf}

Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection