PAUMER: Patch Pausing Transformer for Semantic Segmentation


Evann Courdier (Idiap Research Institute),* Prabhu Teja Sivaprasad (Idiap Research Institute), Fran├žois Fleuret (University of Geneva)
The 33rd British Machine Vision Conference

Abstract

We study the problem of improving the efficiency of segmentation transformers by using disparate amounts of computation different parts of the image. Our method, PAUMER, accomplishes this by pausing computation for patches that are deemed to not need any more computation before the final decoder. We use the entropy of predictions computed from intermediate activations as the pausing criterion, and find this aligns well with semantics of the image. Our method has a unique advantage that a single network trained with the proposed strategy can be effortlessly adapted at inference to various run-time requirements by modulating its pausing parameters. On two standard segmentation datasets, Cityscapes and ADE20K, we show that our method operates with about a 50% higher throughput with an mIoU drop of about 0.65% and 4.6% respectively.

Video



Citation

@inproceedings{Courdier_2022_BMVC,
author    = {Evann Courdier and Prabhu Teja Sivaprasad and Fran├žois Fleuret},
title     = {PAUMER: Patch Pausing Transformer for Semantic Segmentation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0737.pdf}
}


Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection