PAUMER: Patch Pausing Transformer for Semantic Segmentation
Evann Courdier (Idiap Research Institute),* Prabhu Teja Sivaprasad (Idiap Research Institute), François Fleuret (University of Geneva)
The 33rd British Machine Vision Conference
Abstract
We study the problem of improving the efficiency of segmentation transformers by using disparate amounts of computation different parts of the image. Our method, PAUMER, accomplishes this by pausing computation for patches that are deemed to not need any more computation before the final decoder. We use the entropy of predictions computed from intermediate activations as the pausing criterion, and find this aligns well with semantics of the image. Our method has a unique advantage that a single network trained with the proposed strategy can be effortlessly adapted at inference to various run-time requirements by modulating its pausing parameters. On two standard segmentation datasets, Cityscapes and ADE20K, we show that our method operates with about a 50% higher throughput with an mIoU drop of about 0.65% and 4.6% respectively.Video
Citation
@inproceedings{Courdier_2022_BMVC,
author = {Evann Courdier and Prabhu Teja Sivaprasad and François Fleuret},
title = {PAUMER: Patch Pausing Transformer for Semantic Segmentation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0737.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by