SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models
Omiros Pantazis (University College London),* Gabriel Brostow (University College London), Kate Jones (University College London), Oisin Mac Aodha (University of Edinburgh)
The 33rd British Machine Vision Conference
Abstract
Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs, and have been shown to sometimes exhibit impressive zero- and low-shot image classification performance. However, due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required. To combat this, a series of light-weight adaptation methods have been proposed to efficiently adapt such models when limited supervision is available. In this work, we show that while effective on internet-style datasets, even those remedies under-deliver on classification tasks with images that differ significantly from those commonly found online. To address this issue, we present a new approach called SVL-Adapter that combines the complementary strengths of both vision-language pretraining and self-supervised representation learning. We report an average classification accuracy improvement of 10% in the low-shot setting when compared to existing methods, on a set of challenging visual classification tasks. Further, we present a fully automatic way of selecting an important blending hyperparameter for our model that does not require any held-out labeled validation data. Code for our project is available here: https://github.com/omipan/svl_adapter.Video
Citation
@inproceedings{Pantazis_2022_BMVC,
author = {Omiros Pantazis and Gabriel Brostow and Kate Jones and Oisin Mac Aodha},
title = {SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0580.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by