Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer
Guglielmo Camporese (University of Padova), Elena Izzo (University of Padua), Lamberto Ballan (University of Padova)*
The 33rd British Machine Vision Conference
Abstract
Vision Transformers (ViTs) enabled the use of transformer architecture on vision tasks showing impressive performances when trained on big datasets. However, on relatively small datasets, ViTs are less accurate given their lack of inductive bias. To this end, we propose a simple but still effective self-supervised learning (SSL) strategy to train ViTs, that without any external annotation, can significantly improve the results. Specifically, we define a set of SSL tasks based on relations of image patches that the model has to solve before or jointly during the downstream training. Differently from ViT, our RelViT model optimizes all the output tokens of the transformer encoder that are related to the image patches, thus exploiting more training signal at each training step. We investigated our proposed methods on several image benchmarks finding that RelViT improves the SSL state-of-the-art methods by a large margin, especially on small datasets.Video
Citation
@inproceedings{Camporese_2022_BMVC,
author = {Guglielmo Camporese and Elena Izzo and Lamberto Ballan},
title = {Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0032.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by