Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment
Mustafa Shukor (Sorbonne University),* Guillaume Couairon (Facebook AI Research), Matthieu Cord (Sorbonne University)
The 33rd British Machine Vision Conference
Abstract
Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder. Although pretrained on four times less data, our ViCHA strategy outperforms other approaches on several downstream tasks such as Image-Text Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The code will be made publicly available.Video
Citation
@inproceedings{Shukor_2022_BMVC,
author = {Mustafa Shukor and Guillaume Couairon and Matthieu Cord},
title = {Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0353.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by