Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment
Yuchen Ma (Heidelberg University), Yanbei Chen (University of Tübingen),* Zeynep Akata (University of Tübingen)
The 33rd British Machine Vision Conference
Abstract
Recent advances have indicated the strengths of self-supervised pre-training for improving representation learning on downstream tasks. Existing works often utilize self-supervised pre-trained models by fine-tunning on downstream tasks. However, fine-tuning does not generalize to the case when one needs to build a customized model architecture different from the self-supervised model. In this work, we formulate a new knowledge distillation framework to transfer the knowledge from self-supervised pre-trained models to any other student network by a novel approach named Embedding Graph Alignment. Specifically, inspired by the spirit of instance discrimination in self-supervised learning, we model the instance-instance relations by a graph formulation in the feature embedding space and distill the self-supervised teacher knowledge to a student network by aligning the teacher graph and the student graph. Our distillation scheme can be flexibly applied to transfer the self-supervised knowledge to enhance representation learning on various student networks. We demonstrate that our model outperforms multiple representative knowledge distillation methods on three benchmark datasets, including CIFAR100, STL10, and TinyImageNet.Video
Citation
@inproceedings{Ma_2022_BMVC,
author = {Yuchen Ma and Yanbei Chen and Zeynep Akata},
title = {Distilling Knowledge from Self-Supervised Teacher by Embedding Graph Alignment},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0973.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by