Knowledge Distillation Layer that Lets the Student Decide
Ada Gorgun (Middle East Technical University),* Yeti Z. Gurbuz (Tecnische Universitat Berlin), Aydin Alatan (Middle East Technical University, Turkey)
The 34th British Machine Vision Conference
Abstract
Typical technique in knowledge distillation (KD) is regularizing the learning of a limited capacity model (student) by pushing its responses to match a powerful model's (teacher). Albeit useful especially in the penultimate layer and beyond, its action on student's feature transform is rather implicit, limiting its practice in the intermediate layers. To explicitly embed the teacher's knowledge in feature transform, we propose a learnable KD layer for the student which improves KD with two distinct abilities: i) learning how to leverage the teacher's knowledge, enabling to discard nuisance information, and ii) feeding forward the transferred knowledge deeper. Thus, the student enjoys the teacher's knowledge during the inference besides training. Formally, we repurpose 1x1-BN-ReLU-1x1 convolution block to assign a semantic vector to each local region according to the template (supervised by the teacher) that the corresponding region of the student matches. To facilitate template learning in the intermediate layers, we propose a novel form of supervision based on the teacher's decisions. Through rigorous experimentation, we demonstrate the effectiveness of our approach on 3 popular classification benchmarks. Code is available at: https://github.com/adagorgun/letKD-frameworkCitation
@inproceedings{Gorgun_2023_BMVC,
author = {Ada Gorgun and Yeti Z. Gurbuz and Aydin Alatan},
title = {Knowledge Distillation Layer that Lets the Student Decide},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://bmvc2022.mpi-inf.mpg.de/BMVC2023/0123.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by