Distilling Representational Similarity using Centered Kernel Alignment (CKA)
Aninda Saha (The University of Queensland),* Alina N Bialkowski (The University of Queensland), Sara Khalifa (CSIRO)
The 33rd British Machine Vision Conference
Abstract
Representation distillation has emerged as an effective knowledge distillation (KD) technique, which involves the transfer of an inter-example similarity matrix. However, existing methods use inadequate normalisation techniques combined with euclidean distance-based loss functions to distill inter-example similarity matrices. Such approaches are not invariant to uniform feature scaling, which is a key property for neural network similarity metrics. Therefore, we propose a novel loss function for representation distillation by adapting Centered Kernel Alignment (CKA), which computes the cosine similarity between the student and teacher's centered and normalised inter-example similarity matrices. We compare our proposed loss function against three popular representation distillation techniques, demonstrating CKA's outperformance on three benchmark image classification datasets. Our results reveal that distilling a centered and normalised distribution of the similarity matrix using the proposed CKA-based loss function is more effective than existing representation distillation techniques.Video
Citation
@inproceedings{Saha_2022_BMVC,
author = {Aninda Saha and Alina N Bialkowski and Sara Khalifa},
title = {Distilling Representational Similarity using Centered Kernel Alignment (CKA)},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0535.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by