Distilling Representational Similarity using Centered Kernel Alignment (CKA)

Aninda Saha (The University of Queensland),* Alina N Bialkowski (The University of Queensland), Sara Khalifa (CSIRO)
The 33rd British Machine Vision Conference


Representation distillation has emerged as an effective knowledge distillation (KD) technique, which involves the transfer of an inter-example similarity matrix. However, existing methods use inadequate normalisation techniques combined with euclidean distance-based loss functions to distill inter-example similarity matrices. Such approaches are not invariant to uniform feature scaling, which is a key property for neural network similarity metrics. Therefore, we propose a novel loss function for representation distillation by adapting Centered Kernel Alignment (CKA), which computes the cosine similarity between the student and teacher's centered and normalised inter-example similarity matrices. We compare our proposed loss function against three popular representation distillation techniques, demonstrating CKA's outperformance on three benchmark image classification datasets. Our results reveal that distilling a centered and normalised distribution of the similarity matrix using the proposed CKA-based loss function is more effective than existing representation distillation techniques.



author    = {Aninda Saha and Alina N Bialkowski and Sara Khalifa},
title     = {Distilling Representational Similarity using Centered Kernel Alignment (CKA)},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0535.pdf}

Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection