Dist2: Distribution-Guided Distillation for Object Detection
Tianchu Guo (Artificial Intelligence Center, DAMO Academy, Alibaba Group, Hangzhou, China),* Pengyu Li (Alibaba Group), Wei Liu (Alibaba), Bin Luo (DAMO Academy, Alibaba Group), biao wang (Alibaba)
The 33rd British Machine Vision Conference
Abstract
Knowledge distillation has been widely used as an effective technique for model compression. Previous knowledge distillation methods in object detection mainly focus on designing loss functions to minimize the feature distances between the teacher and student networks. However, after these losses converge, the imitations are still far from perfect and the distance of the feature map between teacher and student is still large. In this paper, we propose a \textbf{dist}ribution-guided \textbf{dist}illation method named Dist$^2$ for object detection, which concentrates on eliminating the difference of the feature distributions between the teacher and student networks. The proposed Dist$^2$ consists of a distribution imitation (DI) mechanism and a flexible imitation (FI) strategy. Specifically, the distribution imitation (DI) mechanism guarantees the feature distribution from a certain part of the student network is as close as possible to that of the teacher network. Furthermore, the flexible imitation (FI) strategy is employed, which enables the distribution imitation to perform not only on the same part but also cross different parts between the student and teacher networks. Our experimental results on COCO and PASCAL VOC datasets show that the proposed Dist$^2$ outperforms the previous state-of-the-art feature imitation methods by a large margin.Video
Citation
@inproceedings{Guo_2022_BMVC,
author = {Tianchu Guo and Pengyu Li and Wei Liu and Bin Luo and biao wang},
title = {Dist2: Distribution-Guided Distillation for Object Detection},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0323.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by