Re-examining Distillation for Continual Object Detection

Eli Verwimp (KU Leuven),* Kuo Yang (Huawei Noah's Ark Lab), Sarah Parisot (Huawei Noah's Ark Lab ), Lanqing Hong (Huawei Noah's Ark Lab), Steven McDonagh (Huawei Noah's Ark Lab), Eduardo Pérez Pellitero (Huawei Noah's Ark Lab), Matthias De Lange (KU Leuven), Tinne Tuytelaars (KU Leuven)
The 33rd British Machine Vision Conference


Training models continually to detect and classify objects, from new classes and new domains, remains an open problem. In this work, we conduct an analysis of why and how object detection models forget catastrophically. We focus on distillation-based approaches in two-stage networks; the most-common strategy employed in contemporary continual object detection work. Distillation aims to transfer the knowledge of a model trained on previous tasks -the teacher- to a new model -the student- while it learns the new task. We show forgetting happens mostly in the classification head, where wrong, yet overly confident teacher predictions prevent student models from effective learning. Our analysis provides insights in the effects of using distillation techniques, and serves as a foundation that allows us to propose improvements for existing techniques by detecting incorrect teacher predictions, based on current ground-truth labels, and by employing an adaptive Huber loss as opposed to the mean squared error for the distillation loss in the classification heads.



author    = {Eli Verwimp and Kuo Yang and Sarah Parisot and Lanqing Hong and Steven McDonagh and Eduardo Pérez Pellitero and Matthias De Lange and Tinne Tuytelaars},
title     = {Re-examining Distillation for Continual Object Detection},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {}

Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection