Knowledge Diversification in Ensembles of Identical Neural Networks
Bishshoy Das (IIT Delhi),* Sumantra Dutta Roy (Indian Institute of Technology Delhi)
The 33rd British Machine Vision Conference
Abstract
Diversity in representations is key to enhancing the performance of neural networks in an ensemble. In standard neural network ensemble techniques, two or more networks are trained independently and their logits or predictions are combined using a voting procedure or linear combination strategy. This procedure does not incorporate the exchange of information between the base networks of the ensemble. We propose a method for improving learnt representations in an ensemble by employing feature exchange between base models as a part of the training objective. Feature Difference Loss or FDL compels networks in an ensemble to learn diverse features in a Euclidean sense, thereby directly optimizing model diversity. Experiments with ensembles of two, three and four networks show significant performance boosts over competing ensemble techniques. The gains are larger for datasets with fewer examples per class, such as MNIST, CIFAR-10 and CIFAR-100. Positive gains can also be observed in large datasets such as ImageNet. The gains also generalize across several architectures from simple ConvNets to deeper networks such as VGG and ResNets.Video
Citation
@inproceedings{Das_2022_BMVC,
author = {Bishshoy Das and Sumantra Dutta Roy},
title = {Knowledge Diversification in Ensembles of Identical Neural Networks},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0798.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by