Knowledge Diversification in Ensembles of Identical Neural Networks


Bishshoy Das (IIT Delhi),* Sumantra Dutta Roy (Indian Institute of Technology Delhi)
The 33rd British Machine Vision Conference

Abstract

Diversity in representations is key to enhancing the performance of neural networks in an ensemble. In standard neural network ensemble techniques, two or more networks are trained independently and their logits or predictions are combined using a voting procedure or linear combination strategy. This procedure does not incorporate the exchange of information between the base networks of the ensemble. We propose a method for improving learnt representations in an ensemble by employing feature exchange between base models as a part of the training objective. Feature Difference Loss or FDL compels networks in an ensemble to learn diverse features in a Euclidean sense, thereby directly optimizing model diversity. Experiments with ensembles of two, three and four networks show significant performance boosts over competing ensemble techniques. The gains are larger for datasets with fewer examples per class, such as MNIST, CIFAR-10 and CIFAR-100. Positive gains can also be observed in large datasets such as ImageNet. The gains also generalize across several architectures from simple ConvNets to deeper networks such as VGG and ResNets.

Video



Citation

@inproceedings{Das_2022_BMVC,
author    = {Bishshoy Das and Sumantra Dutta Roy},
title     = {Knowledge Diversification in Ensembles of Identical Neural Networks},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0798.pdf}
}


Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection