An Empirical Verification of Wide Networks Theory
Dario Balboni (Scuola Normale Superiore),* Davide Bacciu (Univeristy of Pisa)
The 33rd British Machine Vision Conference
Abstract
In recent years many theories explaining the behavior of Wide Neural Networks have been proposed, focusing on relations of wide networks with Neural Tangent Kernels and on devising a novel optimization theory for overparameterized models. However, despite the efforts, real-world models are still not well-understood. To this aim, we empirically measure crucial quantities for neural networks in the more realistic setting of mildly overparameterized models and in three main areas: conditioning of the optimization process, training speed, and generalization of the obtained models. We analyze the obtained results and highlight discrepancies between existing theories and realistic models, to guide future works on theoretical refinements. Our contribution is exploratory in nature and aims to encourage the development of mixed theoretical-practical approaches, where experiments are quantitative and aimed at measuring fundamental quantities of the existing theories.Video
Citation
@inproceedings{Balboni_2022_BMVC,
author = {Dario Balboni and Davide Bacciu},
title = {An Empirical Verification of Wide Networks Theory},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0517.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by