Neighbor Regularized Bayesian Optimization for Hyperparameter Optimization
Lei Cui (Tsinghua University), Yangguang Li (SenseTime Group Limited),* Xin Lu (SenseTime Research), Dong An (Institute of Automation, Chinese Academy of Sciences), Fenggang Liu (SenseTime)
The 33rd British Machine Vision Conference
Abstract
Bayesian Optimization (BO) is a common solution to search optimal hyperparameters based on sample observations of a machine learning model. Existing BO algorithms could converge slowly even collapse when the potential observation noise misdirects the optimization. In this paper, we propose a novel BO algorithm called Neighbor Regularized Bayesian Optimization (NRBO) to solve the problem. We first propose a neighbor-based regularization to smooth each sample observation, which could reduce the observation noise efficiently without any extra training cost. Since the neighbor regularization highly depends on the sample density of a neighbor area, we further design a density-based acquisition function to adjust the acquisition reward and obtain more stable statistics. In addition, we design a adjustment mechanism to ensure the framework maintains a reasonable regularization strength and density reward conditioned on remaining computation resources. We conduct experiments on the bayesmark benchmark and important computer vision benchmarks such as ImageNet and COCO. Extensive experiments demonstrate the effectiveness of NRBO and it consistently outperforms other state-of-the-art methods.Video
Citation
@inproceedings{Cui_2022_BMVC,
author = {Lei Cui and Yangguang Li and Xin Lu and Dong An and Fenggang Liu},
title = {Neighbor Regularized Bayesian Optimization for Hyperparameter Optimization},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0479.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by