CNeRV: Content-adaptive Neural Representation for Visual Data
Hao Chen (University of Maryland),* Matthew A Gwilliam (University of Maryland), Bo He (University of Maryland), Ser-Nam Lim (Meta AI), Abhinav Shrivastava (University of Maryland)
The 33rd British Machine Vision Conference
Abstract
Compression and reconstruction of visual data have been widely studied in the computer vision community, even before the popularization of deep learning. More recently, some have used deep learning to improve or refine existing pipelines, while others have proposed end-to-end approaches, including autoencoders and implicit neural representations, such as SIREN and NeRV. In this work, we propose Neural Visual Representation with Content-adaptive Embedding (CNeRV), which combines the generalizability of autoencoders with the simplicity and compactness of implicit representation. We introduce a novel content-adaptive embedding that is unified, concise, and internally (within-video) generalizable, that compliments a powerful decoder with a single-layer encoder. We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images). To achieve similar reconstruction quality on unseen images, NeRV needs 120x more time to overfit per-frame due to its lack of internal generalization. With the same latent code length and similar model size, CNeRV outperforms autoencoders on reconstruction of both seen and unseen images. We also show promising results for visual data compression. We provide code in the supplementary material.Video
Citation
@inproceedings{Chen_2022_BMVC,
author = {Hao Chen and Matthew A Gwilliam and Bo He and Ser-Nam Lim and Abhinav Shrivastava},
title = {CNeRV: Content-adaptive Neural Representation for Visual Data},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0510.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by