Robustifying the Multi-Scale Representation of Neural Radiance Fields
Nishant Jain (iit roorkee), Suryansh Kumar (ETH Zurich),* Luc Van Gool (ETH Zurich)
The 33rd British Machine Vision Conference
Abstract
Neural Radiance Fields (NeRF) recently emerged as a new paradigm for 3D object representation from multi-view (MV) images. Yet, it cannot handle multi-scale (MS) images and camera pose estimation errors, which generally is the case with multi-view images captured from a day-to-day commodity camera. Although recently proposed Mip-NeRF can handle multi-scale imaging problems with NeRF, it cannot handle camera pose estimation error. On the other hand, the newly proposed BARF can solve the camera pose problem with NeRF but fails if the images are multi-scale in nature. To have a unified framework that can solve these problems, we present a method that can jointly handle multi-scale imaging effects and the camera-pose problems with NeRF-inspired approaches. To this end, the paper put forward an approach based on the fundamental principle of scene rigidity. To reduce unpleasant aliasing artifacts due to multi-scale images in the ray space, we leverage Mip-NeRF multi-scale representation. For joint estimation of robust camera pose, we propose graph-neural network-based multiple motion averaging in the neural volume rendering framework. We demonstrate, with examples, that for a robust neural representation of a 3D object from day-to-day acquired multi-view images, it is crucial to have accurate camera-pose estimates. Without considering robustness measures in the camera pose estimation, modeling for multi-scale aliasing artifacts via conical frustum can be counterproductive. We present extensive experiments on the benchmark datasets to demonstrate that our approach provides better results than the recent NeRF-inspired approaches for such realistic settings.Video
Citation
@inproceedings{Jain_2022_BMVC,
author = {Nishant Jain and Suryansh Kumar and Luc Van Gool},
title = {Robustifying the Multi-Scale Representation of Neural Radiance Fields},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0578.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by