VoRF: Volumetric Relightable Faces
Pramod Rao (Max-Planck-Institut für Informatik),* Mallikarjun B R (Max Planck Institute for Informatics), Gereon Fox (Max Planck Institute for Informatics), Tim Weyrich (Friedrich-Alexander Universität Erlangen-Nürnberg), Bernd Bickel (IST Austria), Hanspeter Pfister (Harvard University), Wojciech Matusik (MIT), Ayush Tewari (MIT), Christian Theobalt (MPI Informatik), Mohamed Elgharib (Max Planck Institute for Informatics)
The 33rd British Machine Vision Conference
Best Paper Award Honourable Mention
Abstract
Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handing both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with separate latent spaces for identity and illumination. The prior model is learnt in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis even when applied to unseen subjects under uncontrolled illuminations.Video
Citation
@inproceedings{Rao_2022_BMVC,
author = {Pramod Rao and Mallikarjun B R and Gereon Fox and Tim Weyrich and Bernd Bickel and Hanspeter Pfister and Wojciech Matusik and Ayush Tewari and Christian Theobalt and Mohamed Elgharib},
title = {VoRF: Volumetric Relightable Faces},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0708.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by