HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances
Yue Jiang (Max Planck Institute for Informatics), Marc Habermann (Max Planck Institute for Informatics), Vladislav Golyanik (MPI for Informatics),* Christian Theobalt (MPI Informatik)
The 33rd British Machine Vision Conference
Abstract
Monocular 3D human performance capture is indispensable for many applications in computer graphics and vision for enabling immersive experiences. However, detailed capture of humans requires tracking of multiple aspects, including the skeletal pose, the dynamic surface, which includes clothing, hand gestures as well as facial expressions. No existing monocular method allows joint tracking of all these components. To this end, we propose HiFECap, a new neural human performance capture approach, which simultaneously captures human pose, clothing, facial expression, and hands just from a single RGB video. We demonstrate that our proposed network architecture, the carefully designed training strategy, and the tight integration of parametric face and hand models to a template mesh enable the capture of all these individual aspects. Importantly, our method also captures high-frequency details, such as deforming wrinkles on the clothes, better than the previous works. Furthermore, we show that HiFECap outperforms the state-of-the-art human performance capture approaches qualitatively and quantitatively while for the first time capturing all aspects of the human.Video
Citation
@inproceedings{Jiang_2022_BMVC,
author = {Yue Jiang and Marc Habermann and Vladislav Golyanik and Christian Theobalt},
title = {HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0826.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by