UV-Based 3D Hand-Object Reconstruction with Grasp Optimization

Ziwei Yu (National University of Singapore),* Linlin Yang ( University of Bonn), You Xie (TUM), Ping Chen (Bean Tech), Angela Yao (National University of Singapore)
The 33rd British Machine Vision Conference


We propose a novel framework for 3D hand shape reconstruction and hand-object grasp optimization from a single RGB image. How the hand-object contact regions are represented is critical for accurate reconstructions. Instead of approximating the contact regions with sparse points, as in previous works, we propose a dense representation in the form of a UV coordinate map. Furthermore, we introduce inference-time optimization to fine-tune the grasp and improve interactions between the hand and the object. Our pipeline increases hand shape reconstruction accuracy and produces a vibrant hand texture. Experiments on datasets such as Ho3D, FreiHand, and DexYCB reveal that our proposed method outperforms the state-of-the-art.



author    = {Ziwei Yu and Linlin Yang and You Xie and Ping Chen and Angela Yao},
title     = {UV-Based 3D Hand-Object Reconstruction with Grasp Optimization},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0111.pdf}

Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection