SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB Image
Florian Maximilian Langer (Department of Engineering, University of Cambridge),* Gwangbin Bae (University of Cambridge), Ignas Budvytis (Department of Engineering, University of Cambridge), Roberto Cipolla (University of Cambridge)
The 33rd British Machine Vision Conference
Abstract
Estimating 3D shapes and poses of static objects from a single image has important applications for robotics, augmented reality and digital content creation. Often this is done through direct mesh predictions which produces unrealistic, overly tessellated shapes or by formulating shape prediction as a retrieval task followed by CAD model alignment. Directly predicting CAD model poses from 2D image features is difficult and inaccurate. Some works, such as ROCA, regress normalised object coordinates and use those for computing poses. While this can produce more accurate pose estimates, predicting normalised object coordinates is susceptible to systematic failure. Leveraging efficient transformer architectures we demonstrate that a sparse, iterative, render-and-compare approach is more accurate and robust than relying on normalised object coordinates. For this we combine 2D image information including sparse depth and surface normal values which we estimate directly from the image with 3D CAD model information in early fusion. In particular, we reproject points sampled from the CAD model in an initial, random pose and compute their depth and surface normal values. This combined information is the input to a pose prediction network, SPARC-Net, which we train to predict a 9 DoF CAD model pose update. The CAD model is reprojected again and the next pose update is predicted. Our alignment procedure converges after just 3 iterations, improving the state-of-the-art performance on the challenging real-world dataset ScanNet from 25.0% to 31.8% instance alignment accuracy.Video
Citation
@inproceedings{Langer_2022_BMVC,
author = {Florian Maximilian Langer and Gwangbin Bae and Ignas Budvytis and Roberto Cipolla},
title = {SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB Image},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0072.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by