End-to-End Learning of Multi-category 3D Pose and Shape Estimation
Yigit Baran Can (ETH Zurich),* Alexander Liniger (ETH Zurich), Danda Pani Paudel (ETH Zürich), Luc Van Gool (ETH Zurich)
The 33rd British Machine Vision Conference
Abstract
In this paper, we study the representation of the shape and pose of objects using their keypoints. We propose an end-to-end method that simultaneously detects 2D keypoints from an image and lifts them to 3D. The proposed method learns both 2D detection and 3D lifting only from 2D keypoint annotations. In addition to being end-to-end from images to 3D keypoints, our method also handles objects from multiple categories using a single neural network. We use a Transformer-based architecture to detect the keypoints, as well as to summarize the visual context of the image. This visual context information is used while lifting the keypoints to 3D, to allow context-based reasoning for better performance. Our method can handle occlusions as well as a wide variety of object classes. Our experiments on three benchmarks show that our method performs better than the state-of-the-art. Code https://github.com/ybarancan/end2end3D.Video
Citation
@inproceedings{Can_2022_BMVC,
author = {Yigit Baran Can and Alexander Liniger and Danda Pani Paudel and Luc Van Gool},
title = {End-to-End Learning of Multi-category 3D Pose and Shape Estimation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0200.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by