End-to-End Learning of Multi-category 3D Pose and Shape Estimation

Yigit Baran Can (ETH Zurich),* Alexander Liniger (ETH Zurich), Danda Pani Paudel (ETH Z├╝rich), Luc Van Gool (ETH Zurich)
The 33rd British Machine Vision Conference


In this paper, we study the representation of the shape and pose of objects using their keypoints. We propose an end-to-end method that simultaneously detects 2D keypoints from an image and lifts them to 3D. The proposed method learns both 2D detection and 3D lifting only from 2D keypoint annotations. In addition to being end-to-end from images to 3D keypoints, our method also handles objects from multiple categories using a single neural network. We use a Transformer-based architecture to detect the keypoints, as well as to summarize the visual context of the image. This visual context information is used while lifting the keypoints to 3D, to allow context-based reasoning for better performance. Our method can handle occlusions as well as a wide variety of object classes. Our experiments on three benchmarks show that our method performs better than the state-of-the-art. Code https://github.com/ybarancan/end2end3D.



author    = {Yigit Baran Can and Alexander Liniger and Danda Pani Paudel and Luc Van Gool},
title     = {End-to-End Learning of Multi-category 3D Pose and Shape Estimation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0200.pdf}

Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection