T4DT: Tensorizing Time for Learning Temporal 3D Visual Data


Mikhail Usvyatsov (ETH Zürich),* Rafael Ballester (IE University), Lina Bashaeva (Skolkovo institute of science and technology ), Konrad Schindler (ETH), Gonzalo Ferrer (Skolkovo Institute of Science and Technology), Ivan Oseledets (Skolkovo Institute of Science and Technology)
The 33rd British Machine Vision Conference

Abstract

Unlike 2D raster images, there is no single dominant representation for 3D visual data processing. Different formats like point clouds, meshes, or implicit functions each have their strengths and weaknesses. Still, grid representations such as signed distance functions have attractive properties also in 3D. In particular, they offer constant-time random access and are eminently suitable for modern machine learning. Unfortunately, the storage size of a grid grows exponentially with its dimension. Hence they often exceed memory limits even at moderate resolution. This work proposes using low-rank tensor formats, including the Tucker, tensor train, and quantics tensor train decompositions, to compress time-varying 3D data. Our method iteratively computes, voxelizes, and compresses each frame's truncated signed distance function and applies tensor rank truncation to condense all frames into a single, compressed tensor that represents the entire 4D scene. We show that low-rank tensor compression is extremely compact to store and query time-varying signed distance functions. It significantly reduces the memory footprint of 4D scenes while remarkably preserving their geometric quality. Unlike existing, iterative learning-based approaches like DeepSDF and NeRF, our method uses a closed-form algorithm with theoretical guarantees.

Video



Citation

@inproceedings{Usvyatsov_2022_BMVC,
author    = {Mikhail Usvyatsov and Rafael Ballester and Lina Bashaeva  and Konrad Schindler and Gonzalo Ferrer and Ivan Oseledets},
title     = {T4DT: Tensorizing Time for Learning Temporal 3D Visual Data},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0348.pdf}
}


Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection