T4DT: Tensorizing Time for Learning Temporal 3D Visual Data
Mikhail Usvyatsov (ETH Zürich),* Rafael Ballester (IE University), Lina Bashaeva (Skolkovo institute of science and technology ), Konrad Schindler (ETH), Gonzalo Ferrer (Skolkovo Institute of Science and Technology), Ivan Oseledets (Skolkovo Institute of Science and Technology)
The 33rd British Machine Vision Conference
Abstract
Unlike 2D raster images, there is no single dominant representation for 3D visual data processing. Different formats like point clouds, meshes, or implicit functions each have their strengths and weaknesses. Still, grid representations such as signed distance functions have attractive properties also in 3D. In particular, they offer constant-time random access and are eminently suitable for modern machine learning. Unfortunately, the storage size of a grid grows exponentially with its dimension. Hence they often exceed memory limits even at moderate resolution. This work proposes using low-rank tensor formats, including the Tucker, tensor train, and quantics tensor train decompositions, to compress time-varying 3D data. Our method iteratively computes, voxelizes, and compresses each frame's truncated signed distance function and applies tensor rank truncation to condense all frames into a single, compressed tensor that represents the entire 4D scene. We show that low-rank tensor compression is extremely compact to store and query time-varying signed distance functions. It significantly reduces the memory footprint of 4D scenes while remarkably preserving their geometric quality. Unlike existing, iterative learning-based approaches like DeepSDF and NeRF, our method uses a closed-form algorithm with theoretical guarantees.Video
Citation
@inproceedings{Usvyatsov_2022_BMVC,
author = {Mikhail Usvyatsov and Rafael Ballester and Lina Bashaeva and Konrad Schindler and Gonzalo Ferrer and Ivan Oseledets},
title = {T4DT: Tensorizing Time for Learning Temporal 3D Visual Data},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0348.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by