MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks
Angel Villar-Corrales (University of Bonn),* Ani Karapetyan (University of Bonn), Andreas Boltres (SAP), Sven Behnke (University of Bonn)
The 33rd British Machine Vision Conference
Abstract
Autonomous systems not only need to understand their current environment, but should also be able to predict future actions conditioned on past states, for instance based on captured camera frames. However, existing models mainly focus on forecasting future video frames for short time-horizons, hence being of limited use for long-term action planning. We propose Multi-Scale Hierarchical Prediction (MSPred), a novel video prediction model able to simultaneously forecast future possible outcomes of different levels of granularity at different spatio-temporal scales. By combining spatial and temporal downsampling, MSPred efficiently predicts abstract representations such as human poses or locations over long time horizons, while still maintaining a competitive performance for video frame prediction. In our experiments, we demonstrate that MSPred accurately predicts future video frames as well as high-level representations (e.g. keypoints or semantics) on bin-picking and action recognition datasets, while consistently outperforming popular approaches for future frame prediction. Furthermore, we ablate different modules and design choices in MSPred, experimentally validating that combining features of different spatial and temporal granularity leads to a superior performance. Code and models to reproduce our experiments can be found in https://github.com/AIS-Bonn/MSPred.Video
Citation
@inproceedings{Villar-Corrales_2022_BMVC,
author = {Angel Villar-Corrales and Ani Karapetyan and Andreas Boltres and Sven Behnke},
title = {MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0034.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by