Towards Unified Multi-Excitation for Unsupervised Video Prediction
Junyan Wang (UNSW Sydney),* Qin Likun (Institute of Microelectronics of Chinese Academy of Science), Peng Zhang (Durham University), Yang Long (Durham University), Bingzhang Hu (Hefei CAS Dihuge Automation CO., LTD), Maurice Pagnucco (UNSW), Shizheng Wang (Chinese Academy of Sciences), Yang Song (University of New South Wales)
The 33rd British Machine Vision Conference
Abstract
Unsupervised video prediction aims to forecast future frames conditioned on previous frames with the absence of semantic labels. Most existing methods have applied conventional recurrent neural networks, which focus on past memory, while few draw attention to highlight motion and context information. In this work, we propose a Unified Multi-Excitation (UME) block to enhance long-short-term memory, specifically applying an excitation mechanism to learn both channel-wise inter-dependencies and context correlations. Our contributions include: 1) introducing motion and channel excitation to enhance motion-sensitive channels of the features in the short term; and, 2) proposing an adaptive modeling scheme as context excitation inserted between (2+1)D convolution cells. The overall framework employs a multi-excitation block inserted into each ConvLSTM layer to aggregate the motion, channel, and context excitations. The framework achieves state-of-the-art performance on a variety of spatio-temporal predictive datasets including the Moving MNIST, Sea Surface Temperature, Traffic BJ and Human 3.6 datasets. Extensive ablation studies demonstrate the effectiveness of each component of the method. Code and datasets are available at https://github.com/captaincj/UMENet.git.Video
Citation
@inproceedings{Wang_2022_BMVC,
author = {Junyan Wang and Qin Likun and Peng Zhang and Yang Long and Bingzhang Hu and Maurice Pagnucco and Shizheng Wang and Yang Song},
title = {Towards Unified Multi-Excitation for Unsupervised Video Prediction},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0587.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by