Towards Unified Multi-Excitation for Unsupervised Video Prediction

Junyan Wang (UNSW Sydney),* Qin Likun (Institute of Microelectronics of Chinese Academy of Science), Peng Zhang (Durham University), Yang Long (Durham University), Bingzhang Hu (Hefei CAS Dihuge Automation CO., LTD), Maurice Pagnucco (UNSW), Shizheng Wang (Chinese Academy of Sciences), Yang Song (University of New South Wales)
The 33rd British Machine Vision Conference


Unsupervised video prediction aims to forecast future frames conditioned on previous frames with the absence of semantic labels. Most existing methods have applied conventional recurrent neural networks, which focus on past memory, while few draw attention to highlight motion and context information. In this work, we propose a Unified Multi-Excitation (UME) block to enhance long-short-term memory, specifically applying an excitation mechanism to learn both channel-wise inter-dependencies and context correlations. Our contributions include: 1) introducing motion and channel excitation to enhance motion-sensitive channels of the features in the short term; and, 2) proposing an adaptive modeling scheme as context excitation inserted between (2+1)D convolution cells. The overall framework employs a multi-excitation block inserted into each ConvLSTM layer to aggregate the motion, channel, and context excitations. The framework achieves state-of-the-art performance on a variety of spatio-temporal predictive datasets including the Moving MNIST, Sea Surface Temperature, Traffic BJ and Human 3.6 datasets. Extensive ablation studies demonstrate the effectiveness of each component of the method. Code and datasets are available at



author    = {Junyan Wang and Qin Likun and Peng Zhang and Yang Long and Bingzhang Hu and Maurice Pagnucco and Shizheng Wang and Yang Song},
title     = {Towards Unified Multi-Excitation for Unsupervised Video Prediction},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {}

Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection