Motion and Context-Aware Audio-Visual Conditioned Video Prediction
Yating Xu (National University of Singapore),* Conghui Hu (National University of Singapore), Gim Hee Lee (National University of Singapore)
The 34th British Machine Vision Conference
Abstract
The existing state-of-the-art method for audio-visual conditioned video prediction uses the latent codes of the audio-visual frames from a multimodal stochastic network and a frame encoder to predict the next visual frame. However, a direct inference of per-pixel intensity for the next visual frame is extremely challenging because of the high-dimensional image space. To this end, we decouple the audio-visual conditioned video prediction into motion and appearance modeling. The multimodal motion estimation predicts future optical flow based on the audio-motion correlation. The visual branch recalls from the motion memory built from the audio features to enable better long-term prediction. We further propose context-aware refinement to address the diminishing of the global appearance context in the long-term continuous warping. The global appearance context is extracted by the context encoder and manipulated by motion-conditioned affine transformation before fusion with features of warped frames. Experimental results show that our method achieves competitive results on existing benchmarks.Citation
@inproceedings{Xu_2023_BMVC,
author = {Yating Xu and Conghui Hu and Gim Hee Lee},
title = {Motion and Context-Aware Audio-Visual Conditioned Video Prediction},
booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},
publisher = {BMVA},
year = {2023},
url = {https://bmvc2022.mpi-inf.mpg.de/BMVC2023/0025.pdf}
}
Copyright © 2023
The British Machine Vision Conference is organised by