Two-Stream Transformer Architecture for Long Form Video Understanding
Edward Fish (University of Surrey),* Jon Weinbren (University of Surrey), Andrew Gilbert (University of Surrey)
The 33rd British Machine Vision Conference
Abstract
Pure vision transformer architectures are highly effective for short video classification and action recognition tasks. However, due to the quadratic complexity of self attention and lack of inductive bias, transformers are resource intensive and suffer from data inefficiencies. Long form video understanding tasks amplify data and memory efficiency problems in transformers making current approaches unfeasible to implement on data or memory restricted domains. This paper introduces an efficient Spatio-Temporal Attention Network (STAN) which uses a two-stream transformer architecture to model dependencies between static image features and temporal contextual features. Our proposed approach can classify videos up to two minutes in length on a single GPU, is data efficient, and achieves SOTA performance on several long video understanding tasksVideo
Citation
@inproceedings{Fish_2022_BMVC,
author = {Edward Fish and Jon Weinbren and Andrew Gilbert},
title = {Two-Stream Transformer Architecture for Long Form Video Understanding},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0660.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by