D-STEP: Dynamic Spatio-Temporal Pruning
Avraham Raviv (SIRC),* Yonatan Dinai (Samsung Israel R&D Center), Igor Drozdov (Samsung), Niv Zehngut (Samsung), Ishay Goldin (Samsung)
The 33rd British Machine Vision Conference
Abstract
Video processing requires analysis of spatial features that are changing over time. By combining spatial and temporal modelling together, a neural network can gain a better understanding of the scene with no increase in computation. Spatio-temporal modeling can also be used to identify redundant and sparse information in both the spatial and the temporal domains. In this work we present Dynamic Spatio-Temporal Pruning, D-STEP, a new, simple, yet efficient method for learning the evolution of spatial mapping between frames. More specifically, we used a cascade of lightweight policy networks to dynamically filter out, per input, regions and channels that do not provide information while also sharing information across time. Guided by the policy networks, the model is able to focus on relevant data and filters, avoiding unnecessary computations. Extensive evaluations on Something-Something-V2, Jester and Mini-Kinetics action recognition datasets demonstrate that the proposed method shows a significantly improved accuracy-compute trade-off over the current state-of-the-art methods. We release our code and trained models at https://github.com/DynamicAR/DSTEP.Video
Citation
@inproceedings{Raviv_2022_BMVC,
author = {Avraham Raviv and Yonatan Dinai and Igor Drozdov and Niv Zehngut and Ishay Goldin},
title = {D-STEP: Dynamic Spatio-Temporal Pruning},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0632.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by