Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition


Youngjoon Jang (KAIST),* Youngtaek Oh (KAIST), Jae Won Cho (KAIST), Dong-Jin Kim (Hanyang University), Joon Son Chung (KAIST), In So Kweon (KAIST)
The 33rd British Machine Vision Conference

Abstract

Most existing Continuous Sign Language Recognition (CSLR) benchmarks have fixed backgrounds and are filmed in studios with a static monochromatic background. However, signing is not limited only to studios in the real world. In order to analyze the robustness of CSLR models under background shifts, we first evaluate existing state-of-the-art CSLR models on diverse backgrounds. To synthesize the sign videos with a variety of backgrounds, we propose an algorithm to automatically generate a benchmark dataset utilizing existing CSLR benchmarks. Our newly constructed benchmark dataset simulates diverse scenes to simulate a real-world environment. Interestingly, we observe even the most recent CSLR method cannot recognize glosses well on our new dataset with changed backgrounds. In this regard, we also propose a simple yet effective training scheme including (1) background randomization and (2) feature disentanglement for CSLR models. The experimental results on our dataset demonstrate that our method generalizes well to other unseen background data with minimal additional training images.

Video



Citation

@inproceedings{Jang_2022_BMVC,
author    = {Youngjoon Jang and Youngtaek Oh and Jae Won Cho and Dong-Jin Kim and Joon Son Chung and In So Kweon},
title     = {Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0322.pdf}
}


Copyright © 2022 The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised by The British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).

Imprint | Data Protection