Weakly-supervised Fingerspelling Recognition in British Sign Language Videos
Prajwal K R (VGG, Oxford),* Hannah Bull (LISN (CNRS)), Liliane Momeni (University of Oxford), Samuel Albanie (University of Cambridge), Gul Varol (Ecole des Ponts ParisTech), Andrew Zisserman (University of Oxford)
The 33rd British Machine Vision Conference
Abstract
The goal of this work is to detect and recognize sequences of letters signed using fingerspelling in British Sign Language (BSL). Previous fingerspelling recognition methods have not focused on BSL, which has a very different signing alphabet (e.g., two-handed instead of one-handed) to American Sign Language (ASL). They also use manual annotations for training. In contrast to previous methods, our method only uses weak annotations from subtitles for training. We localize potential instances of fingerspelling using a simple feature similarity method, then automatically annotate these instances by querying subtitle words and searching for corresponding mouthing cues from the signer. We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities. We employ a multi-stage training approach, where we make use of an initial version of our trained model to extend and enhance our training data before re-training again to achieve better performance. Through extensive evaluations, we verify our method for automatic annotation and our model architecture. Moreover, we provide a human expert annotated test set of 5K video clips for evaluating BSL fingerspelling recognition methods to support sign language research.Video
Citation
@inproceedings{R_2022_BMVC,
author = {Prajwal K R and Hannah Bull and Liliane Momeni and Samuel Albanie and Gul Varol and Andrew Zisserman},
title = {Weakly-supervised Fingerspelling Recognition in British Sign Language Videos},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0609.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by