Masked Vision-Language Transformers for Scene Text Recognition
Jie Wu (Westone Information Industry INC.),* Ying Peng (Westone Information Industry INC.), Shengming Zhang (Westone Information Industry INC), Weigang Qi (Westone Information Industry INC.), zhang jian (Westone Information Industry INC)
The 33rd British Machine Vision Conference
Abstract
Scene text recognition (STR) enables computers to recognize and read the text in various real-world scenes. Recent STR models benefit from taking linguistic information in addition to visual cues into consideration. We propose a novel Masked Vision-Language Transformers (MVLT) to capture both the explicit and the implicit linguistic information. Our encoder is a Vision Transformer, and our decoder is a multi-modal Transformer. MVLT is trained in two stages: in the first stage, we design a STR-tailored pretraining method based on a masking strategy; in the second stage, we fine-tune our model and adopt an iterative correction method to improve the performance. MVLT attains superior results compared to state-of-the-art STR models on several benchmarks. Our code and model are available at https://github.com/onealwj/MVLT.Video
Citation
@inproceedings{Wu_2022_BMVC,
author = {Jie Wu and Ying Peng and Shengming Zhang and Weigang Qi and zhang jian},
title = {Masked Vision-Language Transformers for Scene Text Recognition},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0555.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by