Unconditional Image-Text Pair Generation with Multimodal Cross Quantizer
Hyungyung Lee (KAIST GSAI),* Sungjin Park (Korea Advanced Institute of Science and Technology), Joonseok Lee (Google Research & Seoul National University), Edward Choi (KAIST)
The 33rd British Machine Vision Conference
Abstract
Although deep generative models have gained a lot of attention, most of the existing works are designed for unimodal generation. In this paper, we explore a new method for unconditional image-text pair generation. We design Multimodal Cross-Quantization VAE (MXQ-VAE), a novel vector quantizer for joint image-text representations, with which we discover that a joint image-text representation space is effective for semantically consistent image-text pair generation. To learn a multimodal semantic correlation in a quantized space, we combine VQ-VAE with a Transformer encoder and apply an input masking strategy. Specifically, MXQ-VAE accepts a masked image-text pair as input and learns a quantized joint representation space, so that the input can be converted to a unified code sequence, then we perform unconditional image-text pair generation with the code sequence. Extensive experiments show the correlation between the quantized joint space and the multimodal generation capability on synthetic and real-world datasets. In addition, we demonstrate the superiority of our approach in these two aspects over several baselines.Video
Citation
@inproceedings{Lee_2022_BMVC,
author = {Hyungyung Lee and Sungjin Park and Joonseok Lee and Edward Choi},
title = {Unconditional Image-Text Pair Generation with Multimodal Cross Quantizer},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0533.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by