KPE: Keypoint Pose Encoding for Transformer-based Image Generation
Soon Yau Cheong (University of Surrey),* Armin Mustafa (University of Surrey), Andrew Gilbert (University of Surrey)
The 33rd British Machine Vision Conference
Abstract
Transformers have recently been shown to generate high quality images from text input. However, the existing method of pose conditioning using skeleton image tokens is computationally inefficient and generate low quality images. Therefore we propose a new method; Keypoint Pose Encoding (KPE); KPE is 10× more memory efficient and over 73% faster at generating high quality images from text input conditioned on the pose. The pose constraint improves the image quality and reduces errors on body extremities such as arms and legs. The additional benefits include invariance to changes in the target image domain and image resolution, making it easily scalable to higher resolution images. We demonstrate the versatility of KPE by generating photorealistic multiperson images derived from the DeepFashion dataset. We also introduce a evaluation method People Count Error (PCE) that is effective in detecting error in generated human images.Video
Citation
@inproceedings{Cheong_2022_BMVC,
author = {Soon Yau Cheong and Armin Mustafa and Andrew Gilbert},
title = {KPE: Keypoint Pose Encoding for Transformer-based Image Generation},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0163.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by