DiffSketching: Sketch Control Image Synthesis with Diffusion Models
Qiang Wang (Beijing university of posts and telecommunications),* Di Kong (Beijing University of Posts and Telecommunications), Fengyin Lin (Beijing University of Posts and Telecommunications), Yonggang Qi (Beijing University of Posts and Telecommunications)
The 33rd British Machine Vision Conference
Abstract
Creative sketch is a universal way of visual expression, but translating images from an abstract sketch is very challenging. Traditionally, creating a deep learning model for sketch-to-image synthesis needs to overcome the distorted input sketch without visual details, and requires to collect large-scale sketch-image datasets. We first study this task by using diffusion models. Our model matches sketches through the cross domain constraints, and uses the classifier to guide the image synthesis more accurately. Extensive experiments confirmed that our method can not only be faithful to user's input sketches, but also maintain the diversity and imagination of synthetic results. Our model can beat GAN-based method in terms of generation quality and human evaluation, and does not rely on massive sketch-image datasets. Additionally, we demonstrate applications of our method in image editing and interpolation.Video
Citation
@inproceedings{Wang_2022_BMVC,
author = {Qiang Wang and Di Kong and Fengyin Lin and Yonggang Qi},
title = {DiffSketching: Sketch Control Image Synthesis with Diffusion Models},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0067.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by