Anatomy-Aware Self-Supervised Learning for Aligned Multi-Modal Medical Data
Hongyu Hu (Shanghai Jiao Tong University), Tiancheng Lin (Shanghai Jiao Tong University), Yuanfan Guo (SJTU), Chunxiao Li (Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine), Rong Wu (Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine), Yi Xu (Shanghai Jiao Tong University)*
The 33rd British Machine Vision Conference
Abstract
Consistency of anatomical structure naturally exists among medical images from multiple modalities, which provides powerful supervisory signals to self-supervised learning on aligned multi-modal medical images. However, it would lose efficacy due to modality-specific attributes when directly applying current pixel-wise or region-wise contrastive learning methods to pull aligned multi-modal data together in embedding space. To address this issue, we propose a novel anatomy-aware self-supervised learning framework, which represents anatomical structure in each modality using spatial similarity distribution between image patches, to alleviate the ill-effects of modality-specific attributes and obtain a modality-consistent representation of anatomical structure. Significantly, we construct a correlation matrix to represent spatial similarity distribution and design a consistency loss to align the distributions across modalities to maintain anatomical consistency. Furthermore, we integrate it with instance-level discrimination into a unified contrastive framework, where the learned features are augmentation-invariant and modality-consistent. Extensive experiments on two medical datasets for the diagnosis of breast cancer and retinal diseases demonstrate that our proposed method achieves superior performance to current related works.Video
Citation
@inproceedings{Hu_2022_BMVC,
author = {Hongyu Hu and Tiancheng Lin and Yuanfan Guo and Chunxiao Li and Rong Wu and Yi Xu},
title = {Anatomy-Aware Self-Supervised Learning for Aligned Multi-Modal Medical Data},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year = {2022},
url = {https://bmvc2022.mpi-inf.mpg.de/0877.pdf}
}
Copyright © 2022
The British Machine Vision Conference is organised by