Abstract
Image synthesis is one of the key applications of deep learning in neuroimaging, which enables shortening of the scan time and/or improve image quality; therefore, reducing the imaging cost and improving patient experience. Given the multi-modal and large-scale nature of neuroimaging data, the synthesis task is computationally challenging. 2D image synthesis networks do not take advantage of multi-dimensional spatial information and the 3D implementation has dimensionality problem, negatively affecting the network reliability. These limitations hinder the research and clinical applicability of deep learning-based neuroimaging synthesis. In this paper, we proposed a new network that is designed and optimized for the application of multi-modal 3D synthesis of neuroimaging data. The network is based on 3D conditional generative adversarial network (GAN), and employs spectral normalization and feature matching to stabilize the training process and ensure optimization convergence. We also added a self-attention module to model relationships between widely separated voxels. The performance of the network was evaluated by predicting positron emission tomography (PET) images, Fractional anisotropy (FA) and mean diffusivity (MD) maps from multi-modal magnetic resonance images (MRI) of 265 and 497 individuals correspondingly. The proposed network, called self-attention conditional GAN (SC-GAN), significantly outperformed conventional 2D conditional GAN and the 3D implementation, enabling robust 3D deep learning-based neuroimaging synthesis.
Competing Interest Statement
The authors have declared no competing interest.
Footnotes
↵1 Data used in preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (www.adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf.