STDDAE: Identifying spatial domains in spatial transcriptomics by dual denoising autoencoder with attention mechanism

Spatial transcriptomics provides a novel perspective for comprehending the intricate relationship between tissue structure and function, as well as for discovering new cell types and subtypes. However, it remains a significant challenge to accurately identify spatial domains with similar gene expression, which requires efficient combination of gene expression data, histology image information, and spatial location. To address this challenge, a novel dual denoising autoencoder with attention mechanism (STDDAE) is proposed. STDDAE integrates gene expression data, histology image information and spatial location, and the decoder consists of a master decoder and a follower decoder, which are jointly optimized to generate low-dimensional latent embeddings for precise spatial domain identification. The performance of STDDAE was evaluated across four datasets with varying resolutions and platforms. The experimental findings validated that STDDAE outperformed other cutting-edg methods in spatial domain identification, trajectory inference, and data denoising. Additionally, STDDAE successfully detected differentially expressed genes within identified spatial domains, which may be valuable in disease diagnosis, prognostic assessment, and treatment selection.