HiSTaR: identifying spatial domains with hierarchical spatial transcriptomics variational autoencoder.

BACKGROUND: The development of spatial transcriptomics (ST) has enabled biologists to measure transcriptome data on an entire tissue and retain spatial information. It gives us the opportunity to fully understand the tissue microenvironment and identify spatial domains. Deep learning techniques provide an effective way to capture the latent representation of spatial transcriptomics data. METHODS: In this paper, we propose a Hierarchical Spatial Transcriptomics variational autoencoder (HiSTaR) that employs multiple HiSTaR blocks to capture multi-level latent features from spots. These features were subsequently utilized for downstream analyses, including spatial domain identification and batch effect correction. RESULTS: HiSTaR tends to perform well in identifying spatial domains across multiple datasets from diverse platforms, consistently showing superior results compared to existing methods. HiSTaR also supports trajectory analysis and differential gene expression analysis, contributing to its validation. Furthermore, HiSTaR facilitates seamless integration of multiple tissue slices, effectively addressing batch effects without external tools-a key advantage for large-scale spatial transcriptomics studies. CONCLUSION: HiSTaR offers an effective computational tool for spatial transcriptomics research. By capturing hierarchical latent features, it improves spatial domain identification. This framework holds potential to advance understanding of tissue heterogeneity and spatially resolved gene expression patterns.