Contrastive report and multiparametric dual-region magnetic resonance imaging learning for the preoperative prediction of axillary lymph node metastasis in breast cancer.

Background: The accurate prediction of axillary lymph node metastasis (ALNM) is crucial for determining the surgical extent and making treatment decisions for breast cancer patients. However, the method of incorporating clinical diagnostic reports into models to evaluate ALNM is underdeveloped and under validated. This study aimed to investigate the potential of a multimodal deep learning (DL) model that integrates the magnetic resonance imaging (MRI) characteristics of breast tumors and axillary lymph nodes (ALNs) with clinical diagnostic report-derived textual features for accurate ALNM differentiation. Methods: This study retrospectively enrolled 804 breast cancer patients, of whom, 396 were diagnosed with ALNM and 408 with non-ALNM. First, a vision-language model [Breast Axillary Lymph Nodes-Contrastive Language-Image Pre-training, (BALN-CLIP)] with a Vision Transformer (ViT) as the visual encoder and BioClinical Bidirectional Encoder Representations from Transformers (BioClinicalBERT) as the text encoder was constructed. The model was trained using a contrastive learning strategy on the tumor and ALN data. Second, the fine-tuned visual and text encoders were extracted from BALN-CLIP to develop a multimodal model [Multimodal Multiparametric Axillary Lymph Network (MM-AXLNet)] that incorporated orthogonal fusion and cross-attention modules (CAMs), and integrated dynamic contrast-enhanced (DCE) sequences, T2-weighted imaging (T2WI) sequences, and clinical diagnostic reports for ALNM prediction across dual regions. Finally, the performance of our model was compared with that of a single-region model, a model without report features, and radiologist assessments. The accuracy, sensitivity, specificity, receiver operating characteristic (ROC) curves, and area under the curve (AUC) values of the ROC curves of the models were evaluated. Results: In the five-fold cross-validation, the dual-region MM-AXLNet that incorporated report features performed optimally, achieving a mean accuracy of 0.819±0.020, AUC of 0.885±0.015, precision of 0.821±0.016, sensitivity of 0.820±0.007, specificity of 0.846±0.019, and F1-score of 0.819±0.018 in the test set, demonstrating statistically significant superiority compared to the other models. The diagnostic performance of this optimal model was superior to that of the radiologists. Conclusions: MM-AXLNet, which integrates multiparametric dual-region MRI with report information through dual-region contrastive learning, enables the accurate preoperative prediction of ALNM in breast cancer, facilitating clinical treatment decision-making.