Deriving Real-World Evidence from Non-English Electronic Medical Records in Hormone Receptor-Positive Breast Cancer Using Large Language Models.

Background/Objectives: Large language models (LLMs) have been proposed as a means of converting unstructured electronic medical records (EMRs) into structured datasets. However, concerns regarding the reliability of these models in non-English clinical text and their capacity to generate novel insights remain unresolved. We aimed to utilize an LLM to identify a hypothetical "Luminal B poor-prognosis" breast cancer subgroup (LPP) based on progesterone receptor (PR), the Ki-67 proliferation index, and grade characteristics, while concurrently validating the LLM's accuracy. Methods: We retrospectively compiled the EMRs on 7756 female breast cancer patients from five Moscow oncology centers. An LLM with a domain-engineered prompt extracted eight clinicopathological variables (Ki-67, estrogen receptor (ER)/PR Allred status, HER2 status, grade, relapse dates, and multiple primaries). The accuracy of the model was validated in 366 randomly sampled cases against oncologist annotations using Intraclass Correlation Coefficient (ICC) and weighted κ. Following data post-processing, the complete-case cohort (n = 2347) and the HR+/HER2- stage I-III sub-cohort (n = 1419) were analyzed. Survival was estimated with Kaplan-Meier/log-rank and modeled with Cox regression (adjusted for age, stage, and treatment). Ki-67 was modeled continuously; prespecified LPP definitions were compared. Results: LLM-human agreement was high (Ki-67 ICC = 0.882; grade κ = 0.887; ER κ = 0.997; PR κ = 0.975; HER2 κ = 0.935). Date extraction was characterized by a high degree of missing data. In HR+/HER2- stage I-III disease, ER < 5 was non-prognostic; however, PR < 4 and Ki-67 ≥ 40% were indicative of inferior survival (HR 2.25 and 1.85). The most effective LPP definition (PR < 4 and Ki-67 ≥ 40%) identified a subgroup (~5.3%) of patients with markedly poorer outcomes (age, stage, and treatment adjusted HR 2.60, 95% CI 1.53-4.43) compared to the Luminal B (HER2-) subgroup. Conclusions: The developed LLM has demonstrated the ability to reliably structure non-English EMRs and enable discovery of clinically meaningful subgroups. The discovered LPP phenotype defines a small, high-risk subset warranting external validation. Given the retrospective, single-system design of the study, it is imperative to interpret the discovered phenotype features as hypothesis-generating, rather than as definitive evidence.