🤖 Generated with AI assistance · Does not replace individualized medical advice.
LLMs Show High Concordance with Psycho-Oncologists in Korean Distress Screening
AI Transparency Notice: This post was generated by an AI assistant to summarize and contextualize published medical research. It is for informational purposes only and does not constitute medical advice, diagnosis, or treatment recommendations. Always consult a qualified healthcare provider for personal medical concerns.
Lead
In a 2026 exploratory study, large language models (LLMs) demonstrated high agreement with board-certified psycho-oncologists when rating symptom severity in authentic Korean psycho-oncology interviews. While GPT-4o showed the highest concordance, all models exhibited distinct error profiles, highlighting the potential for scalable mental health screening in oncology.
Key Facts
| Metric | Value |
|---|---|
| Study Design | Exploratory, clinician-benchmarked evaluation |
| Sample Size | 101 adults (87.1% female; 69.3% breast cancer) |
| Models Evaluated | GPT-4o, Claude 3.5 Sonnet, Gemini 2.5 Flash |
| Primary Outcome | Item-level symptom rating concordance (ICC 0.816–0.872) |
Why It Matters / Context
Psychiatric distress is prevalent among patients with cancer, yet systematic, interview-based screening remains difficult to scale in routine clinical care due to time constraints and workforce limitations PMID: 42715407. While large language models have shown promise as scalable tools for mental health assessment, most existing evidence is derived from clinician-authored records, translated text, or proxy data rather than authentic, real-time patient interviews PMID: 42715407.
This study addresses a critical gap by evaluating the performance characteristics, error patterns, and explanatory behaviors of contemporary LLMs on authentic, non-English psychiatric interviews PMID: 42715407. The focus on Korean psycho-oncology interviews is significant, as it tests the models’ ability to handle nuanced cultural and linguistic contexts that are often underrepresented in training data. By benchmarking against real-time, time-stamped ratings from board-certified psycho-oncologists, the study provides a rigorous ground truth for assessing LLM reliability in a high-stakes clinical domain PMID: 42715407.
Details
The study evaluated three contemporary LLMs: GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5 Flash. These models generated item scores and brief rationales using identical Korean zero-shot rubrics based on 39 specific symptom items PMID: 42715407. The evaluation focused on concordance, directional bias, and clinician-adjudicated error characteristics.
Performance Metrics:
Across 3,931 item-level ratings, all models showed good agreement with clinicians, with intraclass correlation coefficients (ICC) ranging from 0.816 to 0.872 PMID: 42715407. GPT-4o demonstrated the highest agreement among the three models PMID: 42715407. However, when comparing patient-level total symptom burden, Claude 3.5 and Gemini 2.5 yielded significantly higher scores than GPT-4o, indicating a directional bias toward overestimation in these specific models PMID: 42715407.
Error Analysis and Meta-Evaluation:
Binary mismatches were clinically adjudicated as either ambiguous or definite overestimation/underestimation using an 8-etiology taxonomy PMID: 42715407. To understand why errors occurred, the model-generated rationales were meta-evaluated using newer models (GPT-5.4, Claude Sonnet 4.6, and Gemini Pro 3.1) across four dimensions:
1. Citation: Use of quoted supporting statements from the transcript.
2. Structure: Logical organization of the rationale.
3. Mapping: Consistency between the rationale and the assigned item rating.
4. Expansion: Degree of interpretive elaboration beyond the explicit transcript content.
Associations between these meta-evaluation results and absolute error were examined using cross-classified mixed-effects models PMID: 42715407. This multi-layered approach allows for a deeper understanding of how LLMs process complex clinical narratives, moving beyond simple accuracy metrics to examine the reasoning quality.
Limitations
Critical Limitations:
* Sample Demographics: The cohort was predominantly female (87.1%) and composed largely of breast cancer patients (69.3%), limiting generalizability to other cancer types and male patients PMID: 42715407.
* Exploratory Nature: The study is explicitly exploratory and not a definitive validation trial PMID: 42715407.
* Language Specificity: Findings are specific to Korean-language interviews; performance in other non-English languages remains uncharacterized PMID: 42715407.
* Model Versioning: LLMs are rapidly evolving; the specific versions tested (GPT-4o, Claude 3.5, Gemini 2.5) may not reflect future iterations PMID: 42715407.
* Lack of Independent Cohort: Validation in larger, more diverse, and independent cohorts is required before clinical implementation PMID: 42715407.
Clinical Takeaway
Clinicians should view LLMs as promising but unvalidated adjuncts for psycho-oncology screening, requiring rigorous local validation and human oversight before integration into routine distress assessment workflows.
Full Reference
Kim J, Son KL, Yeom CW, Kim WH, Lee SH, Shin JS. Large Language Models for Distress Rating in Korean Psycho-Oncology Interviews: Exploratory Clinician-Benchmarked Evaluation Study. Journal of Medical Internet Research. 2026. PMID: 42715407 [DOI: 10.2196/91405]