🤖 Generated with AI assistance · Does not replace individualized medical advice.
AI in Oncology: High Accuracy, Low Evidence Support? New Study Quantifies LLM Hallucinations in Risk-of-Bias Assessment
AI Transparency Notice: This post was generated by an AI assistant to summarize and contextualize published scientific literature. It is intended for educational and informational purposes only and does not constitute medical advice, diagnosis, or treatment. Always consult a qualified healthcare provider for clinical decisions.
Lead
A new empirical study in JMIR AI evaluates three large language models (LLMs) on risk-of-bias (RoB 2) assessment for 97 randomized controlled trials. While binary accuracy reached 90–98%, evidence support rates were only 60–65%, revealing a critical gap between correct conclusions and documentary grounding PMID: 42748453.
Key Facts
| Metric | Value | Context |
|---|---|---|
| Binary Accuracy | 90%–98% | Directional correctness of RoB judgments across domains PMID: 42748453. |
| Evidence Support Rate | 60%–65% | Proportion of generated claims directly supported by source text PMID: 42748453. |
| Conservative Hallucination | 34%–37% | Claims not found in source documents or contradicted PMID: 42748453. |
| Cohen κ (GPT-5) | Up to 0.81 | Highest agreement with human reviewers among tested models PMID: 42748453. |
Why It Matters / Context
In oncology, evidence-based practice relies heavily on systematic reviews and meta-analyses. The Risk of Bias 2 (RoB 2) tool is the gold standard for assessing the quality of randomized controlled trials (RCTs), directly influencing the weight of evidence in treatment guidelines. As digital health practitioners increasingly deploy LLMs for evidence synthesis, guideline development, and clinical knowledge management, the assumption that “fluent and seemingly correct” outputs are reliable is dangerous.
This study addresses a fundamental limitation in current AI evaluation: most benchmarks measure agreement with human judgments (accuracy) but fail to verify if the AI’s reasoning is actually grounded in the source documents. An LLM can correctly identify that a trial has “high risk of bias” due to training data patterns, without citing the specific methodological flaw (e.g., lack of blinding) present in the trial report. This creates a “black box” problem where the output is clinically plausible but epistemically unsupported. For oncologists, this distinction is vital; a hallucinated justification for downgrading evidence could lead to incorrect therapeutic recommendations if the underlying reasoning is flawed, even if the final binary judgment happens to be correct.
Details
The study evaluated three models: GPT-5, OpenAI o3-mini, and GPT-3.5. The dataset comprised 97 RCTs from a Cochrane systematic review with complete human RoB 2 annotations and accessible full-text reports. No train-validation split was applied; all 97 studies were used for evaluation.
The methodology employed a “Retrieve-Then-Verify” framework. First, relevant text passages were retrieved from trial reports using the Okapi BM25 algorithm. Second, a verification step assigned evidence verdicts (supported, contradicted, not found, or out of scope) with verbatim quotations.
Performance Comparison:
| Model | Binary Accuracy (Range) | Exact Accuracy (Range) | Mean Evidence Support | Strict Hallucination Rate | Quadratic κ (Max) |
|---|---|---|---|---|---|
| GPT-5 | 90%–98% | 42%–71% | 64.3% | 35.7% | 0.81 |
| o3-mini | 90%–98% | 42%–71% | 60%–65% | 34%–37% | < 0.81 |
| GPT-3.5 | 90%–98% | 42%–71% | 60%–65% | 34%–37% | < 0.81 |
Note: The abstract provides aggregate ranges for binary/exact accuracy and support/hallucination rates across models, with specific peak values for GPT-5. Exact accuracy was substantially lower than binary accuracy across all models, indicating frequent disagreements in severity classification despite correct directional classification PMID: 42748453.
A critical finding was that mean top-1 BM25 retrieval scores were similar across models (approximately 30–31) PMID: 42748453. This suggests that differences in hallucination rates were not primarily attributable to differences in retrieval strength. Instead, the models varied in their ability to correctly interpret and ground the retrieved text in their final judgments. GPT-5 demonstrated the strongest overall performance, achieving perfect binary accuracy for overall risk-of-bias conclusions and the highest agreement with human reviewers PMID: 42748453.
Limitations
⚠️ Critical Limitations1. Single Domain Focus: The study exclusively evaluated RoB 2 assessment. It does not generalize to other clinical tasks such as diagnosis, treatment recommendation, or drug interaction analysis, which may have different hallucination profiles.
2. Retrieval Dependency: The verification step relies on BM25 retrieval. If the relevant evidence is not retrieved due to lexical mismatch, the claim may be falsely labeled as “not found” (hallucination), potentially overestimating hallucination rates.
3. Static Dataset: The evaluation used a fixed set of 97 trials from a single Cochrane review. Performance may vary with different trial populations, disease areas, or document structures.
4. Model Versioning: LLMs are rapidly evolving. The specific versions of GPT-5, o3-mini, and GPT-3.5 tested may not reflect future iterations, and results may not be reproducible with updated model weights.
Clinical Takeaway
Clinicians and researchers should not rely solely on the final binary judgment of LLM-generated evidence appraisals; instead, they must implement verification workflows that check whether specific claims are explicitly supported by verbatim quotations from source documents to mitigate the risk of ungrounded “hallucinated” reasoning.
Full Reference
Liang Z, Sheffield C, Butera G, Antani S. Retrieve-Then-Verify for Evaluating Evidence Support and Hallucination in Large Language Model-Generated Medical Information: Empirical Study. JMIR AI. 2026. PMID: 42748453 [DOI: 10.2196/93761]