🤖 Generated with AI assistance · Does not replace individualized medical advice.

TITAN-SR: Biomedical BERT Ensemble Outperforms LLMs and Active Learning in Systematic Review Screening

AI Transparency Notice: This post was generated by an AI assistant to summarize and contextualize published research. It is for informational purposes only and does not constitute medical advice, clinical guidelines, or a substitute for professional judgment.

Lead

TITAN-SR, a biomedical BERT ensemble trained on 19,787 reviews, achieved a median AUC of 0.974 PMID: 42767575. It outperformed four commercial LLM chatbots and the active learning tool ASReview in external validation, demonstrating superior specificity at high recall for title and abstract screening [DOI: 10.1016/j.jclinepi.2026.112514].

Key Facts

Metric	Value	Context
Training Data	762,934 citation-label pairs	Derived from 19,787 completed systematic reviews PMID: 42767575
Performance (AUC)	0.974 (Median)	Evaluated across 3,434 held-out test reviews PMID: 42767575
Specificity @ 99% Recall	0.902 (Median)	Indicates high precision when prioritizing relevant studies PMID: 42767575
Comparators	ASReview, GPT-4o, Claude Sonnet 4, DeepSeek-V3, Gemini 2.0 Flash	TITAN-SR outperformed all zero-shot LLMs and ASReview PMID: 42767575

Why It Matters / Context

Systematic reviews are the cornerstone of evidence-based oncology, yet the manual screening of titles and abstracts remains a significant bottleneck. As the volume of published literature grows exponentially, the time and cost associated with screening thousands of records per review are unsustainable. Traditional active learning tools, such as ASReview, require hundreds of human decisions to “warm up” before they can effectively prioritize relevant studies, creating a lag in efficiency.

Conversely, while Large Language Models (LLMs) have gained popularity for text analysis, their use in systematic review screening has largely been zero-shot—meaning they are used without task-specific training or fine-tuning. This study addresses a critical gap by developing a purpose-built tool, TITAN-SR, that leverages the specific linguistic nuances of biomedical literature. By training on nearly 20,000 completed reviews, TITAN-SR learns the complex relationship between eligibility criteria and citation relevance, a task that generic LLMs struggle to perform with high precision without extensive prompting or fine-tuning.

The development of TITAN-SR is particularly relevant for oncology researchers who frequently conduct meta-analyses of clinical trial data. The tool’s ability to maintain high specificity (0.902) at a 99% recall threshold suggests it can significantly reduce the number of irrelevant records a human reviewer must examine, thereby accelerating the evidence synthesis process without compromising the sensitivity required to avoid missing key studies.

Details

The study compared TITAN-SR against two categories of tools: commercial LLM chatbots and the active learning platform ASReview. The evaluation was conducted on a stratified benchmark of 500 reviews for the LLM comparison and 22 temporally independent Cochrane reviews for the ASReview comparison.

Model Architecture and Training

TITAN-SR is an ensemble of two biomedical BERT models: PubMedBERT and BioLinkBERT. These models were trained to read a review’s specific eligibility criteria alongside each citation’s title and abstract. A key methodological choice was the loss function, which penalized false negatives (missing a relevant study) 10 times more heavily than false positives (flagging an irrelevant study). This weighting aligns with the clinical priority of high sensitivity in systematic reviews, where missing a pivotal trial is more detrimental than reviewing an extra irrelevant paper.

Comparative Performance

The following table summarizes the comparative performance of TITAN-SR against the evaluated tools. Note that LLMs were evaluated zero-shot, while ASReview received a 20% warm-up of known includes.

Tool	Type	Evaluation Set	Key Outcome vs. TITAN-SR
TITAN-SR	Biomedical BERT Ensemble	3,434 test reviews	Reference Standard (Median AUC 0.974) PMID: 42767575
GPT-4o	Commercial LLM	500-review benchmark	Outperformed by TITAN-SR (p ≤ 5 × 10⁻⁵) PMID: 42767575
Claude Sonnet 4	Commercial LLM	500-review benchmark	Outperformed by TITAN-SR (p ≤ 5 × 10⁻⁵) PMID: 42767575
DeepSeek-V3	Commercial LLM	500-review benchmark	Outperformed by TITAN-SR (p ≤ 5 × 10⁻⁵) PMID: 42767575
Gemini 2.0 Flash	Commercial LLM	500-review benchmark	Outperformed by TITAN-SR (p ≤ 5 × 10⁻⁵) PMID: 42767575
GPT-4o-mini	Commercial LLM	3,434 test reviews	Outperformed by TITAN-SR PMID: 42767575
ASReview	Active Learning	22 Cochrane reviews (142,504 records)	Outperformed by TITAN-SR PMID: 42767575

Note: Statistical significance for LLM comparisons was determined using Bonferroni-corrected paired Wilcoxon tests.

The temporal validation on 22 Cochrane reviews is particularly robust, as these reviews were published after the model development period. This confirms that TITAN-SR generalizes well to new, unseen data, a critical requirement for a tool intended for ongoing use in evolving research fields like oncology.

Limitations

Critical Considerations for Clinical Application

1. Zero-Shot LLM Comparison: The LLMs were evaluated without task-specific training or fine-tuning. While this reflects current common usage, it does not rule out the possibility that fine-tuned LLMs could match or exceed TITAN-SR performance.

2. ASReview Warm-Up: ASReview was given a 20% warm-up of known includes, which is a standard but potentially favorable condition for active learning tools. TITAN-SR’s advantage is therefore established under conditions where ASReview is already optimized.

3. Scope of Validation: While the dataset is large (762,934 pairs), the external validation for ASReview was limited to Cochrane reviews. Performance on non-Cochrane systematic reviews or rapid reviews may vary.

4. Static Model: TITAN-SR is a static model trained on historical data. It does not adapt in real-time to new screening tasks without retraining, unlike active learning tools which learn from user feedback during the screening process.

Clinical Takeaway

Oncology researchers conducting systematic reviews should consider implementing TITAN-SR or similar purpose-built BERT ensembles to reduce screening burden, as they demonstrate superior specificity and AUC compared to zero-shot LLMs and active learning tools in external validation.

Full Reference

Pitre T, Zeraatkar D, Granton J, Busse J, Rochwerg B, Guyatt G. A biomedical BERT ensemble (TITAN-SR) outperformed active learning and LLM chatbots for systematic-review title and abstract screening: development and external validation. Journal of Clinical Epidemiology. 2026. PMID: 42767575 [DOI: 10.1016/j.jclinepi.2026.112514]