Evaluation of NLP Algorithms for Identifying IBD from Clinical Records
The study conducted a comprehensive evaluation of 15 natural language processing (NLP) algorithms to identify patients with inflammatory bowel disease (IBD) from free-text secondary care clinical records. Here are the detailed findings related to the evaluation:
1. Study Objectives
The primary goal was to compare the performance of various NLP algorithms spanning 50 years of evolution, ranging from traditional rule-based models to advanced transformer-based and large language models (LLMs). The evaluation focused on multiple aspects, including accuracy, fairness, cost-efficiency, explainability, and environmental impact.
2. Dataset
The study utilized a robust dataset of 9311 labeled clinical documents from 1612 patients. This dataset ensured high diversity and statistical power, allowing for reliable model comparisons.
3. Algorithm Scope
The algorithms evaluated included:
- Traditional NLP models: Rule-based regex, TF-IDF, and Bag-of-Words (BoW).
- Transformer-based models: BERT, DistilBERT, sBERT.
- Large language models (LLMs): DeepSeek, Mistral, Qwen.
4. Performance Results
Top Performer
- DistilBERT_IBD: Achieved the highest accuracy with a micro F1 score of 93.54%, closely followed by sBERT with a score of 93.05%. However, both models exhibited moderate specificity limitations.
LLM Performance
- LLMs like Mistral, Qwen, and DeepSeek performed well without exposure to training data, achieving micro F1 scores between 86.47% and 92.20%. However, they required longer runtimes and higher computational resources.
Traditional Models
- Regex and spaCy negation models were fast and cost-effective but had poor specificity, making them unsuitable for precise patient identification.
- TF-IDF and BoW demonstrated solid baseline performance (~92% F1) but lacked contextual understanding for complex medical text.
Document vs Patient-Level Accuracy
- Document-level accuracy was higher compared to patient-level accuracy due to misalignment between document predictions and overall patient diagnoses, which reduced recall and calibration.
5. Fairness Findings
- All models exhibited biases, particularly against women, wealthier individuals, and patients of African ethnicity.
- LLMs were more balanced in terms of gender bias but displayed mild age-related bias, overpredicting IBD in older age groups.
6. Economic and Environmental Analysis
Cost Efficiency
- Simple models like regex and TF-IDF were the fastest and cheapest, with processing times under 2 minutes and carbon emissions below 2g CO₂.
- In contrast, the 70B parameter LLM consumed over 33,000g CO₂, highlighting sustainability concerns for large-scale deployment.
Energy Consumption
- Transformer-based and LLM models required significantly more energy (up to 163 kWh) compared to traditional machine learning methods.
7. Explainability
- Explainability tools such as SHAP and LIME were used to analyze model decision-making patterns and detect overfitting.
- LLMs posed unique challenges for interpretability, requiring the development of new frameworks beyond traditional feature attribution methods.
8. Calibration
- sBERT-Base demonstrated the best calibration with the lowest Brier scores, indicating reliable probability predictions compared to other models.
9. Document Contribution
- Clinic letters were found to be the most predictive of IBD, with higher odds ratios (22.69–23.38) compared to endoscopy or pathology reports, reflecting their richer contextual information.
10. Clinical Implications
The study’s findings have significant implications for healthcare:
- Transformer-based models like BERT and DistilBERT offer the best trade-off between accuracy, efficiency, and interpretability, making them ideal for clinical NLP deployment.
- Open-source availability on platforms like Hugging Face and GitHub allows healthcare institutions to replicate or fine-tune models for their local electronic health record (EHR) systems.
11. Future Outlook
The study predicts that once issues related to cost, performance, and bias are addressed, LLMs will become the standard for clinical data retrieval and patient identification within the next decade.
12. Study Strengths
The study is notable for its:
- Transparency and open-source code.
- Multi-metric evaluation encompassing accuracy, fairness, cost, energy consumption, and explainability.
- Robust dataset and diverse algorithm comparison, making it a benchmark for clinical NLP validation.
In summary, the evaluation highlighted the strengths and limitations of different NLP algorithms in identifying IBD from clinical records. While transformer-based models like DistilBERT and sBERT emerged as top performers, challenges such as fairness, environmental impact, and interpretability remain critical areas for improvement.
