Evaluation of NLP Algorithms for Identifying IBD from Clinical Records

The study conducted a comprehensive evaluation of 15 natural language processing (NLP) algorithms to identify patients with inflammatory bowel disease (IBD) from free-text secondary care clinical records. Here are the detailed findings related to the evaluation:

### 1. **Study Objectives**

The primary goal was to compare the performance of various NLP algorithms spanning 50 years of evolution, ranging from traditional rule-based models to advanced transformer-based and large language models (LLMs). The evaluation focused on multiple aspects, including accuracy, fairness, cost-efficiency, explainability, and environmental impact.

---

### 2. **Dataset**

The study utilized a robust dataset of 9311 labeled clinical documents from 1612 patients. This dataset ensured high diversity and statistical power, allowing for reliable model comparisons.

---

### 3. **Algorithm Scope**

The algorithms evaluated included:

• **Traditional NLP models**: Rule-based regex, TF-IDF, and Bag-of-Words (BoW).
• **Transformer-based models**: BERT, DistilBERT, sBERT.
• **Large language models (LLMs)**: DeepSeek, Mistral, Qwen.

---

### 4. **Performance Results**

#### **Top Performer**

• **DistilBERT_IBD**: Achieved the highest accuracy with a micro F1 score of 93.54%, closely followed by sBERT with a score of 93.05%. However, both models exhibited moderate specificity limitations.

#### **LLM Performance**

• LLMs like Mistral, Qwen, and DeepSeek performed well without exposure to training data, achieving micro F1 scores between 86.47% and 92.20%. However, they required longer runtimes and higher computational resources.

#### **Traditional Models**

• Regex and spaCy negation models were fast and cost-effective but had poor specificity, making them unsuitable for precise patient identification.
• TF-IDF and BoW demonstrated solid baseline performance (~92% F1) but lacked contextual understanding for complex medical text.

#### **Document vs Patient-Level Accuracy**

• Document-level accuracy was higher compared to patient-level accuracy due to misalignment between document predictions and overall patient diagnoses, which reduced recall and calibration.

---

### 5. **Fairness Findings**

• All models exhibited biases, particularly against women, wealthier individuals, and patients of African ethnicity.
• LLMs were more balanced in terms of gender bias but displayed mild age-related bias, overpredicting IBD in older age groups.

---

### 6. **Economic and Environmental Analysis**

#### **Cost Efficiency**

• Simple models like regex and TF-IDF were the fastest and cheapest, with processing times under 2 minutes and carbon emissions below 2g CO₂.
• In contrast, the 70B parameter LLM consumed over 33,000g CO₂, highlighting sustainability concerns for large-scale deployment.

#### **Energy Consumption**

• Transformer-based and LLM models required significantly more energy (up to 163 kWh) compared to traditional machine learning methods.

---

### 7. **Explainability**

• Explainability tools such as SHAP and LIME were used to analyze model decision-making patterns and detect overfitting.
• LLMs posed unique challenges for interpretability, requiring the development of new frameworks beyond traditional feature attribution methods.

---

### 8. **Calibration**

• sBERT-Base demonstrated the best calibration with the lowest Brier scores, indicating reliable probability predictions compared to other models.

---

### 9. **Document Contribution**

• Clinic letters were found to be the most predictive of IBD, with higher odds ratios (22.69–23.38) compared to endoscopy or pathology reports, reflecting their richer contextual information.

---

### 10. **Clinical Implications**

The study’s findings have significant implications for healthcare:

• **Transformer-based models** like BERT and DistilBERT offer the best trade-off between accuracy, efficiency, and interpretability, making them ideal for clinical NLP deployment.
• Open-source availability on platforms like Hugging Face and GitHub allows healthcare institutions to replicate or fine-tune models for their local electronic health record (EHR) systems.

---

### 11. **Future Outlook**

The study predicts that once issues related to cost, performance, and bias are addressed, LLMs will become the standard for clinical data retrieval and patient identification within the next decade.

---

### 12. **Study Strengths**

The study is notable for its:

• Transparency and open-source code.
• Multi-metric evaluation encompassing accuracy, fairness, cost, energy consumption, and explainability.
• Robust dataset and diverse algorithm comparison, making it a benchmark for clinical NLP validation.

---

In summary, the evaluation highlighted the strengths and limitations of different NLP algorithms in identifying IBD from clinical records. While transformer-based models like DistilBERT and sBERT emerged as top performers, challenges such as fairness, environmental impact, and interpretability remain critical areas for improvement.