Journal of Medical Internet Research · 20 Nov 2026
Natural Language Processing Supported by Annotation Process Artifacts to Detect Risky Alcohol Use in Electronic Health Records: Algorithm Development and Validation Study
Katherine Weberh-index 4, Asher Strayhorn, Xintong Ju, Jessica Mellinger, Gerald Scott Winder, Anne C Fernandez, VG Vinod Vydiswaranh-index 23
Citations by database
Each database indexes a different set of journals, preprints and books, so they are shown side by side rather than blended.
- OpenCitations0
- Crossref0
- Europe PMC0
Citations per year
The current year is still counting.
- 2026 so far0
Abstract
Background High-risk alcohol use and alcohol use disorder (AUD) are significant contributors to surgical risk and complications yet remain underidentified in surgical populations. While electronic health records (EHRs) contain relevant information about patients’ alcohol use in clinical notes, manual chart review is time-intensive and impractical at scale. Natural language processing (NLP) offers a path to automated identification, but alcohol-related risk classes are highly imbalanced in clinical populations, making multiclass classification particularly challenging. Objective This study aims to develop an automated NLP pipeline for 4-class alcohol risk classification using clinical notes in a surgical population, addressing class imbalance by leveraging intermediate annotation evidence generated during the human labeling process. Methods We manually labeled 1000 surgical patients drawn from a large academic health system into ordinal alcohol risk classes (no-alcohol, low-risk, high-risk, and probable-dependence) and a fifth class indicating no mention of alcohol use. We used a structured annotation process with 6 trained annotators and expert-developed guidelines. We developed a multistage NLP pipeline operating on 3 years of clinical notes per patient. First, a rules-based preprocessing stage filtered notes to clinically relevant sections containing alcohol-related content. No-mention cases were excluded from further modeling. Second, a BioClinicalBERT-based multilabel classifier (“auto-annotator”) was fine-tuned to reproduce the annotations created by human labelers at the section level, producing a structured 26-dimensional probability vector for each section. Finally, a bidirectional long short-term memory (LSTM) network classified patients into the 4 risk categories by processing each patient’s ordered sequence of auto-annotation vectors, allowing the model to capture temporally accumulating evidence across clinical encounters. Results Human annotators achieved 93.7% first-pass agreement overall (Fleiss κ=0.8891), but agreement was as low as 67.5% for probable-dependence cases. The 4-class LSTM classifier achieved a weighted-average F1-score of 0.86 and a macroaverage F1-score of 0.77 on a held-out validation set of 195 patients. One-vs-rest area under the curve (AUC) scores were 0.97 for no-alcohol, 0.88 for low-risk, 0.90 for probable-dependence, and 0.71 for high-risk. Collapsed to a binary high/low risk classification, the model achieved an accuracy of 0.92 and a macro-F1-score of 0.80. Performance was comparable across biological sex without explicit sex-specific training. Conclusions We used a 2-stage NLP pipeline to identify 4 ordinal classes of alcohol use risk in surgical patients. Four-class alcohol-use classification presents a highly imbalanced problem that constrains the availability of high-importance but rare labels in the dataset. We addressed this data scarcity by using intermediate annotation evidence as well as final labels to enrich the feature space for the classification task. The pipeline was able to separate patient records into 4 strata of alcohol-use risk, with “high-risk” patients being most likely to be misclassified into an adjacent class.
What the data shows
- No database records a citation yet, as is usual for a paper this new.
- VG Vinod Vydiswaran has the highest h-index among the authors (23).
Graph
This paper in the middle, its authors and their most-cited papers below, the papers it cites to the left and the papers citing it to the right. The timeline lays every paper out by year.
Select a node for its details. Drag to pan, scroll to zoom, drag a node to move it.
Start with
Cited by 0 papers
Who cites this paper, from open citation data, with Crossref's record of each; the 100 newest when there are more.
References (35)
What it cites, from open citation data, with Crossref's record of each.
Sign in to see 15 more references
A free account opens the rest of the citing papers and references on every paper’s page.