JMIR AI · 20 Nov 2026
Balancing Human and AI Features for Digital Autism Diagnostics from Home Videos: Human-In-The-Loop Machine Learning Study
Abdur Rasoolh-index 20, Kaitlyn Dunlap, Arman Husic, Aaron Kline, Parnian Azizian, Dennis P Wall, Peter Washingtonh-index 29
Citations by database
Each database indexes a different set of journals, preprints and books, so they are shown side by side rather than blended.
- OpenCitations0
- Crossref0
- Europe PMC0
Citations per year
The current year is still counting.
- 2026 so far0
Abstract
Background Autism spectrum disorder (ASD or autism) diagnosis faces significant delays due to specialist shortages and subjective, clinic-based assessments. Although AI offers a potentially scalable solution via automated video analysis, current state-of-the-art deep learning models often fail to capture subtle, clinically nuanced social behaviors essential for reliable detection. Objective This study aimed to develop and evaluate a human-in-the-loop (HITL) framework that combines automated AI feature extraction with crowdsourced human annotation for autism classification on naturalistic home videos. We targeted low-confidence AI facial feature extraction outputs to be replaced by human features. We also sought to quantify the relationship between the confidence threshold used for swapping AI features with human features, the volume of human replacement of AI features, and the predictive diagnostic performance. Methods We implemented a HITL pipeline integrating deep learning models for facial emotion (7-class) feature extraction. A confidence threshold applied to the emitted probability of the AI feature extractor model was used to identify low-confidence facial features or those that the AI model was unable to make a prediction for (henceforth referred to as “unresolved”), which were then replaced by human “feature extraction” (ie, annotation) conducted by rigorously vetted crowd workers. We evaluated this approach on 100 high-quality videos (50 autism and 50 neurotypical) from the GuessWhat? digital therapeutic app. Performance was assessed across decision thresholds (0.1-0.9 in increments of 0.1) using sensitivity, specificity, F1-score, and area under the receiver operating characteristic curve (AUROC), with 5-fold cross-validation. Results Across decision thresholds, performance improved overall but was nonmonotonic at intermediate thresholds. At a threshold of 0.9, 59.7% (597/1000) of frames were replaced by human annotations, and mean sensitivity increased from 0.51 (SD 0.24) at a threshold of 0.1 to 0.66 (SD 0.10) at a threshold of 0.9, an absolute increase of approximately 15 percentage points. The HITL-enhanced multimodal model achieved an aggregate out-of-fold AUROC of 0.76, compared with 0.54 for the AI-only model. Conclusions In this proof-of-concept study, replacing AI features with human features tended to be associated with higher observed autism classification performance in this particular dataset. We targeted low-confidence and unresolved frames for human annotation. We note that these findings are exploratory and thus do not isolate the independent causal effect of human replacement or establish clinical diagnostic validity.
What the data shows
- No database records a citation yet, as is usual for a paper this new.
- Peter Washington has the highest h-index among the authors (29).
Graph
This paper in the middle, its authors and their most-cited papers below, the papers it cites to the left and the papers citing it to the right. The timeline lays every paper out by year.
Select a node for its details. Drag to pan, scroll to zoom, drag a node to move it.
Start with
Cited by 0 papers
Who cites this paper, from open citation data, with Crossref's record of each; the 100 newest when there are more.
References (38)
What it cites, from open citation data, with Crossref's record of each.
Sign in to see 18 more references
A free account opens the rest of the citing papers and references on every paper’s page.