JMIR Formative Research · 13 Nov 2026
Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study
Rezaur Rashidh-index 4, Brianna M White, Fekede Asefa Kumsa, Arash Shaban-Nejadh-index 23
Citations by database
Each database indexes a different set of journals, preprints and books, so they are shown side by side rather than blended.
- OpenCitations0
- Crossref0
Citations per year
The current year is still counting.
- 2026 so far0
Abstract
Abstract Background Pregnancy-related weight gain is both physiologically complex and socially charged, and online discussions often combine clinical information with implicit emotional distress. Although automated sentiment analysis could support research on pregnancy-related discourse, model performance in this context remains inadequately characterized relative to expert human judgment. Objective This study aimed to benchmark contemporary sentiment classification models on pregnancy-related weight discussions, with emphasis on agreement with expert consensus, interpretive ambiguity, task adaptation, and deployment across posts and comments. Methods We curated 15,619 Reddit posts and comments using a large language model (LLM)–guided keyword expansion strategy and established an adjudicated consensus reference set of 200 posts. We evaluated 10 models on a fixed held-out test set (N=150) using aligned text inputs, weighted performance metrics, and 1000-resample bootstrap CIs. RoBERTa (Robustly Optimized Bidirectional Encoder Representations from Transformers Approach) and BERTweet (Bidirectional Encoder Representations from Transformers for Tweets) were additionally evaluated in a 5-fold fine-tuning sensitivity analysis. Gemini-3-Flash (Google DeepMind [Google LLC]) and Generative Pretrained Transformer (GPT)–OSS (OpenAI) were then applied to 400 unlabeled posts and 15,019 comments with intermodel agreement assessment. To expand expert-labeled coverage, a separate 106-post annotation extension was used for secondary evaluation of the 2 models carried forward to post-level corpus inference, and cross-source recoverability of the API-derived retrieval was assessed using Arctic Shift (Arthur Heitmann). Results The 2 primary experts (FAK and BMW) achieved 59.0% preadjudication agreement (Cohen κ=0.344). In the fixed held-out comparison, Gemini-3-Flash had the numerically highest weighted F 1 -score of 0.662 (κ=0.478), followed by GPT-OSS ( F 1 -score=0.645; κ=0.436), with overlapping CIs across leading models. Under the harmonized direct-argmax decision rule with minimally processed text, BERTweet achieved an F 1 -score of 0.623 (κ=0.369), and RoBERTa achieved an F 1 -score of 0.605 (κ=0.339). In the separate fine-tuning sensitivity analysis, the mean cross-validated F 1 -score was 0.708 (SD 0.087) for RoBERTa and 0.616 (SD 0.079) for BERTweet. Among Gemini-3-Flash errors, 55.8% (29/52) occurred in posts on which the 2 primary experts disagreed. In the 106-post annotation extension, primary-expert agreement was 64.2% (68/106; κ=0.367); Gemini-3-Flash and GPT-OSS achieved weighted F 1 -score values of 0.633 and 0.680, respectively, and both identified 56 of 58 expert-negative posts in an exploratory human-review prioritization analysis. All 714 API-retrieved post IDs were independently recoverable through Arctic Shift. Intermodel agreement was 92.3% for posts (κ=0.797; n=400 valid pairs) and 46.3% for comments (κ=0.102; n=1500). Conclusions No single model architecture was uniformly superior, and performance depended on the adaptation regime, annotation ambiguity, and discourse type. These findings define important boundary conditions for sentiment analysis of naturalistic maternal-health discourse: broad polarity classification can support corpus prioritization and hypothesis generation but is insufficient as a stand-alone measure of patient experience.
What the data shows
- No database records a citation yet, as is usual for a paper this new.
- Arash Shaban-Nejad has the highest h-index among the authors (23).
Graph
This paper in the middle, its authors and their most-cited papers below, the papers it cites to the left and the papers citing it to the right. The timeline lays every paper out by year.
Select a node for its details. Drag to pan, scroll to zoom, drag a node to move it.
Start with
Cited by 0 papers
Who cites this paper, from open citation data, with Crossref's record of each; the 100 newest when there are more.
References (44)
What it cites, from open citation data, with Crossref's record of each.
Sign in to see 24 more references
A free account opens the rest of the citing papers and references on every paper’s page.