OpenResearchAlpha

Journal of Medical Internet Research · 13 Nov 2026

Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study

Zuyi Yangh-index 5, Dianzhe Tian, Meiqian He, Erqian Wang, Lvyuxing Zhao, Xinyu Zhao, Xiaokai Chen, Wei Xia, Wei Zhou, Yanhao Ma, Chuanyong Ma, Haiyan Chen, Huiqin Lu, Shulin Liu, Xin Zhao, Yiyao Xu, Xin Lu, Fengdan Wang, Lei Zhang, Youxin Chenh-index 18

0
Citations
no database records one yet
0
Citations in 2026
so far
18
Leading author's h-index
Youxin Chen, in QAI's scholar index
37
References
works it cites, from open citation data

Citations by database

Each database indexes a different set of journals, preprints and books, so they are shown side by side rather than blended.

  • OpenCitations0
  • Crossref0
  • Europe PMC0

Citations per year

The current year is still counting.

  • 2026 so far0

Abstract

Abstract Background DeepSeek, an open-source large language model (LLM) extensively adopted in hospitals across China, presents a locally deployable alternative to proprietary models such as GPT. However, its real-world clinical performance and safeguards remain underexplored. Objective The aim of this study is to evaluate the performance of the general-purpose and reasoning versions of DeepSeek (DeepSeek AI) and GPT models in diagnosis, medical record analysis, and safeguards against adversarial vulnerabilities. Methods This cross-sectional study included 50 real-world medical records from 10 hospitals spanning 14 specialties. A total of 7 LLMs were evaluated, including DeepSeek-V3, DeepSeek-R1, GPT-4o (via ChatGPT; OpenAI), o3-mini (via ChatGPT), o3-mini-high (via ChatGPT), GPT-4o (via Copilot; Microsoft), and GPT-4o-reasoning (via Copilot). Models were tasked with generating diagnoses based on admission notes, answering 12 questions related to extraction, summarization, and inference using discharge summaries, and undergoing 4 rounds of simulated adversarial prompt attacks designed to elicit misinformation. The diagnostic evaluation was performed on 47 records after the exclusion of 3 flawed admission notes, while the other tasks used the full set of 50 records. Diagnostic performance was evaluated based on accuracy, with the causes of misdiagnosis further categorized. Medical record analysis was assessed by scores in extraction, summarization, and inference tasks. Safeguard evaluation included identifying the earliest round of successful prompt attack and verifying reference authenticity and accuracy. Results Among the 7 models included in this study, DeepSeek-R1 and GPT-4o-reasoning (via Copilot) achieved the highest diagnostic accuracy at 64% (30/47). In medical record analysis, both models also excelled across the 3 subdimensions. GPT-4o-reasoning (via Copilot) achieved accuracies of 97.7% (293/300) in extraction, 87.5% (175/200) in summarization, and 87% in inference, while DeepSeek-R1 reached 97.7% (293/300), 83.5% (167/200), and 85% (85/100), respectively. In simulated adversarial safeguard evaluation involving 4 rounds of prompt attacks, all evaluated models showed high vulnerability. DeepSeek-R1 generated medical disinformation in 98% (49/50) of cases, with only a 2% refusal rate. GPT-4o-reasoning (via Copilot) showed the highest refusal rate at 30%, but it still generated disinformation in 70% (35/50) of cases. True hallucination rates ranged from 38.6% (27/70; GPT-4o-reasoning via Copilot) to 73% (73/100; DeepSeek-V3). Conclusions These findings underscore the clinical application potential of DeepSeek models, while also emphasizing the critical need to strengthen safeguards against adversarial vulnerabilities prior to their clinical deployment.

What the data shows

  • No database records a citation yet, as is usual for a paper this new.
  • Youxin Chen has the highest h-index among the authors (18).

Graph

This paper in the middle, its authors and their most-cited papers below, the papers it cites to the left and the papers citing it to the right. The timeline lays every paper out by year.

Citing paper shares an authorAuthor without a profileSize shows citations, or h-index for authors

Select a node for its details. Drag to pan, scroll to zoom, drag a node to move it.

Start with

Authors (20)

Each author's standing in QAI's scholar index: the works their ORCID iD is attached to, and for anyone whose ORCID record names a lab OpenResearch follows, their arXiv preprints matched by name and a shared co-author. Names are as printed on the paper.

ProfileMost-cited paper

Cited by 0 papers

Who cites this paper, from open citation data, with Crossref's record of each; the 100 newest when there are more.

0 shown
No citing papers for this filter.

References (37)

What it cites, from open citation data, with Crossref's record of each.

Sign in to see 17 more references

A free account opens the rest of the citing papers and references on every paper’s page.