OpenResearchAlpha

Frontiers in Digital Health · 18 Nov 2026

Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation

Joe Hasei, Yuichiro Toda, Yuki Nishimura, Taichi Ikezaki, Hiroki Matsuoka, Koichi Takeuchi, Ryuichi Nakahara, Toshifumi Ozakih-index 10

0
Citations
no database records one yet
0
Citations in 2026
so far
10
Leading author's h-index
Toshifumi Ozaki, in QAI's scholar index
27
References
works it cites, from open citation data

Citations by database

Each database indexes a different set of journals, preprints and books, so they are shown side by side rather than blended.

  • OpenCitations0
  • Crossref0

Citations per year

The current year is still counting.

  • 2026 so far0

Abstract

Background We developed a hospital web application that searches 266 destinations and links results to indoor navigation. Its default retrieval tiers run locally; an optional cloud large language model (LLM) tier requires explicit invocation. Synthetic queries were used because default-tier searches are not server-logged. Objective We examined how post hoc review of an LLM-generated benchmark changed the apparent benefit of completed system updates. Methods Version v2 added pronunciation aliases to v1; v3 added colloquial knowledge, pronunciation forms, destination-level score combination, and false-match rules. Two v3 rules were informed by evaluation-query failures. A clinician and a medical administrative coauthor jointly reviewed 563 synthetic queries without blinding, excluding 129 and rewriting 289 under review-defined policies. The final data contained 401 answerable and 33 no-valid-destination queries. Archived versions were compared across sequential review stages and with lexical and embedding implementations. Results Among the same 192 answerable evaluation-query identifiers and final accepted destinations, rewriting increased top-1 accuracy from 43.8% to 62.0% for v1 and from 56.8% to 63.0% for v3. Their difference remained near 13 percentage points through earlier review stages and decreased to 1.0 point at rewriting. The tested multilingual-E5-small implementation achieved higher top-3 accuracy than v3 (87.0% vs. 69.3%; exploratory P < 0.001), with differences in passage scoring, abstention, and evidence display. For v3, candidate returns occurred on 4/11 reviewed no-valid-destination evaluation queries and 11/29 original queries in a retrospective sensitivity analysis. Conclusions Unblinded benchmark editing changed performance estimates for frozen systems within this synthetic corpus. The findings do not establish improved real-user performance, independent generalization, or comparative clinical safety. The version difference is conditional on the review intervention; neither wording distribution was validated against patient input. Review before optimization and independent user evaluation require prospective assessment.

What the data shows

  • No database records a citation yet, as is usual for a paper this new.
  • Toshifumi Ozaki has the highest h-index among the authors (10).

Graph

This paper in the middle, its authors and their most-cited papers below, the papers it cites to the left and the papers citing it to the right. The timeline lays every paper out by year.

Citing paper shares an authorAuthor without a profileSize shows citations, or h-index for authors

Select a node for its details. Drag to pan, scroll to zoom, drag a node to move it.

Start with

Authors (8)

Each author's standing in QAI's scholar index: the works their ORCID iD is attached to, and for anyone whose ORCID record names a lab OpenResearch follows, their arXiv preprints matched by name and a shared co-author. Names are as printed on the paper.

ProfileMost-cited paper

Cited by 0 papers

Who cites this paper, from open citation data, with Crossref's record of each; the 100 newest when there are more.

0 shown
No citing papers for this filter.

References (27)

What it cites, from open citation data, with Crossref's record of each.

Sign in to see 7 more references

A free account opens the rest of the citing papers and references on every paper’s page.

Changes in apparent system performance after post hoc human review of a large language model-generated benchmark for hospital wayfinding search: a comparative methodological evaluation - QAI OpenResearch