EdgeBench
Open-weight models measured on the machine that runs them: how fast they read and write at real prompt lengths, and how often they get real work right, on the exact file being served.
Every configuration answers the same tasks, and each engine runs one standard setup. A row that ran with a setting changed says which.
Devices
Each device EdgeBench has measured, with every configuration on one page
NVIDIA DGX Spark · GB10 · 128 GB unified memory
112 model configurations measured on vLLM, SGLang, llama.cpp, llama.cpp (PrismML fork), Strata and FreeToken, updated . Filter, sort and compare them side by side.
Recorded tests
Single head-to-head runs, replayed token by token
vLLM vs SGLang on Qwen3.8-27B
With the DFlash2 draft model, SGLang wrote 2.0 to 3.3 times as fast as vLLM running without one, on the same machine. vLLM read a brand-new prompt 18% faster.
What it measures
The first three on every configuration; the fourth where a draft model guesses ahead
Reading and writing speed
Reading the prompt (prefill) and writing the answer (decode) are timed apart, at exact prompt lengths counted in the model’s own tokens and built from real source code.
Quality, graded on the device
Tool calls, code, maths, instructions, long-context lookups and bug repairs are graded by running the code or matching the exact answer, with no model as judge, on the exact file being served.
Time per solved task
Seconds per solved task adds the time to read every prompt and write every reply, wrong answers included, and divides by the tasks solved, so a fast model that rambles or answers wrongly does not win.
Speculative decoding, checked
When a small draft model guesses ahead and the main model checks its guesses, the answers are compared task by task with the same model running without the draft, to show whether drafting changed them.
How to read the numbers
Quality and seconds per solved task carry a 95% interval: redraw the measurements at random many times, and 95% of the recomputed figures land inside it. EdgeBench counts two quality scores as different only when their ranges do not overlap. Speeds are medians of repeated runs at each prompt length.
Limits: read before quoting a number
- One quality run per configuration, and every published speed is timed on the first DGX Spark; speeds are medians of repeated timings. 29 rows had their speed re-timed later on that unit, 24 after a firmware update and 5 whose tasks a second, identical DGX Spark had run, so their quality and solved-per-hour come from it. Across those five rows the second unit wrote at 0.95–1.14× the first unit’s speed.
- Energy is the GPU’s own power reading, not power drawn at the wall.
- Quality is a compact check of 86 tasks per configuration, so its ranges are wide. It is not a full coding benchmark.
- Each engine runs one pinned version; other versions may be faster or slower.
- A configuration that would not run properly on this machine is left out, not scored.