measurement methodology · evidence standards

Rules for benchmark measurements.

The four-lane testing protocol, anti-cache inflation guardrails, power sweet spots, and verified coordinates behind every published number on Intel Arc GPUs. No naked tok/s without hardware coordinates.

testing protocol · four measurement lanes

The four protocol lanes

From Lane 0 smoke checks to Lane 1 five-cell model cards (p512/g128, p8192/g128, cold input anchors, n=5 repetitions), Lane 2 speculation screens (MTP / DFlash vs no-spec), and Lane 3 full 128K context characterizations.

Browse measured results View model recipes →

guardrails · anti-inflation methodology

Anti-cache inflation guardrails

Mandatory JIT warmup discard (30–60s), thermal cooldown to ≤55 °C between tests, ≥64 characters of prompt entropy to defeat prefix-cache false acceleration, and interval energy-counter watts over BIOS TDP limits.

Inference repository View telemetry →

core methodology

Four foundational measurement rules.

04 Cookbook Evidence ↗

Rule 01 CLAIM-SIZED TESTING LANES

The Four Protocol Lanes n=5 Median

Bounded, repeatable campaigns instead of unchecked Cartesian sweeps

Standardized Matrix

Lane 1 Model Card: 5 anchor cells, n=5 samples, median ± range

We replace unconstrained grids with four disciplined lanes: Lane 0 smoke check, Lane 1 five-cell model card, Lane 2 tuning screen (n=3 screen, fresh n=5 finalist confirmation), and Lane 3 full context and concurrency characterization.

Lane specifications & sample requirements

Lane 0 (Smoke): 1 warmup + 1 diagnostic observation. Validates weights, VRAM headroom, and clean exit.

Lane 1 (Model Card): Anchor coordinates p512/g128, p8192/g128, cold input p2048/g1, p8192/g1, plus one real workload. n=5 repetitions.

Lane 2 (Tuning): Speculative decoding (MTP1/2/4, DFlash) screened at 2 anchors. Winner confirmed with fresh n=5.

Lane 3 (Characterization): Full 131,072 context ladder, power sweeps (150W vs 230W), or multi-client serving (C1 to C32).

Inspect model results→

Rule 02 ANTI-CACHE GUARDRAIL

Anti-Cache Inflation Zero Delta

Defeating the 5× prefix-cache acceleration illusion

vllm:prefix_cache_hits = 0

≥64 chars unique prompt entropy mandatory on cold runs

In naive benchmark loops sharing a prompt template, prefix caching accelerates repetition 2 from 8,640 to 42,733 t/s (a 5× illusion). Cold runs must inject entropy and verify zero prefix-cache hit delta.

Detection protocol & verification

The Bug: Static prompt fillers cause later repetitions to hit the KV cache, producing false prefill scaling claims.

Verification: Snapshot curl localhost:8001/metrics | grep prefix_cache_hits before and after. Valid cold prefill requires after - before = 0.

Mitigation: Inject ≥64 characters of random entropy at prompt start while preserving exact token counts.

llama.cpp: llama-bench generates synthetic prompts with internal cache bypass by default.

Inspect cache policies→

Rule 03 METRIC HYGIENE

C1 Latency vs Cn Concurrency Monotonic Timing

Never sum individual predicted rates into aggregate throughput

Client Monotonic Timers

Cold input rate = actual prompt tokens / client TTFT

C1 isolates single-user interactive latency (TTFT + TPOT). Cn measures multi-tenant concurrent throughput. Synthetic kernel prefill (llama-bench pp) tests raw FLOPs; cold input rate measures real client wall-clock.

Timing sources & reporting definitions

C1 Single Stream: Single client stream; reports client-measured TTFT and per-token generation rate.

Cn Aggregate: Total tokens generated across N concurrent streams divided by total wall time. Never sum per-request rates.

Cold Input Rate: prompt tokens / TTFT under cold cache state.

Isolated Prefill: Only valid when server-native phase timers exclude queueing and first-token launch overhead.

Compare C1 vs C5 numbers→

Rule 04 POWER TELEMETRY

Configured Cap ≠ Physical Draw Energy Integrals

Energy counters over steady state vs nominal BIOS power limits

Hardware Sysfs Telemetry

MoE 35B self-limits to ~140 W · 150 W cap saves 80 W of heat

A GPU set to a 230 W BIOS ceiling draws only ~140 W during sparse MoE decode because only active experts (~3 GB) are read per token. Running at a 150 W cap achieves identical tok/s with 80 W less heat and lower fan noise.

Telemetry standards & power sweet spots

Measurement: Hardware energy-counter deltas divided by campaign duration via xpu-smi / sysfs.

MoE Sweet Spot: 150 W sustained cap — identical decode throughput, 80 W lower heat.

Dense Sweet Spot: 180 W sustained sweet spot (0.155 tok/s/W efficiency point). 230 W stock cap buys deep prefill bursts.

Cooldown Gate: Thermal cooldown to ≤55 °C required before matched A/B sweeps.

View host telemetry→

telemetry & shell

Host telemetry & execution shell.

Live

Dual Arc Pro B70 (BMG-31) testbed — ASUS ROG Strix X570-F, 32 GB DDR4-3200, 850 W 80+ Gold PSU. Driver xe 6.12.0-intel-xpu · Level Zero 1.3.36.

host-cell-b70 · xpu0 @ /dev/dri/renderD128

bash

sergio@b70-lab:~$ xpu-smi discovery --dump 1,2,18,20,22

# Intel XPU System Management Interface — Target Device Topology

Device ID	Device Name	PCI BDF	Visible VRAM	Memory Bandwidth	Max Power (TBP)
0	Intel(R) Arc(TM) Pro B70 GPU	0000:03:00.0	32,656 MiB	608 GB/s (GDDR6)	230 W
1	Intel(R) Arc(TM) Pro B70 GPU	0000:07:00.0	32,656 MiB	608 GB/s (GDDR6)	230 W (VideoGen Node)

Driver version: xe 6.12.0-intel-xpu · Level Zero Loader: 1.3.36 · Device State: Ready

sergio@b70-lab:~$ docker run --rm -it --device /dev/dri:/dev/dri --ipc=host \

-v /models:/models -e ONEAPI_DEVICE_SELECTOR=level_zero:0 \

vllm/vllm-openai-xpu@sha256:f01e24f6... \

--model /models/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 \

--kv-cache-dtype fp8 --gpu-memory-utilization 0.88 --max-model-len 100000

INFO 08-27 18:20:01 [xpu_executor.py:104] Initialized XPU backend on Intel Arc Pro B70 (BMG-31)

INFO 08-27 18:20:04 [speculative.py:214] MTP draft tensor overlay loaded · 4 speculative tokens

INFO 08-27 18:20:07 [server.py:82] Uvicorn running on http://0.0.0.0:8000 (VRAM resident: 18.2 GiB)

sergio@b70-lab:~$ ./llama-bench -m /models/Muse-Glimmer-30B-Q4_K_M.gguf \

-p 512,2048 -n 128 -t 8 -ngl 99 -fa 1 -ctk q8_0 -ctv q4_1 -r 5

# llama.cpp SYCL DPC++ build (-DGGML_SYCL_F16=ON)

Model	Backend	Threads	Test	Prompt Tokens	Gen Tokens	Rate (tok/s)
Muse-Glimmer-30B	SYCL (XPU0)	8	pp512	512	0	3,240.5 ± 21.3
Muse-Glimmer-30B	SYCL (XPU0)	8	tg128	512	128	42.8 ± 0.4
Muse-Glimmer-30B	SYCL (XPU0)	8	pp2048	2048	0	2,890.1 ± 18.7
Muse-Glimmer-30B	SYCL (XPU0)	8	tg128	2048	128	41.6 ± 0.3