Alibaba · Qwen
EXL3-XPU engine · dense 27B · default route
Qwen3.8-27B EXL3 4bpw + MTP3
ExLlamaV3 trellis on a vLLM-XPU fork — primary route since the 2026-09-27 re-measure
Tested on 1× B70 (32GB)
Best for: the longest context — full 256K on one card.
Loading current benchmark…
The only measured route that reaches the full 262,144-token context on one card — and the only quantized arm that passes the exact-answer battery. Every W4A16/GPTQ export of this model answers "30" on in-head arithmetic; the BF16 base answers 14. OpenVINO int4 is the steady-150W alternate.
Stack & configuration
Image: ghcr.io/0xsero/exl3xpu@sha256:21412bdd… · vLLM 0.26.1.dev (separate engine from mainline vLLM; isolated dockerd required)
Artifact: turboderp/Qwen3.8-27B-exl3 rev 113cf7ab (~15.7 GiB, trellis 4.00bpw, vision tower)
Context: full native 262,144 on one card (fp8 KV pool 287,040 tokens at util 0.94) — 131K completion proven
KV cache: fp8 only (bf16 rejected) · prefix caching ON by default (~5.4× on repeated 10K prefixes)
Power: most cap-sensitive arm: +38–42% sustained decode at 230 W vs 150 W
Closed alternative: W4A16/GPTQ-family exports (GPTQModel, AutoRound compressed-tensors, local auto_gptq) — deterministic "30" defect, ~15 tok/s, 256K unreachable
Alibaba · Qwen
EXL3-XPU mixed-K · dense 27B · quality pick
Qwen3.8-27B SC5 EXL3 5.0bpw mixed MTP3
turboderp's SC_5.00bpw_H6 on exl3xpu with the mixed-K patches
Tested on 1× B70 (32GB)
Best for: lowest measured next-token KL divergence vs full BF16 on one card (quality screen).
Loading current benchmark…
Each tensor gets 4 to 7 bits, chosen by turboderp's calibration. Stock exl3xpu loads only 4- and 6-bit tensors, so this runs on a patched image. Decodes at ~0.89× the speed of plain 4.0bpw because it reads more bytes per token, trading decode speed for lower next-token divergence vs BF16.
KL divergence of next-token probabilities vs BF16 (lower is closer)
Self-reported single-run quality screen; not independently reproduced.
Scroll table horizontally →
Stack & configuration
Launch: b70 launch qwen38-27b-sc5 (b70-launcher 0.6.3+)
Image: b70-launcher 0.6.3+ pins ghcr.io/sergiiob/exl3xpu-mixedk:20261014 (sha256:933eda54…), built from SergiioB/exl3xpu-mixedk on 0xSero's exl3xpu 0.1.0
Artifact: turboderp/Qwen3.8-27B-exl3 branch SC_5.00bpw_H6 rev dfe2f4fd (~18.6 GiB)
Speed: 56.6 vs 63.6 tok/s at 524-token prompts, 56.8 vs 64.3 at 8.1K (0.88–0.89× plain 4.0bpw; 5 interleaved runs per arm, MTP3, one user, 230 W)
Checks: 46/46 kernel checks, 6/6 exact-answer prompts, 3/3 ~8.2K-token prompts
Context: 190K+ tested for single-request inference: 196,438 occupied tokens passed exact retrieval and multi-turn checks with fp8 KV, MTP3 and memory utilization 0.94; 1,069 MiB VRAM remained free. This works through the API or headless CLI, not only the desktop app. For more reserve, 131,072 configured at utilization 0.90 passed at 130,872 occupied tokens with 2,327 MiB free. These are capacity checks, not long-context throughput claims. Plain 4.0bpw's 262K route has a smaller weight footprint.
V6 branch: SC_5.00bpw_H6_V6 has byte-identical language-model tensors (all 2,093); only the vision tower changed. It does not load on this image. Text-weight KL is unchanged; no served V6 speed measurement is claimed.
Alibaba · Qwen
vLLM XPU · MoE 35B (3B active)
Qwen3.6-35B-A3B Native MTP 1/2/4
MTP-preserved community GPTQ, full 128K window
Tested on 1× B70 (32GB)
Best for: everyday chat and coding — fast and cool-running.
Loading current benchmark…
MoE decode reads only the ~3 GB of active experts per token, so it self-limits to ~140 W: run it at a 150 W cap — same speed, 80 W less heat. MTP2 beats MTP4 deep in context (85.8% acceptance at 128K).
Stack & configuration
Image: vllm/vllm-openai-xpu@sha256:2c427ef4… · vLLM 0.26.1rc1.dev457 · kernels 0.1.12
Artifact: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GPTQ-Int4
Context: 131,072 · scheduler 8,192 (16,384 buys +17.6% prefill, +12.0% decode) · util 0.85 · fp16 KV
Patches: mtp_nightly then mtp_boundary
Cold input: client-proxy input rate, flat across context depth — see
benchmarks
Alibaba · Qwen
vLLM XPU · dense 27B · fp8 KV
Qwen3.6-27B Dense + mandatory fp8 KV
MTP-preserved community GPTQ, heretic-v2 base
Tested on 1× B70 (32GB)
Best for: long sessions that re-read the same document — follow-ups get much faster.
Loading current benchmark…
The KV cache decides this recipe: FP16 KV needs ~9.5 GiB at 128K and does not fit; fp8 halves it. In a resident 32K document session, follow-up TTFT drops from 38.2 s cold to ~4.1–4.5 s on 91% prefix hits.
Stack & configuration
Image: 2c427ef4 digest (same as the MoE) · vLLM 0.26.1rc1.dev457
Artifact: llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-GPTQ-Int4
Context: 131,072 safe ceiling — 200K loads with <5 MiB free (abort zone) · util 0.88 for MTP4
Power: dense prefill scales +52% from 165 W to 230 W; 180 W is the sustained sweet spot
NVIDIA
vLLM XPU · Mamba2 + LatentMoE (3B active)
Nemotron-3.5-Lightning-30B-A3B DFlash n=7
GPTQ-INT4 G64 target + NVFP4→BF16 DFlash draft
Tested on 1× B70 (32GB)
Best for: sustained throughput on long prompts.
Loading current benchmark…
Native MTP does not work on this stack; the converted DFlash draft is the working speculator. The capacity campaign and two upstream fixes are documented in the full recipe.
Stack & configuration
Image: vllm/vllm-openai-xpu@sha256:1da0a954… · vLLM 0.26.1rc1.dev668 · kernels 0.1.12.3
Artifacts: target SergiioB/…-GPTQ-INT4-G64-sym (~18 GB) + draft …-DFlash-BF16 (~1.7 GB)
Serving: max-model-len 120,000 · prefix caching ON (production policy) · measured cells: max_num_seqs 1, prefix cache off, 150 W cap
Power draw: 149–160 W cells, 179.3 W peak interval, 68 °C max
Headline rule: use only the representative catalog record; do not select the fastest cell from a wide family
Open weights · local GPTQ
vLLM XPU · hybrid GDN MoE
Ornith-1.5-35B-A3B MTP1 + DraftINT4
qwen3_5_moe: 30 GDN + 10 full-attn layers, 256 experts × 8 active
Tested on 1× B70 (32GB)
Best for: trying the newest hybrid MoE — self-reported results, check the evidence.
Loading current benchmark…
There is no official GPTQ for this architecture, so the target is a local experts-only quant with the MTP layer left in BF16. MixedCal-v2 calibration cuts RTN fallback from 24.76% to 10.37% — a conversion-quality win at speed parity. Self-reported E2 with raw evidence.
Stack & configuration
Image: f01e24f6 digest (same generation as Qwen3.8) · vLLM 0.27.2rc1.dev77 · WNA16 MoE backend
Artifact: SergiioB/Ornith-1.5-35B-A3B-GPTQ-Int4-sym-G128-MTP-BF16-MixedCal-v2 (~22.77 GiB, 6 shards)
Serving: default MTP1 + DraftINT4 · prefix caching ON (serving default; measured cells used cache off) · fp16 KV (only 10 full-attn layers need it)
Context ladder: 65,536 / 131,072 / 262,144 context ladder, loads C1 — see
benchmarks
Power A/B: 150 W holds decode, 230 W buys prefill — see
benchmarks
Meta
llama.cpp SYCL · dense multimodal 30B
Muse-Glimmer-30B Vision + DFlash n2
UD-Q4_K_XL + mmproj + dflash GGUF trio
Tested on 1× B70 (32GB)
Best for: images + text — vision on llama.cpp, no Docker.
Loading current benchmark…
The llama.cpp path supports the reasoning channel, vision through mmproj, and block-diffusion drafting. The -DGGML_SYCL_F16=ON build flag is mandatory for usable prefill.
Stack & configuration
Engine: llama.cpp d2f83055d · IntelLLVM 2026.0 · GGML_SYCL_F16=ON
Artifacts: unsloth UD-Q4_K_XL 15.88 GB · mmproj-kquant 1.40 GB · dflash-kquant 1.63 GB
Serve: -c 131072 --parallel 1 -fa on -ctk q8_0 -ctv q4_1 -ub 8192, draft n-max 2
Failures logged: n8 aborts at 128K; n5–7 collapse to 0.30–0.37 acceptance; vLLM path still experimental
Intel · OpenVINO
OpenVINO GenAI · INT8-OV IR · native MTP5
Qwen3.8-27B INT8-OV + MTP5 Short-context champion
Hub INT8 IR · VLMPipeline · byte-identical greedy output
Tested on 1× B70 (32GB) · 230 W cap
Best for: simplest install (pip, no Docker) and the fastest short-context decode.
Documented capability on 1× B70 at the 230 W cap — MTP5 decode with greedy parity; not a ranked catalog cell methodology →
The production OpenVINO route on one B70. Native MTP5 gives 3.65× over no-MTP with greedy parity holding at every depth. In the 2026-09-15 INT4-class ranking, OpenVINO INT4-GDN8 + grafted MTP5 leads short context at 79.5 tok/s @512; vLLM stays the serving leader and llama.cpp owns the full 256K. OpenVINO's protocol ceiling is 128K.
Stack & configuration
Engine: OpenVINO GenAI 2026.5.0.0-3412 · VLMPipeline (the int8-ov export is a VL split — LLMPipeline dies on input_ids)
Artifact: OpenVINO/Qwen3.8-27B-int8-ov · draft openvino_mtp_model.xml · 25.8/32 GB resident
Serving: ChatML rendered manually (apply_chat_template=False) · num_assistant_tokens=5 · prefix caching OFF (MTP requires it)
Context ladder: 48K clean MTP+f16 KV · 64K with u8-main/f16-draft KV · 96K no-MTP paths (13.3–14.5)
Power: 202–211 W mean at the 230 W cap
Alibaba · QwenvLLM XPU · FP8 · tensor parallel 2
Qwen3.8-27B FP8 TP2 2× B70
One model across two cards; requires worker affinity and oneCCL settings.
Tested on 2× B70 (64GB installed)
Best for: two-card rigs — one big model split across both B70s.
Follow the model-specific image and patch matrix. Multi-card memory is not automatically pooled; the topology and context limit belong to this recipe.
Intel · OpenVINO
Hub INT4 export · artifact failed
Qwen3.8-27B INT4-OV OpenVINO IR
Official Hub export OpenVINO/Qwen3.8-27B-int4-ov
Tested on 1× B70 (32GB) · 150 W
Quality gate failed (greedy token id 0). Speed is not a ranking cell.
The engine is production — the working OpenVINO card above runs the INT8 IR with quality gates passed. This specific Hub all-INT4 export is broken: greedy decode emits token 0 on GPU and CPU alike (openvino.genai#4467). FastDraft companion IR was never published. Do not download this artifact; use INT8 or the mixed GDN8 IR.
What was tested
Load: VLMPipeline GPU.0 LATENCY + plugin cache. generate() must use generation_config= (positional cfg is treated as images)
Decode: 128 greedy tokens run; every id is 0. Do not quote wall tok/s
CB: SchedulerConfig linear-attn cache OOMs or exceeds cache_size
Tokenizer: convert_tokenizer required on OpenVINO 2026.2/2026.3 Intel IRs
Community Labs
Cascadia 0.2.3 · research · not production
Cascadia on Qwen3.8-27B ov-genai + qwen35
Rust OpenAI-compatible worker wrapping OpenVINO
Tested on 1× B70 (32GB) · 150 W
ov-genai SIGSEGV. qwen35 1-stage loads, HTTP content empty.
Cascadia is not a second GPU stack. ov-genai is FFI over GenAI; qwen35 is IR surgery for Qwen3.5/3.8. The 0.2.3 Linux binary ships OpenVINO 2026.2 and crashes on the official VLM IR. After cascadia shard --num-stages 1 the worker serves, engine log ~5 tok/s, usage.completion_tokens 0.
What was tested
ov-genai: SIGSEGV in Tokenizer::setup_tokenizer / VLMPipeline CB (libopenvino.so.2620)
qwen35: 1-stage shard loads in ~46 s. --ov-* flags ignored. Prefix cache works
Quality: shard validator logits max_abs=0. Same token-0 as Python GenAI
Not tested yet: FastDraft (no IR), --cb at C≥4, dual-card --num-stages 2
MiniMax
ComfyUI 0.33.0 · PyTorch 2.13.0+xpu · T2V
MiniMax H3 FL2VA Dual B70
GPU0 UNet+VAE (~26.9 GB) · GPU1 CLIP (~17.0 GB)
Tested on 2× B70 (64GB)
Best for: text-to-video clips — needs two B70s.
ComfyUI T2V wall times per act, OmniXPU ESIMD+FP8 kernels recipe ↗
Two cards, two roles — a role split without tensor parallelism. Native canvas 1344×768; 124 frames ≈ 5 s, 243 ≈ 10 s, longer pieces concatenate from acts. OmniXPU kernels buy 1.15× s/it (grainier); the turbo LoRA cuts sampling to 8 steps.
Stack & configuration
Engine: ComfyUI v0.33.0 checkout · PyTorch 2.13.0+xpu venv · omni_xpu_kernel 0.1.0b8.dev0 · host venv, no Docker
Weights: six files ≈ 40 GiB — UNet FP8 19.5 · CLIP (Qwen3-VL 32B) NVFP4 14.6 · video VAE 4.9 · audio VAE 0.6 · turbo LoRA 0.35 · people LoRA 0.13 GiB
Recipe: 1344×768 · 124f @ 24 fps · er_sde 8 steps · sigma shift 12+3 · turbo 0.75 · 180 W both cards
Resident runs: act 1 ≈ 650 s cold load, acts 2–5 ≈ 444 s each (H3_SKIP_UNLOAD=1)
Energy: 5-act run totals — GPU0 406 kJ, GPU1 113 kJ, ~94× realtime factor
Rendered on this host
Gallery · n=1 wall times · 180 W
Every clip, with its wall time
Scene is finished clip length; generate is ComfyUI wall from prompt to saved file. Single observations at 180 W, 8 steps, turbo 0.75 — planning numbers, published medians need repeats.