BENCHMARKS · PUBLIC ARTIFACTS · WEIGHT MATRIX

Receipts — archive.

This is the historical record. Superseded numbers are preserved here on purpose — current receipts live at fraqtl.ai/benchmark.

Every number fraQtl claims, in one place — measured, retrieval-gated, reproducible. A performance figure is published only if the same run proves retrieval intact; a single missed passkey discards the result.

● R1 · vLLM 9-user flagship ● R2 · model matrix ● R3 · llama.cpp 128K ● R4 · quality ● R5 · memory Artifacts Ship log Archive

● KV · RUNTIME — cache compression measured in live serving (vLLM, llama.cpp) ● HI-FI · WEIGHTS — calibrated weight artifacts with quality receipts (GGUF, KLD, MMLU)

BENCHMARKS

The numbers behind the pilot.

All numbers traceable to the public Hugging Face sidecar repos and the benchmark receipts — and reproducible on your own hardware.

Reproduce these numbers on your hardware → python fraqtl_repro_receipts.py # ships with the runtime repo · rented A100 is enough

Receipts available per lane — maturity varies by lane. The vLLM speed receipt, D1 KV receipt, and Qwen artifact are public today.

R1 · FLAGSHIP · VLLM RUNTIME KV · RUNTIME NEW · JULY 2026

Compressed KV at full speed — and it compounds under load.

The fraQtl runtime reads its packed KV pages straight into the tensor cores — no reconstruction step. In real vLLM serving on one A100-80GB (CUDA graphs on), single-user decode runs at 95–99% of fp16 speed at 8K–32K — and 1.34× faster than fp16 at 128K; under concurrent load the memory advantage becomes a throughput advantage: 2.1× fp16 aggregate tokens/sec at 128K context per user. Two models, one kernel, zero per-model changes. Every number retrieval-verified — never “lossless.”

95–99%

of fp16 decode speed, single user

Mistral-7B · 95% @8K · 98.5% @32K · 1.34× fp16 @128K

9 users

at 128K each, on one A100 — fp16 holds 2

134.1 agg tok/s · 2× fp16 at 4.5× its users · 9/9 needles

2.42×

more KV tokens per GPU

997,200 vs 412,544 pool tokens · Mistral-7B

405/405

needle cells retrieved, exact match

3 models · multi-depth × multi-key grids · all arms

REAL VLLM SERVING · ONE A100-80GB

CUDA GRAPHS ON · TEMPERATURE 0 · FP8 ALWAYS SHOWN

Alone, fraQtl matches fp16 — and beats it at 128K. Under load, the advantage grows.

At one user, weights dominate and everyone is within a few percent. Add users and fp16 runs out of KV memory first, fp8 second — fraQtl keeps serving: parity with fp8 at 1 user grows to 1.4× fp8 at 6 users.

CELL	FRAQTL	FP16	FP8 KV	NEEDLES
Mistral-7B · 8K · batch 1 · decode tok/s	85.7	90.39	90.05	✓
Mistral-7B · 32K · batch 1 · decode tok/s	76.84	77.98	82.81	✓
Qwen3-4B-2507 · 128K · batch 1 · decode tok/s	69.5	52.59	72.7	✓
Mistral-7B · KV pool tokens · one A100	997,200	412,544	825,104	—
Qwen3-4B-2507 · users @128K each · bit-plane V2 (Aug 14)	9	2	4*	✓ 9/9
Qwen3-4B-2507 · aggregate tok/s · bit-plane V2 at 9 users	134.1	66.6	117.6 (peak, at 4)	✓
Qwen3-4B-2507 · users @128K ctx each · V1 layout (Jul 05)	6	2	4*	✓ every user
Qwen3-4B-2507 · aggregate tok/s @128K · V1 clean max (Jul 05)	140.9	66.6	105.0	✓
Qwen3-4B-2507 · 32K · 24 concurrent users · aggregate tok/s	429.7	246.0	432.3	✓ 24/24

JULY V1 LADDER · 128K/USER · EACH CONFIG AT ITS CLEAN MAX · THE PER-LAYOUT BANDWIDTH ROOF

fraQtl

140.9 tok/s · 6 users

fp8 KV

117.6 tok/s · 4 users (receipt-valid)

fp16

66.6 tok/s · 2 users

Prefill at 128K: ~3,950 tok/s — at parity with fp16, faster than fp8. Every user needle-tested at their own depth, every rung of the ladder.

RETRIEVAL VERIFICATION · EVERY ARM, EVERY RECEIPT

Compression is lossy by construction — so a performance number here is published only if the same run proves retrieval is intact. Exact-match passkey grids (7 depths × 3 keys per context) run on every arm of every receipt; a single missed passkey discards the result. Qwen3-4B-2507: 189/189 cells across 8K/32K/128K. Mistral-7B-v0.3: 126/126 across 8K/32K. Llama-3.1-8B: 90/90 — 405/405 total, plus the Aug 14 nine-user run’s 9/9 per-user passkeys at depths spanning 0–100%. Same kernel for all three models, zero per-model kernel changes — a new model’s sidecar builds in under an hour.

Framing: BF16/FP16 is the quality reference; public Q4 is the practical size baseline teams already deploy. Matched Q4_K_M bakeoff in progress — results published when locked. Method: vLLM (real paged serving), CUDA graphs on, prefix caching off, two-pass subtractive decode timing, temperature 0, exact-match passkey retrieval per arm; concurrency cells give every user a full 128K context (130,048-token prompt + 1,024-token decode) with an independent passkey at a distinct depth. fp8 = fp16 weights + fp8-E4M3 KV (FlashInfer) — the strong baseline; weights identical across all arms. fraQtl = rank-protected eigenbasis KV read fragment-native into the tensor cores, zero reconstruction. 128K receipts run with chunked prefill disabled (known perf bug under chunking, fix in progress; stock arms unaffected). A batch-routing bug that dropped needles at batch ≥2 was found 07-03, root-caused, fixed, and fully re-receipted — pre-fix numbers were never published. No 256K claims yet: capacity verified, quality gate open. Qwen3-4B KV pool: 845,136 tokens vs fp16 361,776 (fp8 709,488) — 2.3× capacity. *FP8 user counts show the best receipt-valid rung per the r2 audit (4 users at 117.6 tok/s); higher FP8 rungs did not pass the receipt gate. Aug 14 nine-user cells use bit-plane V2 KV tiering (+45% KV pool vs V1); the July V1-layout receipt (6 users, 140.9 aggregate) remains the per-layout bandwidth roof — V2 trades ~5% aggregate for 3 more users. Per-user decode at 9 users is 14.9 tok/s (fp16: ~33, for its 2 users). All cells reproducible from the public Hugging Face sidecar repos (huggingface.co/fraQtl) with fraqtl_repro_receipts.py; run IDs available on request.

R2 · MODEL MATRIX KV · RUNTIME 3 ARCHITECTURES · ONE KERNEL

One kernel, every model — zero per-model changes.

Each model gets a sidecar from the factory (<1h, weights untouched) and runs on the identical membrane kernel. Same command, same recipe — three architectures, one pattern: ~2.4× fp16 capacity, weight-floor parity at short context, and the speed win grows with context.

MODEL	KV CAPACITY VS FP16	VS FP8	8K DECODE	32K DECODE	128K DECODE	NEEDLE CELLS
Qwen3-4B-Instruct-2507 2025 · native 256K	2.39×	1.21×	92% fp16	97% fp16	1.34× fp16 · 95% fp8	189/189
Llama-3.1-8B-Instruct native 128K	2.35×	1.20×	96% fp16	97% fp16	1.23× fp16 · 1.00× fp8	90/90
Mistral-7B-v0.3 native 32K	2.42×	1.21×	95% fp16	99% fp16	n/a — collapses at 128K in all arms incl. fp16	126/126

Needle cells = exact-match passkey grids, 7 depths × 3 keys per context (Llama 5 × 3), all three arms. Capacity = measured KV pool tokens, 8K config. Cells are receipts; blanks are runs not done, not failures.

R3 · LLAMA.CPP 128K KV · RUNTIME

MISTRAL-7B

Standard low-bit KV loses retrieval at long context. fraQtl doesn't.

Mistral-7B at 128K context in llama.cpp. Same model, same prompts — only the KV-cache representation changes. Q8 and Q4 reduce memory but give up needle retrieval; fraQtl D1 reaches memory below Q8 with retrieval intact.

KV CONFIG @ 128K	LIVE VRAM	NIAH	VS FP16
fp16 KV (baseline)	22,657 MiB	5 / 5	—
Q8 KV	15,437 MiB	1 / 5	−31.9%
Q4 KV	11,287 MiB	0 / 5	−50.2%
fraQtl D1	13,261 MiB	5 / 5	−41.5%

      Mistral-7B-Instruct · 128K context · llama.cpp · NIAH = 5-needle retrieval. Live VRAM is measured resident KV-cache footprint.

      fraQtl D1 uses 14.1% less live VRAM than Q8 KV (13,261 vs 15,437 MiB) and keeps 5/5 vs Q8's 1/5 — lower memory and intact retrieval at the same time.

      Retrieval gating has since been strengthened: the vLLM runtime receipts below use 7-depth × 3-key exact-match passkey grids — 189/189 cells on Qwen3-4B-2507, 126/126 on Mistral-7B. See the lead receipt ↑

      RESEARCH BACKING — partial-stack bake-off vs published methods (NIAH, PPL Δ, KIVI / TOVA / SnapKV)
      expand ▾
    

Four numbers vs the nearest published method.

Partial-stack research bake-off (Mistral / Llama, sub-4-bit, C44). The headline customer claim is the D1 llama.cpp receipt above; these are a separate research comparison, reproducible on the benchmark page.

METRIC	FRAQTL	NEAREST PUBLISHED	DIFFERENCE
NIAH retention (1080 trials)	98.5%	97.8% · TOVA	+0.7 pp
PPL Δ at 3.5× compression	+0.012	+0.214 · SnapKV	~18× tighter
NIAH at 128K (Llama 3.1 8B)	100%	0% · KIVI-2	+100 pp
GPUs needed at 128K (35B MoE)	1× A100-80GB	2× A100-80GB · FP16	1× vs 2×

      Sources: Mistral-7B-Instruct C44 bake-off (1080 NIAH trials, 8K–31K, 3 needle types). Llama 3.1 8B C44d/e (128K, 3 needles × 5 trials per depth). Qwen 3.6 35B-A3B VRAM measured on a single A100-80GB.

      Honest note: at matched 4-bit, fraQtl ties KIVI-4 / KVQuant-4 within sampling noise. The wins above are at sub-4-bit, at 128K, vs eviction methods, and on hardware footprint.

      Show full method breakdown · 7 baselines, 7 axes
      +
    

METHOD	MEMORY	QUALITY	CALIBRATION	OVERHEAD	NIAH 1080 TRIALS	PPL Δ V-ONLY
fraQtl (V-only)	3.5× ✓	✓ within noise	✓ 0.3s	no measured (C44)	98.5%	+0.012
fraQtl (V+K)	~3× ✓	✓ within noise	✓ 0.3s	benchmark-dependent	97.8%	—
FP16 baseline	1×	—	—	—	97.0%	0
TOVA	✓	⚠ degrades	✓ none	per-token	97.8%	+0.259
SnapKV	✓	⚠ degrades	✓ none	per-token	94.1%	+0.214
StreamingLLM	✓	✗ poor	✓ none	per-token	37.8%	+0.548
H2O^†	✓	✗ OOM	✓ none	eager attn	0.0%	—
KVQuant-2	✓ 2-bit	⚠ moderate	✗ 5–15 min	custom kernels	—	+0.27
KIVI-2	✓ 2-bit	✗ collapses at 128K	✓ none	per-token	37.8% → 0%@128K	+1.00

NIAH: needle-in-a-haystack retrieval. C44 bake-off, Mistral-7B-Instruct, 8K–31K context, 3 needle types × 360 trials per method.
† H2O: OOM at 16K+ context (eager attention required).

R4 · QUALITY HI-FI · WEIGHTS

Quality matches FP16 within sampling noise.

Qwen 3.6 35B-A3B compressed (single-seed) · vs FP16 baseline

BENCHMARK	FP16	FRAQTL	Δ
MMLU 0-shot	82.40%	82.24%	−0.16 pp
∞Bench Passkey @ 125,315 tokens	30 / 30	30 / 30	parity
HumanEval pass@1	reference	100% retention	within sampling noise
Wikitext-2 PPL Δ	—	+0.033	tight

      Single-seed numbers from the public Qwen 3.6 artifact. 3-seed bit-identical verification on PPL: σ = 0 across seeds 7 / 123 / 2024. Architecture-level 3-seed validation (V/K theorem) on Mistral 7B GQA-4 + Qwen 2.5 3B GQA-2 + Llama 3.2 3B + Phi-3.
    

R5 · MEMORY HI-FI · WEIGHTS

Memory savings break the multi-GPU requirement.

VRAM · Single A100-80GB

QWEN 3.6 35B-A3B

FP16 OOMs at 64K. fraQtl runs 128K with 28 GB headroom.

Same model. Same hardware. 8× the context. The difference between needing 1 GPU and needing 2.

CONTEXT	FP16 BASELINE	FRAQTL	HARDWARE
16K	~71 GB	25.6 GB	Both fit on 1× A100-80GB
64K	82.9 GB → OOM	36.8 GB	FP16 needs 2 GPUs; fraQtl 1
128K	85+ GB	51.7 GB	FP16 needs 2 GPUs; fraQtl 1 · 28 GB free

FP16 64K = 82.86 GB measured (OOMs on 80 GB ceiling). 16K and 128K FP16 figures are model + KV-cache extrapolations — conservative lower bounds.
fraQtl numbers measured on a single A100-80GB with the public 35B artifact at huggingface.co/fraQtl/Qwen3.6-35B-A3B-compressed.

PUBLIC ARTIFACTS

Everything above is downloadable. Nothing is a demo.

The receipts on this page trace to public repos on huggingface.co/fraQtl — KV sidecar kits with the runtime and repro script, and calibrated Hi-Fi GGUFs with KLD receipts, in two collections.

qwen3-4b-instruct-2507-kv-sidecars KV KIT

KV sidecars behind the 9-user flagship · 189/189 grids · ships with fraqtl_repro_receipts.py

mistral-7b-instruct-v0.3-kv-sidecars KV KIT

2.4× fp16 KV capacity · 126/126 grids · independently reproduced Jul 3

fraqtl-sm80-runtime RUNTIME

The vLLM runtime wheel (A100/SM80) + repro script — free for evaluation

Gemma-4-E2B-it · Gemma-4-26B-A4B-it Hi-Fi HI-FI GGUF

Calibrated Hi-Fi GGUFs with KLD receipts vs bf16 — the phone-demo lineage

Qwen3.6-35B-A3B Hi-Fi · compressed HI-FI GGUF

35B MoE at 23.8 GB · 128K on one A100 · MMLU within 0.16pp of FP16

Collections: Hi-Fi Quants · KV-Cache Receipts ORG

Two curated collections — calibrated GGUFs with KLD receipts, and the KV receipts lineage

RELEASES SHIP LOG · LATEST FIRST

Compressed models, ready to load.

Everything shipped, dated, latest first — runtime integrations, receipts, and public artifacts on HuggingFace. Reproducible numbers throughout.

2026 · AUG 14 RECEIPT

Nine 128K users on one A100 (bit-plane V2 KV) — 134.1 aggregate tok/s, 2× fp16 throughput at 4.5× its users, 9/9 per-user needles, +45% KV pool Lead receipt ↓

2026 · JUL 22 HI-FI GGUF

Gemma-4 Hi-Fi GGUFs: E2B-it + 26B-A4B-it — calibrated GGUFs with KLD receipts, in the Hi-Fi Quants collection HuggingFace →

2026 · JUL 10 RECEIPT

Third architecture receipted: Llama-3.1-8B — 2.35× fp16 KV capacity, 1.23× fp16 decode @128K, 90/90 needle grids — model matrix complete, three architectures on one unchanged kernel See the matrix ↓

2026 · JUL 07 KV · REPRO KIT

KV sidecar repro kit public — Mistral-7B + Qwen3-4B sidecar repos + SM80 runtime wheel; fraqtl_repro_receipts.py reproduces every cell on a rented A100 HuggingFace →

2026 · JUL 05 RECEIPT

128K concurrency receipt lands (vLLM) — 6 users at 128K context each on one A100 — 140.9 vs fp16’s 66.6 aggregate tok/s (2.1×), every user’s needle retrieved Lead receipt ↓

2026 · JUL 02 RUNTIME

Membrane kernel lands in vLLM (warptile_v32) — fragment-native compressed-KV reader — packed page straight into the tensor cores, zero reconstruction; 95–99% of fp16 decode single-user

2026 · MAY RUNTIME

llama.cpp 128K KV integration (D1) — Mistral-7B at 128K below Q8’s live VRAM with 5/5 retrieval where Q8 drops to 1/5 — the original moat receipt D1 receipt ↓

PUBLIC ARTIFACT 2026 · APR 27

Qwen 3.6 35B-A3B compressed

MoE · 35B params (3B active) · weight-compressed artifact · 23.8 GB on disk

HuggingFace →

MMLU

82.24%

FP16: 82.40% · −0.16pp

∞Bench Passkey

30 / 30

at 125,315 tokens

VRAM @ 128K

51.7 GB

FP16 OOMs · 1× A100

    Next: derotated-K kernel (8 users/A100 target) · 256K quality gate · chunked-prefill fix · subscribe to release updates →
  

ARCHIVE — research-grade bake-offs, cross-arch tables & 2026-H1 receipts (superseded numbers preserved for the record)expand ▾

Lead receipt: Mistral-7B at 128K in llama.cpp — fraQtl D1 KV below Q8 memory with retrieval preserved. Plus the public Qwen 3.6 35B-A3B compressed artifact (MMLU + ∞Bench + VRAM) and the matched-4-bit weight matrix across MHA, GQA-2, GQA-3, and GQA-4. Every number traces to a committed script and raw JSON, reproducible under golden_eval v1.

Headline receipt: Mistral-7B at 128K context in llama.cpp — fraQtl D1 runs at 13,261 MiB live VRAM while keeping 5/5 needle retrieval. That is 41.5% below the fp16 KV baseline and 14.1% below Q8 KV — and at the same long context Q8 KV drops to 1/5 retrieval and Q4 KV to 0/5.
Public artifact: Qwen 3.6 35B-A3B compressed runs 128K context on 1× A100-80GB at 51.7 GB VRAM, MMLU 82.24% (FP16: 82.40%), ∞Bench passkey 30/30 at 125,315 tokens.
Weight matrix: matched-4-bit results across 4 architecture families using the same core recipe (K_{GATE_UP} = K_DOWN = 256, INT3 + sign correction), with validation reported per model. GOLDEN_EVAL_V1 · WikiText-2 64×512 · 256+256 prefix+continuation · PPL + KL(FP16 ‖ compressed)

● Measured vs extrapolated are labeled separately Public artifact Runtime KV Research-grade

D1Long-context KV · Mistral-7B 128K · llama.cpp

Live VRAM and needle-in-a-haystack retrieval at 128K context, measured in llama.cpp. Same model, same prompt set — only the KV-cache representation changes. Standard low-bit KV (Q8, Q4) reduces memory but loses retrieval at this context length; fraQtl D1 reduces memory below Q8 while retrieval stays intact.

KV-cache config	Live VRAM @ 128K	NIAH retrieval	VRAM vs fp16	Retrieval at long context
fp16 KV (baseline)	22,657 MiB	5 / 5	—	full retrieval, full memory
Q8 KV	15,437 MiB	1 / 5	−31.9%	retrieval breaks
Q4 KV	11,287 MiB	0 / 5	−50.2%	retrieval gone
fraQtl D1	13,261 MiB	5 / 5	−41.5%	below Q8 memory, retrieval preserved

NIAH = needle-in-a-haystack retrieval at 128K context (5 needles). Live VRAM is the measured resident footprint reported by llama.cpp for the KV-cache configuration shown.

fraQtl D1 vs Q8 KV: 14.1% lower live VRAM (13,261 vs 15,437 MiB) and 5/5 vs 1/5 retrieval — lower memory and intact retrieval at the same time.

The clean claim: at 128K, fraQtl D1 holds full 5/5 retrieval at 13,261 MiB — 41.5% under fp16 KV and 14.1% under Q8 KV. Standard low-bit KV gives up retrieval to reach lower memory; fraQtl reaches lower memory and keeps retrieval. Importance-aware compression, not uniform low-bit quantization.

00Public artifact · Qwen 3.6 35B-A3B

Live on Hugging Face. Loads through standard Transformers workflow. 23.8 GB on disk. MoE: 35B params, 3B active.

Metric	fraQtl compressed	FP16 reference	Δ	Notes
MMLU (5-shot)	82.24%	82.40%	−0.16 pp	14,042 questions, 57 subjects
∞Bench passkey	30 / 30	30 / 30	tied	at 125,315 tokens
VRAM @ 16K context	25.6 GB	~71 GB	−45 GB	both fit on 1× A100-80GB
VRAM @ 64K context	36.8 GB	82.86 GB → OOM	FP16 needs 2× A100	measured OOM on 80 GB ceiling
VRAM @ 128K context	51.7 GB	85+ GB	FP16 needs 2× A100	28 GB headroom on 1× A100
Disk size	23.8 GB	~70 GB	~3× smaller	safetensors, full artifact

VRAM at 16K and 128K for FP16 are model + KV-cache extrapolations — conservative lower bounds. 64K FP16 = 82.86 GB measured (OOMs on 80 GB ceiling).

Weight-compressed artifact. Runtime KV-cache compression is a separate early-access layer and is not stacked on top of this artifact in these numbers.

Reproducible: huggingface.co/fraQtl/Qwen3.6-35B-A3B-compressed · loads through the standard Transformers workflow

01The weight matrix

All Δ PPL and KL are against each row's own FP16 baseline (same eval set). Single-seed unless marked. 3-seed Mistral row is mean ± std.

Model / Architecture	FP16 PPL	bnb NF4 Δ / KL / b/w	AWQ 4-bit Δ / KL / b/w	GPTQ 4-bit Δ / KL / b/w	fraQtl INT3+sign Δ / KL / b/w
Mistral-7B-Instruct-v0.2 (GQA-4)	6.6068	+0.1430 / 0.0274 / 4.50 (3-seed)	+0.1590 ±0.0001 / 0.0293 / 4.12 (3-seed)	+0.1721 / 0.0436 / 4.12	+0.0504 ±0.0108 / 0.0165 / 3.62 (3-seed)
Llama-3.2-3B-Instruct (GQA-3)	12.3720	+0.7445 / 0.0644 / 4.50	+0.8015 / 0.0605 / 4.12	BROKEN ¹	+0.4279 / 0.0254 / 3.86
Qwen2.5-3B-Instruct (GQA-2)	8.3597	+0.5091 / 0.0843 / 4.50	+0.5865 / 0.0775 / 4.12 ²	+0.5945 / 0.0991 / 4.12	+0.2241 / 0.0362 / 4.18
Phi-3-mini-4k-instruct (TRUE MHA)	6.5048	+0.5873 / 0.0965 / 4.50	FAILED ³	n/a ⁴	+0.2061 / 0.0466 / 3.86

¹ gptqmodel 1.9.0 predates Llama 3 (released 2024-04): library-version failure, not a GPTQ-method failure. Newer gptqmodel 2.x has its own PyPI-drift instability (blocked original C61). Excluded from ratio/scoreboard math.

² AWQ on Qwen2 needed a nn.Module.__getattr__ monkey-patch inside awq.quantize() to forward Catcher's missing attention_type attribute. Script: notebooks/benchmarks/C60_qwen_awq_patched.py.

³ AWQ on Phi-3 hits a DIFFERENT AutoAWQ bug (KeyError: 'type') than Qwen2's. Not chased per sunk-cost rule; deferred to llm-compressor sprint. Script: C63_phi3_mha_golden.py.

⁴ GPTQ not attempted on Phi-3 this session (focus: MHA universality via bnb/fraQtl).

fraQtl ratios vs each peer

Ratios are stable across architectures. The MHA result (Phi-3, 2.85× vs bnb) lands in the same band as the GQA-4 result (Mistral, 2.84× vs bnb).

Model	Δ vs bnb NF4	Δ vs AWQ 4-bit	Δ vs GPTQ 4-bit	KL vs bnb	KL vs AWQ	KL vs GPTQ
Mistral 7B Instruct	2.84× tighter	3.16× tighter	3.41× tighter	1.66×	1.78×	2.64×
Llama 3.2 3B Instruct	1.74× tighter	1.87× tighter	n/a (GPTQ broken)	2.54×	2.39×	n/a
Qwen 2.5 3B Instruct	2.27× tighter	2.62× tighter	2.65× tighter	2.33×	2.14×	2.74×
Phi-3-mini-4k (MHA)	2.85× tighter	n/a (AWQ failed)	n/a (not attempted)	2.07×	n/a	n/a

Scoreboard — honest counts

BROKEN peer results (Llama GPTQ) and library failures (AWQ on Phi-3) are NOT counted as fraQtl wins — they're counted as "peer didn't run."

Peer	Attempted	Usable	fraQtl wins (matched)
bnb NF4	4/4	4/4	4/4
AWQ 4-bit	4/4	3/4 (Qwen needed patch, Phi-3 failed)	3/3
GPTQ 4-bit	3/4 (Phi-3 not attempted)	2/3 (Llama broken)	2/2

Total: 9/9 matched-bits wins where the peer produced a usable number, across 4 architectures. Same core recipe with per-model validation; no architecture-specific algorithm changes.

02fraQtl config key

fraQtl has TWO configs in the KV lane. Don't conflate them. Read before the KV tables below.

PARTIAL STACK · EXPERIMENT-GRADE

fraQtl V-only 4b

Eigenbasis only + INT4 uniform quantization. No sign correction.

C44b (1080-cell NIAH) · C44c sanity · C44d (128K multi-needle) · C44e (128K depth-sweep)

FULL STACK · MIXED-PRECISION LOW-BIT COMPRESSION

fraQtl V-only INT3 + sign

Eigenbasis + k=16 FP16 protect + LM-INT3 on sacrifice dims + sign correction. 3-seed-validated, pinned-eval.

C38a cross-arch 3-seed · C25q Qwen 3.6 35B-A3B MoE

How to read these: headline numbers use the full stack (INT3+sign) from C38a. Long-context retrieval comparisons use the partial stack (V-only 4b) from the C44 family — both are reported and labeled per row.

03KV cache substrate

The matrix above compresses weights. fraQtl's V-theorem + sign-correction mechanism also applies to KV cache (runtime-dynamic tensor). Cross-architecture KV results follow — per-needle NIAH and 3-seed PPL/KL where measured.

Cross-arch NIAH at matched 4-bit vs KIVI / KVQuant / eviction peers

Config = fraQtl V-only 4b (partial stack, eigenbasis + INT4 uniform, no sign correction). Full-stack INT3+sign numbers in the PPL/KL table below.

Model / Arch	Protocol	fraQtl V-only 4b	fraQtl V+K 4b	KIVI-4	KVQuant-4	KIVI-2	PyramidKV 0.7
Mistral 7B Instruct (GQA-4)	C44b · 1080-cell NIAH 4K–31K	94.4%	93.3%	93.3%	93.3%	37.8%	86.1%
Qwen 2.5 3B Instruct (GQA-2)	C44b · 1080-cell NIAH 4K–31K	99.4%	79.4% ⁵	98.9%	97.8% / 100% (sink0)	1.7%	69.4%
Llama 3.1 8B Instruct (GQA-8)	C44d · 128K multi-needle	93.3%	n/a	93.3%	n/a	0.0%	n/a
Llama 3.1 8B Instruct (GQA-8)	C44e · 128K depth-sweep ⁶	100%	n/a	100%	n/a	0.0%	n/a

⁵ fraQtl V+K on Qwen 3B GQA-2 used Mistral's blind k_protect — per-model calibration pending (see EVAL-PROTOCOL-LOCKED). Not a fraQtl limitation claim.

⁶ notebooks/benchmarks/C44e_llama3_8b_128k_shallow_depth.py (separate from C44e_pyramidkv_bakeoff.py).

The partial-stack V-only 4b configuration already ties KIVI-4 on this retrieval grid; the full-stack INT3+sign numbers in the table below extend the matched-bits margin further.

Cross-arch KV PPL/KL 3-seed pinned

The 3-seed rows bolded below are the headline numbers. Full-stack = INT3 + sign correction.

Model / Arch	Config	Δ PPL (3-seed)	KL (3-seed)	NIAH (3-seed)	Source
Partial stack · C25 family
Mistral 7B Instruct (GQA-4)	V-only k=16 LM-INT3	+0.027 ±0.005	0.00381 ±0.00001	—	C25
Mistral 7B Instruct (GQA-4)	V+K k=16 LM-INT3	+0.062 ±0.001	0.00751 ±0.00013	—	C25
Full stack · INT3 + sign · C38a (3-seed)
Mistral 7B Instruct (GQA-4)	V-only k=16 INT3+sign	+0.0015 ±0.0044	0.00317	—	C38a v2
Mistral 7B Instruct (GQA-4)	K k=16 INT3+sign	+0.0070 ±0.0043	—	—	C38a
Llama 3.2 3B Instruct (GQA-3)	V k=8 INT3+sign	+0.0181 ±0.015	0.00308	99.4% (179/180)	C38a
Llama 3.2 3B Instruct (GQA-3)	K k=16 INT3+sign	+0.0221 ±0.0079	0.00312	100% (180/180)	C38a
Qwen 2.5 3B Instruct (GQA-2)	V k=8 INT3+sign	+0.0542 ±0.0116	0.00362	51.7% ⁷	C38a
Phi-3-mini-128k-instruct (TRUE MHA)	V-only k=16 INT3+sign ⁹	+0.0002 (1-seed)	0.00122	93.3% (56/60)	C62 2026-04-21
Phi-3-mini-128k-instruct (TRUE MHA)	V+K k=16 INT3+sign ⁹	+0.0073 (1-seed)	0.00383	95.0% (57/60 — ties FP16)	C62 2026-04-21
Partial stack · MoE hybrid attention · C25q
Qwen 3.6 35B-A3B (MoE hybrid)	V-only k=16 INT3	+0.045 ±0.011	0.0183 ±0.0023	—	C25q
Qwen 3.6 35B-A3B (MoE hybrid)	V+K k=16 INT3	+0.166 ±0.049	0.0221 ±0.0015	—	C25q

⁷ Qwen 2.5 3B V NIAH: 3-seed mean 51.7%, FP16 baseline 58.3% — small-model short-context NIAH has low ceiling. KLD is 1.5× tighter than INT4 uniform; honest mixed result on GQA-2 V cache, not a clean NIAH win. The sign-correction paradigm's KLD advantage is the cross-arch invariant; PPL/NIAH narrow on GQA-2.

⁹ Phi-3 MHA rows are 1-seed (C62 patched run, source commit 3a0bff7). 1-seed is reported for the ratio comparisons here — 140× tighter Δ PPL vs KIVI-4 and 15.5× vs KVQuant-4; a 3-seed re-run is the basis for any absolute-delta public citation. Raw JSON verified against run output.

Full-stack carries the cross-architecture story onto MHA. Partial-stack on MHA KV previously lost to KIVI-4 (73.3% vs 91.7%); full-stack V-only INT3+sign flips it — +0.0002 Δ PPL (140× tighter than KIVI-4), 93.3% NIAH. V+K ties FP16 at 95.0%.

Caveat: the Phi-3 MHA V and V+K rows are 1-seed; a 3-seed re-run is the basis for any absolute-delta public citation. The ratio comparisons above hold at 1-seed.

04KIVI-2 long-context retrieval failure mode

Combined signal from C44b + C44c sanity + C44d + C44e shallow-depth. Every context length. Every needle type. Every depth position.

Context	Architecture	Grid	KIVI-2 retention
4K–31K	Mistral 7B Instruct (GQA-4)	1080 cells · 3 needles × 4 ctx × 5 depths × 3 trials	37.8%
4K–31K	Qwen 2.5 3B Instruct (GQA-2)	1080 cells	1.7%
128K	Llama 3.1 8B Instruct (GQA-8)	15 cells · 3 needles × 5 trials @ depth 50 (C44d)	0.0%
128K	Llama 3.1 8B Instruct (GQA-8)	9 cells · technical_password × 3 depths × 3 trials (C44e)	0.0%

Pattern: KIVI-2's per-token K quantization collapses at EVERY context length tested, across EVERY needle type, across EVERY depth position. fraQtl V-only 4-bit ties or beats KIVI-4 everywhere. fraQtl 2-bit regimes via V-only k=16 INT3 sit at 3.5× (different Pareto point from KIVI-2's 8×).

05KV substrate — scoreboard

fraQtl V-only 4b (partial stack) vs peer KV families. The full-stack INT3+sign numbers in §03 extend the margin further.

Peer (KV)	Architectures tested	fraQtl partial-stack V-only 4b result
KIVI-2 (per-token K 2-bit)	Mistral GQA-4 / Qwen GQA-2 / Llama 3.1 GQA-8 @128K × 2 protocols / Phi-3 MHA	5/5 catastrophic margins (fraQtl V-only ≥73% vs KIVI-2 ≤38%)
KIVI-4 (per-token K 4-bit)	Mistral / Qwen / Llama 3.1 @128K × 2 / Phi-3 MHA	4/5 non-losses — Mistral + Qwen + 2× Llama 128K non-losses; Phi-3 MHA partial-stack 73.3% vs KIVI-4 91.7% — LOSES (full-stack V-only is 140× tighter Δ PPL vs KIVI-4 — see §03)
KVQuant-4	Mistral / Qwen / Phi-3 MHA	2/3 ties or wins on GQA (Mistral 94.4>93.3 / Qwen 99.4>97.8 / Qwen sink0 100). Phi-3 MHA partial-stack 73.3% vs KVQuant-4 93.3% — LOSES (277× Δ PPL gap); full-stack V-only flips it at 15.5× tighter Δ PPL, V+K ties NIAH at 95.0% — see §03
PyramidKV 0.7	Mistral / Qwen	2/2 wins (+8.3 pp / +30.0 pp aggregate over PyramidKV)
SnapKV / H2O / StreamingLLM / TOVA / ExpectedAttention (C44 original, Mistral only)	Mistral 7B Instruct	fraQtl 100% · TOVA 97.8% (near-tie) · SnapKV 94.1% · ExpectedAttention 53.3% · StreamingLLM 37.8% · H2O 22.0% — 3 of 5 competitors fail catastrophically (<54%)

Source for eviction-peer row: docs/MLP-QUANTIZATION-C14-C17-RESULTS.md L1916–1982. Original "fraQtl is the ONLY method" framing retracted per source L1978.

06Memory lane — runtime GPU memory

Distinct from the weight-compression-ratio matrix above. Runtime GPU memory story on Mistral 7B at 32K context.

Artifact	State	Measurement
FP16 baseline @ 32K	measured	13.91 GB weights · 23.12 GB peak inference · 56.88 GB headroom (A100-80GB)
fraQtl-packed (current loader) @ 32K	measured	9.84 GB on disk (30% smaller) · 14.03 GB weights in memory (loader dequantizes to FP16 by design) · coherent generation · 20.4 tok/s

Disk compression is real and shipped: a 9.84 GB Mistral-7B artifact (30% smaller) with coherent generation verified at 20.4 tok/s. Runtime GPU-memory packing is in development and reported here only where measured.

07Boundaries — what's NOT in this matrix

Hold-the-line

GPTQ on Phi-3: not attempted this session. No implicit claim.
AWQ 3/5-bit: AutoAWQ 0.2.9 is 4-bit only; multi-bit pending llm-compressor scoped image (C50D-AWQ-MULTI-BIT.md).
KV cache compression: different substrate. See C44B-KIVI-KVQUANT-BAKEOFF.md, C44E-PYRAMIDKV-BAKEOFF.md.
MoE matched-protocol: Qwen 3.6 35B-A3B has KV-cache numbers; weight-compression matched-bits vs AWQ/GPTQ on MoE is next-session C50d work.
Throughput / latency / memory: infra agent lane, see C51-THROUGHPUT-BAKEOFF.md.
Multi-seed beyond Mistral: per EVAL-PROTOCOL-LOCKED ratio rule, 1-seed is acceptable for ratio comparisons above 1.5× threshold. All reported ratios exceed 1.5×.

08Artifacts + commit hashes

Every number in this matrix traces to a committed script + raw JSON on the fraqtl-hf-cache Modal volume.

Scripts, raw JSONs, commit hashes · per row

Row	Script	Raw JSON (fraqtl-hf-cache:fraqtl-results/)	Commit hash(es)
Mistral 3-seed (fraQtl, bnb, AWQ)	C60_golden_mistral_instruct.py	c60_golden_mistralai_Mistral-7B-Instruct-v02_seed{42,123,2024}.json	1209261, 9038743, d3af566
Mistral GPTQ 1-seed	C64_gptq_pinned.py via modal_run_gptq.py	c64_gptq_pinned_mistralai_Mistral-7B-Instruct-v02_seed42.json	17b563f
Llama 3B golden (fraQtl, bnb, AWQ) 1-seed	C60_golden_mistral_instruct.py (env MODEL=…)	c60_golden_meta-llama_Llama-32-3B-Instruct_seed42.json	1209261
Llama 3B GPTQ BROKEN 1-seed	C64_gptq_pinned.py (env MODEL=…)	c64_gptq_pinned_meta-llama_Llama-32-3B-Instruct_seed42.json	cc24b1d
Qwen 3B golden (fraQtl, bnb) 1-seed	C60_golden_mistral_instruct.py (env MODEL=…)	c60_golden_Qwen_Qwen25-3B-Instruct_seed42.json	1209261
Qwen 3B AWQ (Catcher-patched) 1-seed	C60_qwen_awq_patched.py	c60_qwen_awq_patched_seed42.json	19bea7f
Qwen 3B GPTQ 1-seed	C64_gptq_pinned.py (env MODEL=…)	c64_gptq_pinned_Qwen_Qwen25-3B-Instruct_seed42.json	cc24b1d
Phi-3 MHA golden (fraQtl, bnb) 1-seed	C63_phi3_mha_golden.py	c63_phi3_mha_golden_seed42.json	cc24b1d
C44b KIVI/KVQuant 1080-cell (Mistral + Qwen)	C44b_kivi_kvquant_bakeoff.py	c44b_{mistral,qwen3b}_full_seed42.json	1209261
C44e PyramidKV 1080-cell (Mistral + Qwen)	C44e_pyramidkv_bakeoff.py	c44e_pyramidkv_{mistral,qwen3b}_seed42.json	3b5e8a8, c1388b8
C44d Llama 3.1 8B 128K multi-needle NIAH	C44d_llama3_8b_128k_3needle.py	c44d_llama3_8b_128k_3needle_seed42.json (volume)	see docs/C44D-MULTI-NEEDLE-128K.md
C44e-shallow Llama 3.1 8B 128K depth sweep	C44e_llama3_8b_128k_shallow_depth.py	c44e_llama3_8b_128k_shallow_depth_seed42.json (volume)	see docs/C44E-SHALLOW-DEPTH-128K.md
C62 Phi-3 MHA full-stack KV (V-only + V+K INT3+sign, 1-seed)	notebooks/benchmarks/C62_phi3_fullstack.py	c62_phi3_mha_seed42.json (8 configs, partial + full-stack side-by-side)	3a0bff7 · 527afad (matrix mirror)
Memory @ 32K (FP16 + fraQtl-packed)	fraqtl/docs/MEASURED-MEMORY-32K.md	nvidia-smi trace @ A100-80GB	f5db558
PackedLinear scaffold + sanity test	fraqtl/src/fraqtl/packed_linear.py · experiments/packed_linear_sanity.py	0.04% mean-rel error vs nn.Linear	19e27c3

One compression principle. Multiple architectures.

Try the public Qwen 3.6 35B-A3B compressed artifact, or pilot fraQtl on your own model stack.

Hugging Face Request Pilot KV Cache Explainer

Receipt-backed Every number traces to a committed script and raw JSON (source data commit 3a0bff7).
Protocol: golden_eval v1 · WikiText-2 test 64×512 · 256+256 prefix+continuation.
Reproduce: github.com/fraqtl · HF: huggingface.co/fraqtl · Questions: samuel@fraqtl.ai