Four Impossible Things Fell in Two Days. The Fifth Was Real.

A field note from a forty-eight-hour sprint through inherited impossibility: “unsupported,” “unusable,” “blocked,” and “brutally slow” all dismantled by measurement on one all-Intel workstation — and the two walls that held, which is the part that makes the rest trustworthy. The first half of this note is about how to work when your map is made of labels. The second half is the case, with the numbers.

A label is a conclusion with the evidence removed

Every technical field accumulates labels. “That backend is slow.” “That chip can’t run this workload.” “That flag doesn’t work on Windows.” Labels are compression: someone, somewhere, once did the work, reached a conclusion, and shipped the conclusion without the evidence. Then the conclusion outlived its evidence. The driver updated. The compiler grew a new pass. The benchmark that earned the label was run with a build flag left at its default.

The dangerous thing about a label is not that it’s wrong. Most labels are right, most of the time — that’s why they exist. The dangerous thing is that a label gives you no way to tell whether it’s still right, because the evidence isn’t attached. A label is a battery with no charge indicator.

Over two days in late August, my lab — which is me plus a set of AI agents working under a standing experimental discipline — ran headlong into five of these. Four collapsed under measurement. One held, and the way it held taught us more than the four that fell. This note is the method first, then the story, then the raw numbers.

Eight habits for tearing at walls

1. Treat every label as a hypothesis with an expiry date. The question is never “is this supported?” It is “what measurement earned this label, on what version, and does that measurement still reproduce?” Twice in these two days, the answer was that the original measurement was real but the conditions had rotted out from under it.

2. Decompose “blocked.” A wall is almost never one wall. When we tore into a single “this workload is unsupported on this chip” label, it turned out to be a missing Python dependency, a file-ownership quirk, a wrong enum string, a compiler ABI version seam, a pattern-matcher that wanted one specific operator form, a partitioning step that silently didn’t engage, a shipped output-boundary bug, a request-pooling tax, debug logging, host profiling, and a power-management default — eleven distinct blockers wearing one label. Several were worth orders of magnitude. You cannot fix a plural problem while you believe it is singular.

3. One edge per lap. Each experiment gets one question and ends at the first thing it proves or breaks. The temptation after a good result is to keep going — harden it, generalize it, chase a second finding in the same run. That is where rigor quietly dies, because your second finding inherits the uncontrolled state your first finding left behind. Find the edge, write it down, reset, go again.

4. The strongest proof predicts the number goes down. Anyone can predict their fix will make things faster; wanting it biases everything downstream. When we needed to prove that a hidden engine mechanism was really engaging, we didn’t predict a speedup — we predicted that forcing the mechanism on, in a regime where it should hurt, would make a specific number worse by a specific amount. It did: 24.5 fell to 7.4. A prediction that costs you something when it lands is worth ten that flatter you.

5. Audit your wins harder than your losses. The single best lap in this whole story is one where an agent overturned its own success. A graph compiled, executed, and produced route-dependent outputs — every surface signal said the mechanism we wanted was active. An audit of the partition plan said otherwise: the runtime had silently fallen back to a dense expansion, executing 512 experts to fake the behavior of 10. Compilation plus plausible output is not mechanism. Make the system show you the mechanism.

6. Let identical outcomes stop you. How do you know when a wall is real? When two structurally different attacks produce byte-identical results. We built two very different graphs — different precision schedules, different source structures, 210 operations versus 157 — and the compiler lowered both to the identical 235 operations with the identical 2.03080-millisecond estimate. That identity is the wall introducing itself. Nothing you do above that layer will move it.

7. Set the price before you shop. Before the expensive next step — in our case, weeks of bespoke low-level runtime work — declare the number that would fund it. Ours was: the compiled expert layer must beat 1.50 milliseconds, or the runtime doesn’t get built. It came in at 2.03. The project died cleanly, by a gate we set when we were still infatuated, instead of slowly, by sunk cost. The gate is a promise you make to your future, dumber self.

8. Retract where you bragged. I published a wrong mechanism in this very sprint — confidently attributed a failure to a hardware memory limit when the real cause was a scheduling default three layers up. The correction went into the same ledger, the same day, marked as a retraction, with the evidence ladder that killed it. A lab whose ledger only ever says “we were right” is not a ledger. It’s marketing.

None of this is exotic. It is mostly the scientific method with a shorter feedback loop and a merciless filing system. What makes it new is who does the running: the agents execute laps around the clock, and the discipline — one edge per lap, signed predictions, predeclared gates — is enforced by the harness they work in, not by anyone’s memory of good intentions. My job is hunches and verdicts. Theirs is receipts.

The box and the question

The machine is an all-Intel workstation: a Core Ultra 9 285K, 128 GB of DDR5, two Arc Pro B70 GPUs with 32 GB each, and — because it’s a modern Intel desktop part — two more compute engines almost everyone ignores: an integrated GPU and a 13-TOPS neural processing unit sharing system memory with the CPU. Production serving runs on the two big GPUs via llama.cpp’s Vulkan backend.

The question that started the sprint: the models I most want to serve are giant mixtures-of-experts — hundreds of expert networks per layer, only ten consulted per token. The weights don’t fit in GPU memory, but only a sliver is touched per token. Could the ignored silicon — the iGPU, the NPU, the shared-memory fabric they sit on — help serve the expert stream? The prevailing labels said no, four different ways.

The first wall was real, which mattered

Day one began by closing a door honestly. The obvious dream on a dual-GPU box is peer-to-peer: cards talking directly across PCIe instead of bouncing through system RAM. We measured it at the API level — zeDeviceCanAccessPeer returned 0 in both directions — and then found the community receipts that explain why no driver update will fix it: on consumer Intel chipsets, the root complex physically does not route peer traffic between separate CPU-attached slots. Researchers who forced the driver’s hand got transfers that reported success while returning nothing but 0xFF bytes. Silicon, not software.

I’m leading with the wall that held because it’s what makes the rest of this note worth reading. A method that only ever produces “the impossible was possible all along!” is a hype machine. The same two days that tore down four labels also confirmed real limits and closed them with evidence. The method doesn’t care which way the answer goes. That’s the point of it.

The receipt that lied

Our own ledger held a damning number: the big MoE model, with its experts parked in system RAM and computed by the CPU, decoded fine but prefilled — ingested prompts — at a catastrophic 11.7 tokens per second. “Deep prompts unusable” had gone onto the board as a conclusion, and plans were being drawn around it.

Then a research pass turned up something buried in the engine: llama.cpp has quietly shipped, since early 2024, a mechanism that streams CPU-resident weights to the GPU and does the math there — but only when a batch has at least 32 tokens (PR #6083). We re-read our receipt. The catastrophic prefill number had been measured on a 22-token prompt. The mechanism that makes deep prompts fast had never armed. We had measured the doorstep and labeled the house.

The kill required two shots, per the discipline. First the signed prediction: lower the engine’s threshold to 16 so the mechanism arms on a 22-token prompt — where streaming an entire expert bank for 22 tokens should be ruinous — and prefill should collapse. It did: 24.5–29 tokens per second fell to 7.4–8.4. Mechanism proven by making things worse on purpose. Then the real measurement: a 512-token prompt, run while production serving stayed live on the same GPUs, hit 32.6–52.5 tokens per second — up to 4.5× the number our plans were being built around. The cliff was never a cliff. It was a first-eval artifact standing on a tiny prompt.

The same sweep caught a second labels-versus-reality gap: --direct-io, a flag we credited for fast model loads, turns out to be a no-op at the Windows file layer — the fast loads came from a different side effect entirely, and the drive, not the I/O mode, was the true ceiling (unbuffered Windows loads exist upstream, but our disk tops out at 3.0 GB/s either way — measured, gate failed, port parked).

The hunch, the free seat, and my lab’s wrong answer

Then came the hunch that drove the rest of the sprint. Everything in this box’s memory hierarchy speaks the same low-level language, and the integrated GPU and NPU read system RAM directly — no PCIe hop. The expert stream is a bandwidth problem, not a compute problem. So: could the integrated GPU execute the experts straight out of DDR5?

The first probe fell over — the scheduler refused to place expert weights on the iGPU, and the agent running the lap attributed it to a shared-memory limit in the iGPU’s matrix kernels. A tidy, technical, wrong answer. The next lap falsified it from three directions in twenty minutes: the same weights ran fine on the iGPU alone, a different target worked in the dual-GPU case, and the failure tracked one variable only — whether a discrete GPU was present. The real cause: llama.cpp drops integrated GPUs from its device list when a discrete GPU exists. The buffers were orphaned, not oversized. One explicit --device flag unlocked everything. The retraction went on the board the same day, struck through, next to the evidence that killed it.

And here’s what tumbled out of the falsification ladder: while proving the iGPU’s kernels worked, we ran our entire production model — a 30B mixture-of-experts — wholly on the integrated GPU. It served at 13 tokens per second with coherent, correct output. A production-grade model, on the graphics silicon that ships inside the CPU, using zero bytes of discrete GPU memory and zero CPU threads. Nobody had bothered to look, because the label said integrated graphics can’t serve.

The full venue matrix — who should compute the experts — came back with a twist worth more than the hunch: the boring CPU path won the idle regime at 23–26 tokens per second, the hybrid split lost to a “hop tax” (ninety-six per-layer engine crossings per token that ate the iGPU’s bandwidth advantage whole), and the iGPU’s 13 survives as the seat you buy when the CPU is busy doing everything else — which, on a box running production plus a fleet of agents, is always.

“If it’s software, write software”

That left the NPU — the strangest silicon on the die, 13 TOPS, sharing DDR5 with everyone. Every published source and our own research pass agreed: mixture-of-experts is unsupported on NPUs. Dynamic expert selection, static-shape compiler, forget it. I pushed back with the sentence that named this campaign: if the blocks are software, we can write software. How do we get to the lower level and unblock this?

Twenty-one laps followed, run by an agent over a day and a night against the released OpenVINO stack. The label decomposed exactly as principle 2 predicts. “Unsupported” turned out to mean: a missing dependency, a read-only file mapping, a wrong logging enum, a compiler ABI mismatch (8.1 vs 8.2), a pattern matcher that accepted only the one-input form of an activation function, a partitioner that silently expanded 512 experts densely instead of routing 10 (caught by the audit in principle 5), a shipped output-boundary bug, and then — once the mechanism was genuinely working — a stack of runtime taxes: request pooling, debug logging, host profiling, power defaults. One compiler property alone, dynamic quantization, was worth 7.52×.

By the end, the routed expert layer had gone from 76.6 milliseconds to 1.906 milliseconds — a 40.19× peel — with the NPU verifiably executing the ten selected experts, matching the CPU reference to a cosine similarity of 0.9999995, and replaying routes exactly. The software stack was unblocked. The label was dead.

And then the campaign stopped anyway — by principle 6 and principle 7, not by fatigue. Two structurally different final graphs lowered to the identical 235 operations and the identical 2.03080-millisecond estimate: the compiler floor, introducing itself. Projected across 48 layers, the NPU’s best case was 10–12.6 tokens per second of expert work — against a CPU lane already measured at 23–26. The predeclared 1.50-millisecond gate that would have funded a bespoke low-level runtime failed. As the campaign log put it: software opened the door; measurement decided not to walk farther through it — yet. The lane closed with explicit reopen conditions (new compiler lowering, new silicon, a batch workload with expert reuse) instead of a vague “someday.”

Then the internet dissolved a fourth label

The epilogue arrived from outside. A community tuning report (llama.cpp discussion #27593) on the same GPU we run showed that the “brutally slow” reputation of a rival backend was, to the tune of 3.72×, a build flag left at its default. Our own verdict against that backend survives on correctness grounds — but the throughput half of it was confounded, and our re-check triggers were amended the same day. The same thread independently measured a multi-token prediction speedup of 3.2× that lands within 3% of a receipt already in our ledger from different hardware — two labs, two platforms, one mechanism corroborated. That’s the flywheel working in both directions: outside evidence auditing our labels, our receipts corroborating theirs.

Four labels down — MoE-on-NPU, the prefill cliff, integrated-GPU serving, the backend’s speed — and every one fell the same way: not to optimism, but to a measurement someone finally re-ran with the evidence attached.

The numbers

Everything below is from the campaign ledgers: timestamped JSONL receipts written by the probe harnesses at run time, 163 records for the NPU campaign alone.

Where should the experts live? The venue matrix for the 30B mixture-of-experts (Qwen3-30B-A3B, Q4_K_M), decoding single-stream, measured beside resident production serving:

Greedy decode, temperature 0, prompt cache disabled, two timed repetitions after an untimed warm-up; llama.cpp Vulkan backend, one serving instance per cell; NPU row projects 48 observed expert layers at 1.906 ms p95 plus nothing else — attention, routing, and hops would only lower it.

The prefill cliff that wasn’t. Same model family, experts CPU-resident, engine’s batch-offload mechanism at its default threshold of 32:

88 GiB mixture-of-experts (512 experts, top-10), expert tensors file-backed in system RAM via tensor overrides; the signed-prediction row is the engagement proof — the number was predicted to fall before the run, and fell.

The NPU runtime peel. Alternating-route p95 per expert layer, one change per lap, correctness gates green throughout:

Intel AI Boost NPU (architecture 3720), OpenVINO 2026.3 host-routed top-10 expert graph, real model weights (1.26 GB layer cache, hash-verified); the two final direct graphs both lowered to 235 operations and 2.03080 ms estimated — the identical-outcome stop.

The corrections ledger. What we retracted, in public, inside 48 hours:

Every row carries a strikethrough-and-replace entry in the same board the original claim was published on, dated, with the evidence ladder inline.

What survived

Two walls held: peer-to-peer between the discrete GPUs (silicon — the consumer root complex does not route it, ever) and the NPU’s single-stream economics (arithmetic — the compiler floor times 48 layers loses to a CPU that’s already there). Both are closed with receipts and reopen conditions, which is a different thing entirely from being closed with a shrug.

The meta-lesson is the one I keep re-learning at larger scale: the phrase “software-blocked” should begin an investigation, not end one — and the investigation needs a discipline that is symmetric about which answer it wants. Four labels fell in two days not because we were optimists, but because the method treats “impossible” and “possible” as the same kind of claim: measurable, decomposable, and due for an audit the moment the evidence goes missing. The agents make the audit cheap enough to actually happen. The predeclared gates make the stopping trustworthy. And the ledger — bragging and retractions in the same file — is what lets me believe my own lab.