Greedy-Exact Speculative Decoding

Autoregressive generation pays one full forward pass of the target model per token. Speculative decoding attacks that cost directly, and Zenith implements the simplest honest variant: greedy-exact speculation.

The mechanism

A small draft model proposes a run of k tokens cheaply. The target model then verifies all k proposals in a single forward pass — the same parallelism it enjoys during training — and accepts the longest prefix that matches what it would have produced greedily itself. Rejected positions fall back to the target's own token, and the cycle repeats.

Why "greedy-exact" matters

The general form of speculative decoding (accept/reject sampling against the target distribution) preserves the sampling distribution but is easy to get subtly wrong. The greedy-exact variant makes correctness testable: the speculative output must be byte-identical to plain greedy decoding. Zenith treats that as the contract — an exactness test, not a statistical argument — which turns a research technique into something you can gate in CI.

The payoff

Measured in Zenith's benchmarks, speculation needs up to 3.7× fewer target forward passes at identical output. The catch is that the draft model must actually agree with the target often enough: acceptance rate is the whole economics. A draft that is too weak burns verification passes; one that is too big erodes the savings.

The takeaway

Speculative decoding is a bet that most tokens are easy. The target model's capacity is only really needed at the hard tokens; a small draft handles the easy ones, and the verification pass makes the shortcut safe.