‘Speculative decoding is an AI inference optimization technique that accelerates Large Language Models (LLMs) by predicting and verifying multiple tokens at once. It solves the autoregressive bottleneck – where models typically generate text one slow token at a time – yielding up to 2× to 3× faster speeds without altering output quality.” – Speculative decoding – Artificial intelligence

Latency in large language model inference is dominated by the strict sequentiality of autoregressive generation, where each token depends on all previous ones and must be produced in a separate forward pass through a large network.^1,3,15 This creates a throughput ceiling even on powerful hardware, because the model cannot fully exploit parallel compute when constrained to one-token-at-a-time decoding.^1,7 Speculative decoding tackles this bottleneck by restructuring how predictions are proposed and confirmed, turning part of that inherently sequential workload into batched, parallel verification while keeping the underlying probability distribution unchanged.^10,13,16

Autoregressive bottleneck and why it matters

In an autoregressive transformer, inference proceeds in two phases: prefill, where the full input sequence is processed once, and decoding, where the model iteratively appends one token based on the growing context.³ Each new token requires attention over all past tokens plus recomputation of the output head, so the cost scales linearly with sequence length and cannot be parallelised across future positions.^3,15 For a 70B-parameter model, each forward pass carries substantial memory bandwidth and compute cost, and even highly optimised kernels end up under-utilising hardware because they operate on a single position.^1,4,7 Users experience this as slow response times in chat systems, sluggish code completion, and high per-request serving costs, especially for long replies.^8,13,16 The bottleneck becomes more acute when models are offloaded across devices or storage tiers, as each token triggers repeated parameter transfers.^14,17

Draft and verify: practical mechanism

The core operational idea is to introduce a fast mechanism that runs ahead of the main model, proposing several candidate tokens at low cost, and then use the large model to verify these proposals in one batched step.^1,4,8,13,16 In the classic draft-target design, a smaller drafter model, often a distilled version of the main model, generates a short sequence of speculative tokens, typically in the range of 3 to 12 positions.^1,4,8,13 The large target model then processes the original context plus all speculated tokens in parallel, computing probability distributions for each new position.^1,4 Tokens are accepted as long as the target model agrees with the draft at each step; once the first disagreement occurs, remaining draft tokens are discarded and the target model directly samples the next token after the last accepted position.^4,8,13,16 By repeating this cycle, the system converts what would have been K sequential forward passes into a single parallel verification pass plus occasional corrective steps, reducing end-to-end latency by factors typically between 2× and 3× for language tasks.^4,6,8,16

Lossless acceleration: preserving distributions

An important property of modern speculative decoding schemes is that they are lossless: they guarantee that the output distribution over tokens is identical to what would be produced by standard autoregressive sampling from the target model alone.^10,13,16 This is achieved by treating the draft output purely as a proposal and performing exact rejection sampling with respect to the target distribution.^4,7,10 Intuitively, the drafter suggests a path through the probability space; the target model then evaluates that path and accepts the longest prefix for which the draft tokens could have been drawn from its own distribution under the chosen sampling rule.^4,10,13 Once acceptance breaks, the target samples its own next token, rejoining the same stochastic process as vanilla decoding.^4,10 Recent work has formalised this process and shown that even when drafter and target do not share a vocabulary or architecture, carefully designed algorithms can maintain distributional equivalence while delivering speedups up to 2,8×.^2,5 For visual and video autoregressive models, information-theoretic coupling strategies have been introduced to stabilise drafting trajectories and achieve speedups of up to 4,2× for images and 13,6× for video without any change in output quality.^9,12

Mathematical specification and acceptance dynamics

From a probabilistic standpoint, speculative decoding can be seen as sampling from the target model distribution p_T(x_t \mid x_{<t}) using a proposal distribution from the drafter p_D(x_t \mid x_{<t}).^10,13 At each iteration, the drafter proposes a sequence (y_1, \dots, y_K), and the target evaluates the conditional probabilities p_T(y_i \mid x_{<t+i-1}) for i = 1, \dots, K.^4,8 A simple acceptance rule compares p_T and p_D at each position and accepts y_i while p_T(y_i \mid x_{<t+i-1}) \geq p_D(y_i \mid x_{<t+i-1}), rejecting the first token that violates this inequality and discarding the rest.⁴ More sophisticated schemes reweight tokens to ensure exactness relative to p_T.¹⁰ Performance is governed by the acceptance rate \alpha, the expected fraction of draft tokens ultimately accepted, and the speculative length \gamma, the number of tokens proposed per cycle.^8,13 Empirical studies show that latency reductions and throughput gains scale almost linearly with \alpha, with practical systems often targeting \alpha \geq 0,6 and \gamma \geq 5 to achieve 2×-3× speedups in language applications.⁸ If \alpha drops too low, the overhead of repeatedly discarding drafts and recomputing corrections can negate any benefit, making the choice and training of the drafter a central design problem.^8,11

Architectural variants: draft-target and native speculators

The original and still common pattern is the dual-model draft-target architecture, where the drafter is a separate, small model trained to mimic the target on the same data distribution.^1,3,8,13 This scheme is straightforward to integrate with existing LLM deployments but introduces its own trade-offs: the drafter must be fast enough that its forward passes are cheap compared with the saved target passes, and closely aligned so that its proposals enjoy high acceptance rates.^8,18 More recent work has explored speculators embedded directly into the target model, adding multiple lightweight heads that predict tokens several steps ahead using intermediate hidden states.^1,7,21 Approaches such as EAGLE operate at the feature level, extrapolating from the hidden state near the output head and using autoregressive prediction heads to generate speculative sequences without a distinct drafter network.^1,21 This internal speculator design can simplify deployment, avoid cross-model synchronisation, and leverage shared training infrastructure, while still achieving substantial latency improvements.^1,7,21 Meanwhile, alternative decoding schedules, such as big-little decoders mixing small autoregressive and large non-autoregressive passes, demonstrate that speculative-style acceleration can be achieved within broader architectural innovations.⁶

Implementation constraints and workload suitability

Engineering speculative decoding into production systems raises several practical constraints around tokenisation, memory, and serving architecture.^1,13,16 Drafter and target models typically need compatible tokenisers and similar training distributions; mismatched vocabularies historically limited adoption, although newer algorithms have removed this requirement by working at the distribution level rather than exact token matches.^2,5 On the systems side, KV caches storing past key-value pairs are essential: during verification, only the speculative tokens incur new computation, while the original prefix reuses cached activations.^1,13,16 Gains are most pronounced for workloads where many sequential tokens are predictable at low entropy, such as structured code, boilerplate text, or image and video regions with strong local regularities.^9,12,19 In domains with highly unpredictable or adversarial inputs, acceptance rates fall and speculative decoding can even slow down inference if not carefully tuned.^8,16 Offloaded and distributed setups further complicate matters, but recent work has shown that speculative schemes can help amortise parameter transfers and improve overall throughput for remote or tiered storage deployments.^14,17

Debates, limitations, and evolving techniques

Despite strong empirical gains, there are active debates over how broadly speculative decoding should be deployed and what constitutes an optimal implementation.^11,16,20 One line of criticism notes that reliance on a separate drafter model increases system complexity, introduces new failure modes, and may struggle in highly domain-specific settings unless the drafter is retrained on niche data.^8,18,19 Others point out that latency improvements are sensitive to hardware, batch size, and serving patterns; in heavily batched environments or where sequence lengths are short, the relative advantage over well-optimised standard decoding can shrink.^6,16 There is also discussion over the trade-off between lossless methods, which preserve the exact target distribution, and approximate variants that allow controlled deviations to achieve higher speedups.¹¹ Lossless approaches provide stronger guarantees for safety, evaluation, and reproducibility, but may require more sophisticated rejection sampling and coupling machinery.^2,5,9,10 As LLMs expand beyond text into multimodal and generative media, research is exploring how speculative ideas can adapt to continuous outputs and more complex dependency structures, with early results in video suggesting substantial headroom.^9,12

Strategic significance for AI deployment

Speculative decoding matters because it shifts the economics and user experience of large-scale AI systems without demanding new model families or retraining from scratch.^10,13,16,20 By reducing per-token latency and compute cost while preserving output quality, it enables high-parameter models to serve interactive workloads that would otherwise be impractically slow or expensive.^4,6,8,16,20 In enterprise contexts, this translates into lower infrastructure spend per request and the ability to meet strict response-time service levels, particularly for code assistants, document summarisation, and real-time conversational agents.^8,13,16,20 For researchers and platform providers, speculative decoding offers a lever to push model size and capability upward while keeping serving costs under control, effectively widening the feasible design space for foundation models.^1,10,21 Ongoing advances in lossless algorithms, native speculators, and domain-adaptive drafter training suggest that the technique will remain central to inference optimisation, even as other acceleration strategies such as quantisation, pruning, and hardware specialisation evolve alongside it.^6,11,16,20

 

References

1. An Introduction to Speculative Decoding for Reducing Latency in AI … – 2025-09-17 – https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/

2. Accelerating LLM Inference with Lossless Speculative Decoding … – 2025-05-08 – https://icml.cc/virtual/2025/poster/43675

3. [PDF] Decoding Speculative Decoding – ACL Anthology – 2025-04-29 – https://aclanthology.org/2025.naacl-long.328.pdf

4. Faster LLMs: Accelerate Inference with Speculative Decoding – 2025-06-04 – https://www.youtube.com/watch?v=VkWlLSTdHs8

5. Accelerating LLM Inference with Lossless Speculative Decoding… – 2025-05-01 – https://openreview.net/forum?id=vQubr1uBUw&noteId=8I4uskz0rl

6. [PDF] Speculative Decoding with Big Little Decoder – Stat.berkeley.edu – https://www.stat.berkeley.edu/~mmahoney/pubs/neurips-2023-speculative-decoding.pdf

7. A Hitchhiker’s Guide to Speculative Decoding – PyTorch – 2024-05-02 – https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/

8. Get 3× Faster LLM Inference with Speculative Decoding Using the … – 2025-08-08 – https://www.bentoml.com/blog/3x-faster-llm-inference-with-speculative-decoding

9. Your Autoregressive Visual Model is a Natively Multi-Token Predictor – 2026-03-02 – https://openreview.net/forum?id=eSufe2qcFf

10. Looking back at speculative decoding – Google Research – 2024-12-06 – https://research.google/blog/looking-back-at-speculative-decoding/

11. [PDF] Speculative Speculative Decoding – arXiv – https://arxiv.org/pdf/2603.03251

12. Speculative Decoding for Autoregressive Video Generation – arXiv – 2026-04-19 – https://arxiv.org/abs/2604.17397

13. Speculative decoding | LLM Inference Handbook – BentoML – https://bentoml.com/llm/inference-optimization/speculative-decoding

14. Lossless and Training-Free Acceleration for Offloaded LLMs via … – 2025-09-19 – https://neurips.cc/virtual/2025/poster/117367

15. Speculative Decoding Reduces Autoregressive Generation Costs – 2026-05-07 – https://www.linkedin.com/posts/vaishnav-damani-65638517a_why-do-we-need-speculative-decoding-autoregressive-activity-7458114138968436736-pTX1

16. Speculative decoding: how it works & when to use it – Redis – 2026-04-22 – https://redis.io/blog/speculative-decoding-llm/

17. Fast collaborative inference via distributed speculative decoding – https://www.sciencedirect.com/science/article/pii/S2949715925000782

18. Speculative Decoding in LLMs Explained | Avi Chawla posted on … – 2026-05-13 – https://www.linkedin.com/posts/avi-chawla_speculative-decoding-in-llms-explained-visually-activity-7460285369758986240-YRE-

19. r/LocalLLaMA on Reddit: speculative decoding …. is it still used – 2025-12-19 – https://www.reddit.com/r/LocalLLaMA/comments/1pqh7ay/speculative_decoding_is_it_still_used/

20. Speculative decoding: cost-effective AI inferencing – IBM Research – 2024-06-24 – https://research.ibm.com/blog/speculative-decoding?share=0443ca34-3f67-4a79-8cce-45f715e258c6&channel=linkedin&userID=b143aa0f-630b-4514-a76e-f1fba74e4a6f&es_id=e56af33b6a

21. Efficient Speculative Decoding for Llama at Scale – Meta AI – 2025-08-12 – https://ai.meta.com/research/publications/efficient-speculative-decoding-for-llama-at-scale-challenges-and-solutions/