ARC-Decode: Accelerated Decoding with Risk-Bounded Acceptance

Abstract

As larger language models deliver stronger capabilities, their autoregressive inference becomes increasingly expensive. Speculative decoding accelerates generation by letting a fast draft propose tokens that the target model verifies in parallel. Yet under sampling (T > 0), observed speedups consistently lag behind those under greedy decoding, as the classical lossless verification rule tends to over-reject low-risk drafts, leading to lower acceptance rates and limited acceleration. To address this gap, we propose ARC-Decode (Acceptance with Risk Control), a training-free method that augments speculative decoding without extra forward passes. ARC-Decode enables relaxed acceptance by identifying drafts whose acceptance is expected to induce limited local distributional deviation under a calibrated risk criterion based on Jensen–Shannon divergence. It combines confidence-based pre-verification filtering with a risk-bounded acceptance criterion derived from an analytic upper bound on the potential distributional deviation. Integrated into the state-of-the-art EAGLE-3 pipeline, ARC-Decode increases accept length per cycle and reduces verification compute, achieving up to 1.6× end-to-end speedup over EAGLE-3 under sampling with comparable generation quality across the evaluated benchmarks.

Problem

Why does speculative decoding slow down under sampling?

Key Problem. EAGLE-3 exhibits significant degradation in decoding efficiency under sampling, with performance further deteriorating as temperature increases, leading to reduced acceptance length in speculative decoding.

Motivation

Are the rejected tokens under sampling actually harmful?

Many rejected drafts yield highly consistent continuations under multiple surrogate semantic metrics.

Observation 1. Many rejected drafts yield highly consistent continuations under multiple surrogate semantic metrics, suggesting that token-level rejection does not necessarily imply a downstream change.

Counterfactual rollouts reveal substantial recoverable acceptance space.

Observation 2. Counterfactual rollouts reveal substantial recoverable acceptance space: many full-rejection events contain rejected draft candidates that still preserve the correct final answer.

Method

Overview of ARC-Decode: entropy-guided pruning, local shift estimation, and risk-bounded acceptance.

1. Entropy-guided pruning reduces verification cost. Draft nodes are ranked by path probability, target entropy, and depth to form a compact prefix-closed subtree.
2. Local shift estimation finds low-risk relaxation opportunities. For rejected tokens, ARC-Decode combines target logit margin and whitened embedding distance to estimate local distribution shift.
3. Risk-bounded acceptance extends accepted sequences. A calibrated Local Tolerance Score selectively relaxes low-shift rejections while retaining standard posterior acceptance as the floor.

Results

Full-data results on NVIDIA A100 GPUs, temperature=1.0, total_tokens=60, depth=7. Each benchmark reports accept length, throughput (tok/s), and speedup (method / Base). Best in bold.

Quality: HumanEval = pass@10, GSM8K = exact match. Best in bold.

Additional efficiency results on a single NVIDIA A6000 GPU (total_tokens=32, depth=6).

BibTeX

@inproceedings{li2026arcdecode,
  title     = {ARC-Decode: Accelerated Decoding with Risk-Bounded Acceptance},
  author    = {Li, Ying and Wang, Zhaode and Chen, Zhiwen and Lv, Chengfei and Wang, Huan},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year      = {2026}
}