|

Tencent Open-Sources AngelSpec: A Unified Training Framework for MTP and Block-Parallel Speculative Decoding on Hy3 Models

ByRicardo

Tencent has launched AngelSpec, an open-source, torch-native coaching framework for speculative-decoding draft fashions. The launch covers each autoregressive multi-token prediction (MTP) and the block-parallel DFlash household.

Most speculative-decoding work searches for one drafter that scores properly on an averaged benchmark combination. Real serving visitors doesn’t appear like that combination. AngelSpec treats workload heterogeneity as a first-class design constraint, and specializes construction, coaching information, and verification depth round it.

Why one common drafter underperforms

Speculative decoding is lossless. A light-weight drafter proposes a number of future tokens, and the goal mannequin verifies them collectively in a single ahead cross utilizing rejection sampling. Acceleration then relies upon on two issues: what number of draft tokens get accepted, and how lengthy the entire draft–confirm spherical takes.

Those two portions transfer in reverse instructions throughout domains. In high-entropy open-ended dialog, many continuations are semantically legitimate. The goal could decide any certainly one of them, so acceptance decays shortly with proposal depth. Generating and verifying a protracted block wastes compute. Autoregressive MTP drafting suits this regime as a result of it proposes a shorter candidate sequence.

Code and mathematical reasoning behave in a different way. Programming syntax, repeated identifiers, formal expressions, and step-by-step derivations constrain future tokens extra strongly. These workloads create longer predictable spans, which is strictly what block-parallel drafting amortizes properly.

AngelSpec due to this fact ships two complementary drafters, not one compromise. The MTP mannequin is educated on wealthy, various conversation-oriented information. The block-diffusion mannequin is strengthened with code- and mathematics-focused samples.

The MTP path: Training-Time Test and target-model rollout

The unique Hy3 mannequin is educated with a single MTP layer and no recurrent self-conditioned unrolling. At inference the block might be reused recurrently, however the coaching goal by no means ready it for a protracted self-generated chain. Errors accumulate with depth, so the second and third draft positions settle for at considerably decrease charges than the primary.

AngelSpec addresses this prepare–inference mismatch with a shared-parameter, multi-depth scheme. It retains D logical prediction depths however reuses one bodily MTP block. During coaching that block is autoregressively unrolled for D steps, with every prediction fed into the following invocation. Following the Training-Time Test precept of EAGLE-3, depth ok+1 receives the arg max prediction from depth ok as an alternative of the ground-truth token. Parameters are shared, however supervision stays depth-specific: the instructor goal advances one future place at each depth.

Two additional selections carry a lot of the achieve. First, the goal spine and goal language-model head are frozen, and MTP inputs from the spine are indifferent. The drafter improves its proposal distribution with out touching the distribution that verification should protect. Second, coaching makes use of target-model rollout: responses are generated by the frozen goal reasonably than taken from the unique reference corpus. That produces the precise token selections, hidden-state trajectories, and native uncertainty patterns MTP has to approximate at serving time.

The measured impact is concentrated the place it ought to be. At T = 0, imply acceptance strikes from 52.8% to 66.4%, and imply accepted size from 2.58 to 2.99. The first-position price is sort of preserved at 0.799 → 0.814. The deeper positions carry the delta: p3 climbs from 0.290 to 0.706 on GSM8K, and from 0.387 to 0.757 on HumanEval.


DFly: hybrid goal conditioning and a predecessor-conditioned AR head

DFly is the block-diffusion structure. It builds on DFlash with two structural adjustments.

Hybrid target-conditioning spine: DFlash concatenates hidden states from a number of goal layers and transforms them with a completely linked layer, producing one shared context characteristic. That single context is then equipped to each draft layer, which limits layer specialization. DFlare as an alternative learns per-layer fusion weights, giving every draft layer its personal view of the goal hierarchy, however drops DFlash’s realized cross-layer transformation. DFly composes them: the FC department establishes a standard semantic foundation, and the per-layer fusion is utilized as a residual refinement on high. The additional department introduces solely D × T scalar weights, and its softmax coefficients might be precomputed after coaching.

Predecessor-conditioned autoregressive head: A parallel spine predicts every block place from the accepted context solely. It can not see which continuation was truly chosen at earlier draft positions, which produces suffix acceptance decay. DFly locations a small sequential head after the parallel spine, changing position-wise marginal predictions into prefix-conditioned distributions. The costly spine stays absolutely parallel; solely the small head runs left to proper.

Early Performance

On Qwen3-8B, DFly reaches 5.41 common imply accepted size, towards 5.32 for DSpark, 4.57 for DFlash, and 3.24 for MTP. It takes the very best end result on all 5 math and code benchmarks. DSpark stays barely forward on MT-Bench at 3.77 versus 3.67, which is per DFly being positioned for code and math.

On Hy3-A21B, the margin is wider. DFly reaches 4.79 towards 3.69 for DFlash and 3.00 for MTP — relative beneficial properties of 29.8% and 59.7%. It improves each one of many six reported benchmarks.

The cumulative ablation on Hy3-A21B underneath grasping decoding traces the place that comes from: DFlash spine 3.77, DFly spine 4.40, plus Markov head 4.56, and hidden correction 4.60, and code/math information 4.75. The information enlargement provides 700K prompts — 500K code from OpenCodeInstruct and OpenCodeReasoning, 200K math from Big-Math. Any immediate sharing a contiguous 16-token span with an analysis instance is eliminated earlier than response technology.

Inside the framework

AngelSpec is constructed on TorchSpec and extends it in a number of locations. The basis is disaggregated: inference engines run the frozen goal mannequin and stream hidden states by a Mooncake-backed RDMA retailer on to distributed coaching employees. Hidden states are captured inside vLLM employee processes by public vLLM APIs — a speculative hidden-state extraction hook and a customized KV connector — with out forking the engine.

The extensions that matter for this work:

  • TTT rollout unrolled in parallel over the entire sequence, with the causal-prefix-plus-diagonal consideration construction reproduced implicitly through compiled FlexAttention and logsumexp merging. Memory stays near a single causal cross.
  • Long-context coaching with Ulysses sequence parallelism, validated at context lengths as much as 128k tokens. Each native shard carries a D-token halo so depth-shifted supervision stays rank-local.
  • Document-aware sequence packing with three isolation mechanisms — an consideration doc gate, a depth-shift doc gate particular to the MTP path, and document-local place encoding. Cross-document isolation is roofed by unit checks verifying zero consideration leakage.
  • Evaluation server that periodically runs real speculative decoding towards the most recent checkpoint on devoted GPUs, reporting imply accepted size and per-position acceptance as measured by the serving engine itself.
  • Pluggable interfaces at three ranges: targets (runtime vLLM plugin entry level, no supply patch), targets (composed over a shared base, chosen by config), and optimizers (Muon as a drop-in different to AdamW).

Backends are tiered: vLLM is first-class, with SGLang and HuggingFace Transformers supported at neighborhood tier.


Key Takeaways

  • Seven checkpoints ship on Hugging Face and ModelScope, together with no-think and high-think DFly variants.
  • AngelSpec trains six draft architectures — DFly, DFlash, DFlare, Eagle3, DSpark, MTP — behind one config-driven pipeline.
  • DFly lifts imply accepted size on Hy3-A21B to 4.79, versus 3.69 for DFlash and 3.00 for MTP.
  • On HY3-295B-A21B with TP=8, DFly-8 delivers a 1.98–2.40× speedup over autoregressive decoding throughout concurrency 4 to 64.
  • D-cut pushes live-traffic throughput to 981 tok/s at c64, +15.7% over DFly, whereas giving up 2.8% acceptance.

Check out the Paper, GitHub Repo, Documentation, Hugging Face Collection and ModelScope Collection. All credit score for this analysis goes to the researchers of this venture.

The put up Tencent Open-Sources AngelSpec: A Unified Training Framework for MTP and Block-Parallel Speculative Decoding on Hy3 Models appeared first on MarkTechPost.

Similar Posts