|

Alibaba’s Qwen Team Releases Qwen3.8-Flash-Next: A 125B Multimodal MoE With 6B Active Parameters Previewing the Qwen4 Architecture

ByRicardo

Alibaba’s Qwen workforce has launched Qwen3.8-Flash-Next, an open-weight multimodal Mixture-of-Experts mannequin constructed for price per token. The checkpoint pairs a 125B spine with a 51B N-gram embedding desk and a 4B multi-token prediction module. Only 6B parameters activate per token. The workforce positions it as an early preview of the structure that may underpin Qwen4, the identical function Qwen3-Next performed for Qwen3.5. Four modifications carry the launch: a Gated DeltaNet and Qwen Sparse Attention hybrid, Gated Residual, N-gram Embedding, and the Muon optimizer. Qwen workforce stories coaching price at roughly one-ninth that of Qwen3.7-Plus.

Is it deployable?

Yes however not on a workstation. The FP8 checkpoint is 172.78 GiB and the BF16 checkpoint is 335.28 GiB. Per vLLM recipes, TP2 is the minimal validated FP8 configuration on GB300 and TP4 is really helpful. On an 8×H200 node, use TEP8; plain TP8 is incompatible with the checkpoint’s 128-wide quantization blocks. Sparse activation cuts compute, not storage.

https://qwen.ai/blog?id=qwen3.8-flash-next

What is definitely new

Qwen3.8-Flash-Next pairs a 125B foremost mannequin with 51B N-gram embedding parameters and a 4B multi-token prediction module, totaling 180B on disk. Only 6B parameters activate per token. Four modifications drive this:

  • Hybrid consideration (GDN + QSA): Three of each 4 layers use Gated DeltaNet, a linear-attention layer that compresses historical past right into a fixed-size recurrent state. The fourth layer runs Qwen Sparse Attention (QSA), which makes use of a light-weight indexer to pick context at micro-block granularity fairly than per token. The layer structure is 12 × (3 × GDN → 1 × QSA) throughout 48 layers, with a QSA finances of 512 blocks or 2048 tokens.
  • Gated Residual: The residual stream widens into 4 parallel branches, with an element-wise learn gate and a per-branch scalar write gate, at bottleneck rank 320.
  • N-gram Embedding: A 20,000,000-entry bigram/trigram desk at layer 2 provides capability by way of deterministic lookups. It will be offloaded to host reminiscence with asynchronous prefetch — although offload at present runs solely on NVIDIA gadgets.
  • Training recipe.:The Muon optimizer is utilized alongside AdamW to particular weight classes, with batch-size warmup eradicated and scaling legal guidelines refitted.

The MoE layer carries 512 specialists, activating 10 routed plus 1 shared, at skilled intermediate dimension 640.