Soofi Consortium Releases Soofi S 30B-A3B: An Open Hybrid Mamba-Transformer MoE Foundation Model For German And English

A German analysis consortium has printed the pretraining report for Soofi S 30B-A3B. It is an open base mannequin for German and English. Training ran finish to finish on Deutsche Telekom’s Industrial AI Cloud in Munich. Preview weights are on Hugging Face. It is value noting that amongst among the totally open base fashions examined, Soofi S data the best English and German combination scores.

What is Soofi S 30B-A3B?

Soofi S is a Mixture-of-Experts (MoE) hybrid Mamba Transformer basis mannequin. It totals ~31.6B parameters and prompts ~3.2B per token. As a base mannequin, it has no instruction tuning, alignment, or security tuning. The KI Bundesverband coordinates the consortium, funded by the German Federal Ministry for Economic Affairs and Energy. Participants embrace Fraunhofer IAIS, DFKI, TU Darmstadt, ellamind, and Merantix Momentum.

How the structure works?

The effectivity declare begins with the layer stack. The community holds 52 layers. That is 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and 6 Grouped-Query Attention (GQA) layers. Only these 6 GQA layers preserve a KV cache. Each MoE layer holds 128 routed consultants, prompts 6 per token, and provides 2 shared consultants. Other particulars: mannequin dimension 2688, squared ReLU, RMSNorm, and no positional embeddings.

Soofi S adopts the Nemotron 3 Nano reference design with out modification. The analysis crew provides three causes for that alternative. Those are deployability on stacks comparable to vLLM, serving effectivity, and scientific management. Because the spine is fastened, Nemotron 3 Nano turns into an architecture-identical baseline. The information recipe is the one transferring half.