|

NVIDIA Releases Audex (Nemotron-Labs-Audex-30B-A3B): A Unified Audio-Text LLM That Preserves the Text Intelligence of Its Backbone

ByRicardo

NVIDIA has launched Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text giant language mannequin. It understands and generates each audio and speech. It additionally retains the textual content intelligence of its spine. The checkpoints, together with a smaller Audex-2B, are launched underneath a noncommercial license.

Most multimodal fashions pay a textual content tax. When labs add audio or imaginative and prescient output, textual content benchmarks typically drop. NVIDIA analysis group reviews this even for speech-only output fashions. Audex is designed to keep away from that regression.

TL;DR

  • Audex is a single 30B-A3B MoE mannequin that handles audio out and in.
  • Audio inputs enter the textual content embedding house; audio outputs are handled like textual content tokens.
  • Text scores match the spine, with small beneficial properties and small losses per benchmark.
  • Multi-stage SFT plus text-only Cascade RL avoids the ordinary multimodal textual content regression.
  • It is few of the open fashions that generate common audio past speech.

What is Audex?

Audex is a single Mixture-of-Experts (MoE) Transformer decoder. It has 30B complete parameters and 3B activated per token. The spine is Nemotron-Cascade-2-30B-A3B, a text-only MoE LLM. That spine is a hybrid Mamba-Transformer with 52 layers. It makes use of 128 routable specialists and 6 activated specialists.

The design is intentionally easy. Audio inputs are encoded and projected into the textual content embedding house. Text tokens and quantized audio tokens are then handled uniformly throughout technology. There is not any thinker-talker cut up and no stacked cascade of fashions.

Because the design stays easy, Audex runs on commonplace LLM stacks. These embrace Megatron-LM for coaching and vLLM for inference. It helps each an instruct mode and a pondering mode. Context size reaches 1M tokens.

How the Unified Design Works

Three elements sit round the LLM spine:

  • An audio encoder reads sound. Audex makes use of AF-Whisper from Audio Flamingo 3. It shares the Whisper Large-v3 structure and handles 16kHz enter.
  • Two-layer MLP adapters map audio options into the mannequin dimension.
  • An prolonged vocabulary holds discrete audio output tokens. The unique 131,072 tokens develop to 205,312.

Audex makes use of two codecs for output. Speech makes use of X-Codec2 at 50 tokens per second. It applies single-layer finite scalar quantization (FSQ) with a 65,536 codebook.

Non-speech sound makes use of X-Codec at 200 tokens per second. It makes use of 4 flattened residual vector quantization (RVQ) layers. Complex sound will get a bigger token finances than speech. The interactive demo under computes these token counts for any period.