NVIDIA Releases NemotronLabs VoiceChat 11B: An Open Full-Duplex Speech-to-Speech Model with ~450 ms Turn-Taking and Live Tool Calling
NVIDIA has launched NemotronLabs VoiceChat 11B, an open 11B end-to-end speech-to-speech mannequin for real-time, full-duplex dialog. Instead of chaining ASR, an LLM, and TTS, it performs streaming speech understanding and speech technology in a single unified community. That removes the multi-model orchestration and API handoffs a cascaded stack requires, and cuts end-to-end latency: measured {smooth} turn-taking latency is 448 ms on Full-Duplex-Bench 1.0. The mannequin listens whereas it speaks, so a consumer can barge in mid-turn and the agent yields, with a take-over fee of 1.00 at 480 ms. It can also be first open full-duplex mannequin to assist instrument calling whereas dialog retains flowing, utilizing a separate output channel for <TOOLCALL> scripts alongside with operator-defined “on-hold” strains that fill the hole whereas an API runs.
Is it deployable?
PARTIAL — deployable at present for pilots, not for manufacturing. Weights and container are each public, and the license is permissive. But NVIDIA workforce states the checkpoint is ‘prepared for analysis functions solely,’ and the repo paperwork actual failure modes: a two-minute audio context ceiling, degradation into non-recoverable gibberish after a number of turns, runaway self-talk after a flip ends, and dropped phrases in consumer transcription.
- Which corporations: any workforce that may allocate one GPU with at the very least 80 GB of VRAM — A100, H100, RTX 6000 Pro, or B200 on x86_64 Linux. That covers AI-native startups, funded scaleups, enterprise R&D and innovation labs, GPU cloud suppliers, and college speech teams. There isn’t any hosted API and no inference supplier at the moment serves the mannequin, so groups with out GPU entry might not consider it.
- Industries: contact facilities and CX platforms, automotive in-cabin assistants, retail and drive-thru ordering, telecom IVR modernization, video games and NPC dialogue, and accessibility tooling.
- Applications: barge-in-capable voice brokers, voice front-ends over inner APIs, live-lookup assistants (climate, pricing, order standing), and duplex latency benchmarking harnesses.
Architecture
The mannequin is a hybrid Mamba/Transformer, assembled from three current NVIDIA parts alongside with one new output path:
- A Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b, which encodes the incoming 16 kHz stream repeatedly.
- The NVIDIA Nemotron Nano v2 LLM spine, which consumes audio tokens and predicts textual content tokens.
- An NVIDIA TTS decoder and codec that predicts audio codes, rendered as 22.05 kHz agent speech.
- A separate output channel devoted to tool-calling scripts.
Outputs embody agent audio, agent textual content, and a working consumer transcription. Training used roughly 550k hours of audio throughout actual and artificial corpora, constructing on SALM-Duplex and Audio Flamingo 3.
Tool calling with out lifeless air
Tool calls are emitted on the facet channel as a <TOOLCALL> block; your code returns leads to a <TOOL_RESPONSE> block. The notable piece is the on-hold message: per instrument, an operator defines a line the agent speaks the second the mannequin generates the textual content triggering the decision, so the dialog doesn’t fall silent whereas an API runs.
Constraints are express. NVIDIA recommends a most of 5 instruments per session, the mannequin can not reliably name a number of instruments concurrently, and the consumer can not interrupt the agent throughout instrument execution. System prompts and instrument responses have to be ASCII-only and TTS-friendly.
Performance
On Full-Duplex-Bench 1.0: {smooth} turn-taking TOR 0.82 at 448 ms, user-interruption TOR 1.00 at 480 ms, and pause-handling TOR of 0.153 (artificial) and 0.255 (Candor), the place decrease is healthier.
On AU Harness BFCL-v3 spoken instrument calling: 58.5% easy, 62.5% a number of, 42.5% parallel, 27.5% parallel-multiple, 89.6% irrelevance, 56.1% common. On Full-Duplex-Bench v3: 82.5% instrument choice, 44.2% argument accuracy, 33% go@1.
NVIDIA studies the mannequin ranks #2 amongst open full-duplex fashions on VoiceBench and #2 amongst open fashions on Full-Duplex-Bench 1.0.