OpenAI Releases GPT-Realtime-2.1 and GPT-Realtime-2.1-mini for Low-Latency Voice Agents in the API

OpenAI has launched two new Realtime fashions in its API. They are named gpt-realtime-2.1 and gpt-realtime-2.1-mini. Both goal low-latency voice and multimodal experiences. The mini mannequin is the notable a part of this launch. It is a mini reasoning mannequin for realtime voice. It ships at the identical value as the earlier gpt-realtime-mini. OpenAI additionally decreased p95 latency by at the least 25% throughout Realtime voice fashions. That discount comes from improved caching.

What is GPT-Realtime-2.1-mini

gpt-realtime-2.1-mini is a mini reasoning mannequin for realtime voice interactions. It responds to audio and textual content inputs over a dwell connection. OpenAI positions it as the sooner, extra cost-efficient possibility in the lineup.

The Realtime API processes and generates audio by way of a single mannequin. This avoids chaining separate speech-to-text and text-to-speech techniques. That single-model design reduces latency and preserves nuance in speech.

Reasoning is the most important functionality right here. It means the mannequin can suppose internally earlier than it speaks. The mini tier additionally helps software use, or perform calling, by way of the Realtime API. Together these let the mini mannequin plan a step, name your perform, then reply.

The bigger sibling is gpt-realtime-2.1. It updates GPT-Realtime-2 with improved alphanumeric recognition. It additionally improves silence and noise dealing with, and interruption conduct. It helps speech-to-speech with configurable reasoning effort, instruction following, and software use.

A fast method to decide on between them: use gpt-realtime-2.1 whenever you need the strongest realtime reasoning, software use, instruction following, and voice-agent conduct. Use gpt-realtime-2.1-mini whenever you desire a sooner, extra cost-efficient possibility.

Why Reasoning and Tool Use Matter right here

Voice brokers typically stall throughout software calls. The mannequin fires a perform name, then goes silent. Users assume the name dropped and interrupt. That creates partial outcomes and confused dialog state.

Reasoning and a spoken preamble fixes this sample. The mannequin can say ‘I’ll test that order now’ earlier than performing. It retains speaking whereas it really works by way of a request. That conduct retains multi-step voice duties coherent.

Reasoning effort is configurable throughout ranges. Developers can choose minimal, low, medium, excessive, or xhigh. Low is the default and retains latency down for easy turns. Higher effort will increase latency and output token utilization. OpenAI advises beginning low for most manufacturing voice brokers.

The Latency and Caching Improvement

p95 latency is the Ninety fifth-percentile response time. It captures the sluggish tail that customers truly really feel. A minimize of at the least 25% on that tail helps dwell voice. Improved caching drives this discount throughout the Realtime voice fashions.

Caching additionally lowers value, not simply latency. Cached enter tokens are billed at a steep low cost. For gpt-realtime-2.1-mini, cached audio enter drops to $0.30 per 1M. Fresh audio enter prices $10.00 per 1M by comparability. Long classes profit most, as a result of the system immediate caches after the first flip.

Pricing and Comparison

Pricing is per 1M tokens, cut up by textual content, audio, and picture. The mini retains the earlier mini price whereas including reasoning. The desk under lists the printed figures.

The mini audio output price is $20.00 per 1M. The full gpt-realtime-2.1 expenses $64.00 for the identical. That is roughly a 3x hole on audio output. Teams can commerce some functionality for decrease value whereas maintaining reasoning.

Use Cases with Examples

• Customer help triage: A caller experiences a billing error over the telephone. The mini mannequin causes about the problem at low effort. It calls a lookup_account software, then a check_invoice software. It narrates every step so the caller stays knowledgeable.
• Appointment scheduling: A person asks to maneuver a reserving to subsequent Tuesday. The mannequin captures the precise date character by character. It confirms particulars, then calls a reschedule perform. Confirmed values stop software calls on guessed inputs.
• In-app voice assistant: A cell app streams microphone audio over WebRTC. The mini mannequin solutions product questions in one or two quick sentences. Lower value lets the characteristic run at larger quantity.
• Field knowledge seize: A technician asks the agent to log a component quantity. Improved alphanumeric recognition helps seize codes like ‘8-3-5-7-1’. The mannequin reads the worth again for affirmation earlier than performing.

Minimal Implementation

Browser shoppers join over WebRTC. Your server mints a short-lived consumer secret first. The browser then connects on to the Realtime API. Server media pipelines use WebSockets, and telephony makes use of SIP.

First, the server creates an ephemeral consumer secret. Keep your customary API key on the server. The session config units the mannequin, low reasoning effort, and one software.

// Server: mint a short-lived consumer secret (API key stays server-side)
const r = await fetch("https://api.openai.com/v1/realtime/client_secrets", {
  methodology: "POST",
  headers: {
    Authorization: `Bearer ${course of.env.OPENAI_API_KEY}`,
    "Content-Type": "utility/json"
  },
  physique: JSON.stringify({
    session: {
      kind: "realtime",
      mannequin: "gpt-realtime-2.1-mini",
      directions: "You are a help agent. Reply in one or two quick sentences.",
      reasoning: { effort: "low" },
      instruments: [
        {
          type: "function",
          name: "lookup_account",
          description: "Look up a customer account by email.",
          parameters: {
            type: "object",
            properties: { email: { type: "string" } },
            required: ["email"]
          }
        }
      ],
      tool_choice: "auto"
    }
  })
});
const { worth: EPHEMERAL_KEY } = await r.json(); // cross this to the browser

Next, the browser opens a WebRTC peer connection. It provides the microphone monitor and an information channel for occasions. It then posts its SDP supply to the calls endpoint.

// Browser: connect with the Realtime API over WebRTC
// EPHEMERAL_KEY comes out of your server endpoint above.
const computer = new RTCPeerConnection();

const audioEl = doc.createElement("audio");
audioEl.autoplay = true;
computer.ontrack = (e) => { audioEl.srcObject = e.streams[0]; };

const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
computer.addTrack(mic.getTracks()[0]);

const occasions = computer.createDataChannel("oai-events");
occasions.addEventListener("message", (e) => console.log(JSON.parse(e.knowledge)));

const supply = await computer.createOffer();
await computer.setLocalDescription(supply);

const sdp = await fetch("https://api.openai.com/v1/realtime/calls", {
  methodology: "POST",
  physique: supply.sdp,
  headers: {
    Authorization: `Bearer ${EPHEMERAL_KEY}`,
    "Content-Type": "utility/sdp"
  }
});
await computer.setRemoteDescription({ kind: "reply", sdp: await sdp.textual content() });

Start reasoning effort at low, then elevate it solely for more durable duties. Add quick directions that separate onerous guidelines from defaults. Run evals earlier than and after any mannequin migration.

Strengths and Weaknesses

Strengths:

• Reasoning now reaches the low-cost mini tier.
• Pricing matches the prior gpt-realtime-mini price.
• p95 latency is down at the least 25% throughout Realtime voice fashions.
• Configurable reasoning effort trades latency for depth per process.
• Single-model audio pipeline retains conversations pure.

Weaknesses:

• Audio token pricing is difficult to transform into per-call value.
• Higher reasoning effort raises each latency and output tokens.
• Long classes resubmit context and develop enter value with out pruning.
• The mini tier trades some functionality in opposition to the full gpt-realtime-2.1.

Interactive Explainer

p95 latency decreased ≥25% throughout Realtime voice fashions</span>

<div class=”grid”>
<div class=”card full”>
<div class=”mname f”>gpt-realtime-2.1</div>
<div class=”price-total” id=”fullTotal”>$0.00</div>
<div class=”perunit” id=”fullPer”>$0.00 / minute</div>
<div class=”bars” id=”fullBars”></div>
</div>
<div class=”card mini”>
<div class=”mname m”>gpt-realtime-2.1-mini</div>
<div class=”price-total” id=”miniTotal”>$0.00</div>
<div class=”perunit” id=”miniPer”>$0.00 / minute</div>
<div class=”bars” id=”miniBars”></div>
</div>
</div>

<div class=”abstract”>
<div class=”save-big” id=”saveBig”>—</div>
<div class=”save-lbl” id=”saveLbl”>estimated saving with the mini mannequin on this session</div>
</div>

<div class=”lat”>
<div class=”lat-ico”><span id=”latEff”>L</span></div>
<div class=”lat-txt” id=”latTxt”></div>
</div>

<div class=”controls”>
<div class=”ctitle”>Session Configuration</div>

<div class=”crow”>
<label>Session size <b><span id=”vMin”>5</span> min</b></label>
<enter kind=”vary” id=”sMin” min=”1″ max=”20″ worth=”5″ step=”1″>
</div>
<div class=”crow”>
<label>Caller discuss share <b><span id=”vCaller”>60</span>%</b></label>
<enter kind=”vary” id=”sCaller” min=”10″ max=”90″ worth=”60″ step=”5″>
</div>
<div class=”crow”>
<label>System immediate dimension <b><span id=”vPrompt”>6000</span> tokens</b></label>
<enter kind=”vary” id=”sPrompt” min=”500″ max=”22000″ worth=”6000″ step=”500″>
</div>
<div class=”crow”>
<label>Conversation turns <b><span id=”vTurns”>8</span></b></label>
<enter kind=”vary” id=”sTurns” min=”1″ max=”30″ worth=”8″ step=”1″>
</div>

<div class=”crow”>
<label>Reasoning effort</label>
<div class=”seg” id=”segEffort”>
<button data-e=”minimal”>minimal</button>
<button data-e=”low” class=”on”>low</button>
<button data-e=”medium”>medium</button>
<button data-e=”excessive”>excessive</button>
<button data-e=”xhigh”>xhigh</button>
</div>
</div>

<div class=”crow”>
<div class=”toggle” id=”cacheToggle”>
<div class=”swap on” id=”cacheSwitch”><div class=”knob”></div></div>
<div type=”font-size:13px;”>Prompt caching on <span type=”coloration:var(–muted)”>(system immediate caches after flip 1)</span></div>
</div>
</div>
</div>

<p class=”word”>
Illustrative estimate for planning solely. Audio tokens modeled at ~600 tokens per minute of caller speech and ~1,200 tokens per minute of agent speech. Reasoning effort provides modeled text-output tokens per flip. Caching applies the cached text-input price to the system immediate after the first flip. Real value depends upon your precise visitors, software outputs, and context pruning. Verify in opposition to your individual session utilization occasions.
</p>

<div class=”foot”>Interactive constructed for <b>Marktechpost</b> · Rates per OpenAI Realtime API pricing</div>
</div>

<script>
(perform(){
var PRICE = {
full: { tin:4.00, tcache:0.40, tout:24.00, ain:32.00, aout:64.00 },
mini: { tin:0.60, tcache:0.06, tout:2.40, ain:10.00, aout:20.00 }
};
var EFFORT = { minimal:20, low:60, medium:150, excessive:350, xhigh:700 };
var EFFORT_LAT = {
minimal:”lowest latency, minimal inside reasoning”,
low:”low latency — the beneficial default for most voice brokers”,
medium:”extra deliberate; latency rises with output tokens”,
excessive:”deeper planning; larger latency and output tokens”,
xhigh:”most reasoning; highest latency and output tokens”
};
var st = { min:5, caller:60, immediate:6000, turns:8, effort:”low”, cache:true };

perform per(p, tok){ return (tok/1e6)*p; }

perform calc(m){
var callerMin = st.min * (st.caller/100);
var agentMin = st.min * ((100-st.caller)/100);
var audioInTok = callerMin * 600;
var audioOutTok = agentMin * 1200;
var audioIn = per(m.ain, audioInTok);
var audioOut = per(m.aout, audioOutTok);

var textual contentInCost;
if(st.cache){
textual contentInCost = per(m.tin, st.immediate) + per(m.tcache, st.immediate*(st.turns-1));
} else {
textual contentInCost = per(m.tin, st.immediate*st.turns);
}
var reasonTok = EFFORT[st.effort] * st.turns;
var textOut = per(m.tout, reasonTok);

var whole = audioIn + audioOut + textual contentInCost + textOut;
return { whole:whole, audioIn:audioIn, audioOut:audioOut, textual contentIn:textual contentInCost, textOut:textOut };
}

perform cash(v){
if(v >= 1) return “$”+v.toFixed(2);
if(v >= 0.01) return “$”+v.toFixed(3);
return “$”+v.toFixed(4);
}

perform bars(el, r, max, coloration){
var elements = [
[“Audio input”, r.audioIn],
[“Audio output”, r.audioOut],
[“Text input”, r.textIn],
[“Reasoning + text out”, r.textOut]
];
var html = “”;
elements.forEach(perform(p){
var pct = max>0 ? Math.max(2,(p[1]/max)*100) : 0;
html += ‘<div class=”forehead”><div class=”blabel”><span>’+p[0]+'</span><b>’+cash(p[1])+'</b></div>’+
‘<div class=”btrack”><div class=”bfill” type=”width:’+pct+’%;background:’+coloration+'”></div></div></div>’;
});
el.innerHTML = html;
}

perform render(){
var f = calc(PRICE.full), m = calc(PRICE.mini);
var maxComp = Math.max(f.audioIn,f.audioOut,f.textual contentIn,f.textOut,m.audioIn,m.audioOut,m.textual contentIn,m.textOut);

doc.getElementById(“fullTotal”).textual contentContent = cash(f.whole);
doc.getElementById(“miniTotal”).textual contentContent = cash(m.whole);
doc.getElementById(“fullPer”).textual contentContent = cash(f.whole/st.min)+” / minute”;
doc.getElementById(“miniPer”).textual contentContent = cash(m.whole/st.min)+” / minute”;
bars(doc.getElementById(“fullBars”), f, maxComp, “#4a90d9”);
bars(doc.getElementById(“miniBars”), m, maxComp, “#76B900”);

var save = f.whole – m.whole;
var pct = f.whole>0 ? (save/f.whole*100) : 0;
doc.getElementById(“saveBig”).textual contentContent = cash(save)+” (“+pct.toFixed(0)+”% decrease)”;
doc.getElementById(“saveLbl”).textual contentContent =
“estimated saving per session by selecting gpt-realtime-2.1-mini”;

doc.getElementById(“latEff”).textual contentContent = st.effort.charAt(0).toUpperCase();
doc.getElementById(“latTxt”).innerHTML =
“<b>Reasoning effort: “+st.effort+”</b> — “+EFFORT_LAT[st.effort]+
“. Improved caching cuts p95 latency by at the least 25% throughout Realtime voice fashions.”;

resize();
}

perform bind(id, key, disp){
var el = doc.getElementById(id);
el.addEventListener(“enter”, perform(){
st[key] = parseInt(el.worth,10);
if(disp) doc.getElementById(disp).textual contentContent = el.worth;
render();
});
}
bind(“sMin”,”min”,”vMin”);
bind(“sCaller”,”caller”,”vCaller”);
bind(“sPrompt”,”immediate”,”vPrompt”);
bind(“sTurns”,”turns”,”vTurns”);

var seg = doc.getElementById(“segEffort”);
seg.querySelectorAll(“button”).forEach(perform(b){
b.addEventListener(“click on”, perform(){
seg.querySelectorAll(“button”).forEach(perform(x){x.classList.take away(“on”);});
b.classList.add(“on”); st.effort = b.getAttribute(“data-e”); render();
});
});

var ct = doc.getElementById(“cacheToggle”);
ct.addEventListener(“click on”, perform(){
st.cache = !st.cache;
doc.getElementById(“cacheSwitch”).classList.toggle(“on”, st.cache);
render();
});

perform resize(){
attempt {
var h = doc.getElementById(“rtx”).offsetHeight + 40;
dad or mum.postMessage({ kind:”resize-rtx”, top:h }, “*”);
} catch(e){}
}
window.addEventListener(“load”, resize);
render();
})();
</script>
</physique>
</html>
”
type=”width:100%;border:0;top:600px;overflow:hidden;show:block;”
title=”Realtime Voice Cost and Latency Explorer”
loading=”lazy”>

Check out the Technical details here. Also, be happy to comply with us on Twitter and don’t overlook to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The put up OpenAI Releases GPT-Realtime-2.1 and GPT-Realtime-2.1-mini for Low-Latency Voice Agents in the API appeared first on MarkTechPost.