|

Meet Switchyard: A Rust Proxy and Library That Routes and Translates LLM Traffic Across OpenAI and Anthropic APIs

ByRicardo

Teams operating coding brokers hit the identical wall. Claude Code speaks the Anthropic Messages API, Codex CLI speaks OpenAI, and the mannequin a group truly needs to serve sits behind vLLM, NVIDIA NIM, or Ollama. Rewriting the agent is just not an choice, so the interpretation layer has to dwell some other place.

Switchyard is NVIDIA’s reply: a Rust proxy and library for LLM site visitors that routes requests throughout suppliers, interprets between OpenAI and Anthropic codecs, data operational metrics, and exposes typed, composable routing algorithms. It is launched beneath Apache 2.0 with documentation at docs.nvidia.com/nemo/switchyard.

Is it deployable? Yes, however for analysis solely. The binary installs from crates.io and the launcher from PyPI, and it self-hosts anyplace, however NVIDIA labels Switchyard pre-alpha and experimental, warns it isn’t for manufacturing use, and expects the API and algorithms to alter considerably earlier than v1.0.

What Switchyard does

Clients preserve their native API. Switchyard decodes the inbound request into provider-neutral Rust sorts, runs a routing algorithm to choose a backend, re-encodes the request in that backend’s personal wire format, calls it, and interprets the response, together with streaming occasions, again into the form the shopper expects.

The server accepts three inbound codecs: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Any of the three can handle any route, and every configured LLM shopper selects one upstream format of its personal. That decoupling is the purpose: the agent’s API and the backend’s API now not should match.

Three methods to run it

The launcher path targets coding brokers. Install the revealed instrument with uv instrument set up --python 3.12 "nemo-switchyard[cli]", then run switchyard launch claude, switchyard launch codex, or switchyard launch openclaw in opposition to a packaged deployment or your personal TOML file.

The server path installs the standalone proxy with cargo set up --locked switchyard-server, validates a config with --dry-run, and serves on a number and port you select.

The library path makes use of switchyard-libsy, which embeds the routing algorithms in a Rust utility with out proudly owning an HTTP stack. It by no means calls a mannequin itself; the algorithm decides which goal to make use of and fingers each mannequin name again to the caller.

Routing algorithms

A route is one client-visible mannequin ID plus the algorithm behind it. The server helps:

  • passthrough sends each request to 1 goal.
  • random splits site visitors throughout targets utilizing optionally available relative weights, with an optionally available seed that reproduces the choice sequence. This is the A/B and cost-experiment path.
  • llm_classifier calls a classifier goal for a functionality verdict, then routes to a weak or sturdy goal. base_threshold is required; min_confidence, capability_elevated_floor, and session_affinity tune it, and something the choose can’t resolve falls via to the sturdy goal. Setting mode = "escalation" runs each activate the weak tier first and lets a choose resolve whether or not to rerun it on the sturdy tier.
  • stage_router scores tool-result and agent-progress indicators from current turns to choose a succesful or environment friendly goal, avoiding an additional classifier name on most turns.

Strong, weak, succesful, and environment friendly are roles inside a route, not fastened properties of a mannequin. The similar upstream mannequin can serve completely different roles in numerous routes.

Observability

GET /metrics returns Prometheus textual content from the server’s process-wide OpenTelemetry supplier. The households cowl requests, errors, model-call latency, full-turn latency, immediate, completion, cached, cache-creation, and reasoning tokens, and upstream HTTP makes an attempt by final result and code. A tier label carries sturdy or weak for distinguishable classifier selections, and classifier calls are excluded from these households.

The extra fascinating metric is switchyard_routing_overhead_ms, which experiences the algorithm’s run time minus the decision that served the request. Classifier calls usually are not subtracted, so an LLM-classifier route experiences its classification time right here whereas passthrough and random report the sub-millisecond price of selecting a goal. Buckets begin at 0.1 ms. Separately, --routing-log-file appends a JSON report per accomplished response, and GET /v1/routing/session-stats returns per-session name and token totals from that log.

Configuration

A TOML deployment has three layers: llm_clients outline base URL, wire format, credential surroundings variable, and retry coverage; targets bind one upstream mannequin ID to a shopper; routes expose one client-visible mannequin ID and its algorithm. Secrets by no means sit within the file, since api_key_env solely names an surroundings variable. max_retries defaults to 2 and applies to move failures, timeouts, HTTP 408/429, and 5xx responses.

Key Takeaways

  • Switchyard is an Apache-2.0 Rust proxy and library that routes and interprets LLM site visitors.
  • It bridges OpenAI Chat, OpenAI Responses, and Anthropic Messages in each instructions, together with streams.
  • Four route sorts ship: passthrough, random, LLM-classifier, and signal-driven stage router.
  • Prometheus metrics isolate routing overhead from model-call latency, per mannequin and tier.
  • It is pre-alpha and explicitly not for manufacturing, so deal with it as an analysis instrument.

Check out the GitHub Repo and Documentation. Also, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us

The publish Meet Switchyard: A Rust Proxy and Library That Routes and Translates LLM Traffic Across OpenAI and Anthropic APIs appeared first on MarkTechPost.

Similar Posts