Validating Distributed LLM Serving Benchmarks with NVIDIA srt-slurm, SLURM Recipes, Parameter Sweeps, and Pareto Analysis
In this tutorial, we discover NVIDIA’s srt-slurm framework and find out how we use srtctl to transform declarative YAML configurations into reproducible SLURM benchmark workflows for distributed LLM serving. We arrange the undertaking in Google Colab, examine its inside structure, outline a cluster configuration, dry-run built-in and customized recipes, and mannequin a disaggregated prefill-and-decode deployment for DeepSeek-R1. We additionally generate parameter sweeps, work together with the typed Python API, validate expanded configurations, and analyze simulated benchmark outcomes via a throughput-versus-latency Pareto frontier. Although Colab doesn’t present an actual SLURM setting, we use it as a sensible improvement workspace to know, validate, and put together production-grade benchmark recipes earlier than we submit them to an precise GPU cluster.
import os, sys, subprocess, textwrap, json, shutil, importlib
from pathlib import Path
def run(cmd, verify=True, quiet=False):
"""Run a shell command, stream output."""
print(f"n$ {cmd}")
r = subprocess.run(cmd, shell=True, textual content=True, capture_output=True)
out = (r.stdout or "") + (r.stderr or "")
if not quiet:
print(out[-6000:])
if verify and r.returncode != 0:
elevate RuntimeError(f"Command failed ({r.returncode}): {cmd}")
return out
def part(title):
print("n" + "═"*78 + f"n {title}n" + "═"*78)
part("1. Install srt-slurm")
REPO = Path("/content material/srt-slurm") if Path("/content material").exists() else Path.cwd()/"srt-slurm"
if not REPO.exists():
run(f"git clone --depth 1 https://github.com/NVIDIA/srt-slurm.git {REPO}", quiet=True)
run(f"{sys.executable} -m pip set up -q -e {REPO}", quiet=True)
sys.path.insert(0, str(REPO / "src"))
importlib.invalidate_caches()
os.chdir(REPO)
run("srtctl --help")
We put together the Colab setting by importing the required modules and defining reusable helper capabilities for command execution and part formatting. We clone the NVIDIA srt-slurm repository, set up it in editable mode, and expose its supply listing to the lively Python runtime. We then swap to the repository listing and confirm that the srtctl command-line interface is put in accurately.
part("2. Repository structure")
print(textwrap.dedent("""
src/srtctl/
cli/ submit.py (apply/dry-run/preflight/monitor), do_sweep, interactive
core/ schema.py (typed config), sweep.py, slurm.py (sbatch gen),
validation.py, well being.py, topology.py, fingerprint.py
backends/ sglang.py, trtllm.py, vllm.py, mocker.py ← engine adapters
frontends/ Dynamo / router frontends
templates/ Jinja2 → sbatch + orchestrator scripts
recipes/ ready-made benchmarks per platform (gb200-fp4, h100, b200-fp8,
qwen3-32b, dsv4-pro, mocker smoke assessments, ...)
evaluation/ srtlog (log parsers) + Streamlit dashboard (Pareto, latency...)
docs/ sweeps.md, profiling.md, analyzing.md, config-reference.md
"""))
for d in ["recipes", "docs"]:
print(f"{d}/ →", ", ".be a part of(sorted(p.title for p in (REPO/d).iterdir()))[:300])
part("3. Cluster configuration (srtslurm.yaml)")
(REPO/"srtslurm.yaml").write_text(textwrap.dedent("""
cluster: "colab-demo"
default_account: "demo-account"
default_partition: "gpu"
default_time_limit: "01:00:00"
gpus_per_node: 4
use_gpus_per_node_directive: true
use_segment_sbatch_directive: true
containers:
dynamo-sglang: "/containers/dynamo-sglang.sqsh"
lmsysorg+sglang+v0.5.5.post2.sqsh: "/containers/sglang-v0.5.5.sqsh"
model_paths:
deepseek-r1: "/fashions/DeepSeek-R1"
"""))
print((REPO/"srtslurm.yaml").read_text())
We examine the repository construction to know how srtctl organizes its command-line instruments, schemas, backends, templates, recipes, and evaluation parts. We then create an area srtslurm.yaml file containing simulated cluster defaults, container aliases, GPU settings, and mannequin paths. We use this configuration to resolve recipe references in Colab with out requiring entry to an precise SLURM cluster.
part("4. Dry-run: mocker smoke take a look at → generated sbatch script")
run("srtctl dry-run -f recipes/mocker/agg.yaml", verify=False)
part("5. Custom disaggregated recipe (prefill/decode break up)")
(REPO/"my-disagg.yaml").write_text(textwrap.dedent("""
title: "colab-disagg-demo"
mannequin:
path: "deepseek-r1"
container: "lmsysorg+sglang+v0.5.5.post2.sqsh"
precision: "fp8"
sources:
gpu_type: "gb200"
gpus_per_node: 4
prefill_nodes: 1
decode_nodes: 2
prefill_workers: 1
decode_workers: 2
backend:
prefill_environment: { PYTHONUNBUFFERED: "1" }
decode_environment: { PYTHONUNBUFFERED: "1" }
sglang_config:
prefill:
served-model-name: "deepseek-ai/DeepSeek-R1"
model-path: "/mannequin/"
trust-remote-code: true
kv-cache-dtype: "fp8_e4m3"
tensor-parallel-size: 4
disaggregation-mode: "prefill"
decode:
served-model-name: "deepseek-ai/DeepSeek-R1"
model-path: "/mannequin/"
trust-remote-code: true
kv-cache-dtype: "fp8_e4m3"
tensor-parallel-size: 4
disaggregation-mode: "decode"
benchmark:
kind: "sa-bench"
isl: 1024
osl: 1024
concurrencies: [64, 128, 256]
req_rate: "inf"
"""))
run("srtctl dry-run -f my-disagg.yaml", verify=False)
We dry-run the built-in mocker recipe to look at how srtctl validates configurations and generates SLURM submission artifacts with out executing an actual benchmark. We then outline a complicated DeepSeek-R1 recipe that separates prefill and decode workloads throughout impartial node and employee swimming pools. We validate this disaggregated SGLang configuration via one other dry run and examine how the serving parameters are translated into job scripts.
part("6. Parameter sweep (grid search) — dry-run + growth on disk")
run("srtctl dry-run -f examples/example-sweep.yaml", verify=False)
sweep_dirs = sorted((REPO/"dry-runs").glob("example-sweep_sweep_*"))
if sweep_dirs:
newest = sweep_dirs[-1]
print("Per-job configs generated by the sweep expander:")
for p in sorted(newest.rglob("config.yaml")):
print(" ", p.relative_to(REPO))
part("7. Programmatic use of srtctl's Python API")
import yaml
from srtctl.core.config import load_config
from srtctl.core.sweep import generate_sweep_configs, expand_template
from srtctl.core.schema import BenchmarkType, Precision, GpuType
cfg = load_config("my-disagg.yaml")
print(f"Loaded : {cfg.title}")
print(f"Model : {cfg.mannequin.path} ({cfg.mannequin.precision}) on {cfg.sources.gpu_type}")
print(f"Layout : {cfg.sources.prefill_nodes}P + {cfg.sources.decode_nodes}D nodes, "
f"{cfg.sources.gpus_per_node} GPUs/node")
print(f"Bench : {cfg.benchmark.kind} isl={cfg.benchmark.isl} osl={cfg.benchmark.osl} "
f"concurrencies={cfg.benchmark.concurrencies}")
print(f"Enums : benchmarks={[b.value for b in BenchmarkType]}")
print(f" precisions={[p.value for p in Precision]}, gpus={[g.value for g in GpuType]}")
raw_sweep = yaml.safe_load(Path("examples/example-sweep.yaml").read_text())
jobs = generate_sweep_configs(raw_sweep)
print(f"nSweep expands to {len(jobs)} jobs:")
for job_cfg, params in jobs:
pf = job_cfg["backend"]["sglang_config"]["prefill"]
print(f" {params} → chunked-prefill-size={pf['chunked-prefill-size']}, "
f"max-total-tokens={pf['max-total-tokens']}")
print("nTemplate substitution:",
expand_template({"flag": "{x}", "n": "{y}"}, {"x": 4096, "y": 2}))
We execute the instance parameter sweep and examine the person job configurations created from its Cartesian search area. We load our customized recipe via the typed Python API and look at its mannequin, precision, GPU topology, benchmark settings, and supported enumeration values. We additionally programmatically broaden sweep templates and confirm how every parameter mixture impacts the generated backend configuration.
part("8. Analysis: Pareto frontier from (simulated) benchmark outcomes")
import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(0)
def simulate(variant, base_tps, base_itl):
rows = []
tps_gpu = base_tps * c / (c + 90) * rng.uniform(.97, 1.03)
itl = base_itl * (1 + c/220) * rng.uniform(.97, 1.03)
rows.append({"variant": variant, "concurrency": c,
"tok_s_gpu": tps_gpu, "itl_ms": itl})
return rows
outcomes = simulate("chunked=4096", 260, 9.5) + simulate("chunked=8192", 300, 11.5)
print(json.dumps(outcomes[:3], indent=2), "...")
plt.determine(figsize=(8, 5))
for variant in ("chunked=4096", "chunked=8192"):
pts = [(r["itl_ms"], r["tok_s_gpu"], r["concurrency"])
for r in outcomes if r["variant"] == variant]
xs, ys, cs = zip(*pts)
plt.plot(xs, ys, "o-", label=variant)
for x, y, c in pts:
plt.annotate(str(c), (x, y), fontsize=7, xytext=(3, 3),
textcoords="offset factors")
plt.xlabel("Inter-token latency (ms/token) → worse")
plt.ylabel("Throughput (tokens/s/GPU) → higher")
plt.title("Pareto frontier: sweep variants (factors labeled by concurrency)")
plt.legend(); plt.grid(alpha=.3); plt.tight_layout(); plt.present()
We simulate benchmark observations for 2 chunked-prefill variants throughout growing concurrency ranges. We calculate consultant throughput per GPU and inter-token latency values to mannequin the saturation and latency development generally noticed in distributed serving techniques. We then visualize these leads to a Pareto-style plot to check throughput and responsiveness throughout configurations.
part("9. Next steps on an actual cluster")
print(textwrap.dedent("""
make setup ARCH=aarch64|x86_64
srtctl preflight -f my-disagg.yaml
srtctl apply -f my-disagg.yaml
srtctl apply -f sweep.yaml
srtctl monitor
uv run streamlit run evaluation/dashboard/app.py
srtctl diff runA runB
Logs land in outputs/{JOB_ID}/logs/; evaluation/srtlog parses them
(NodeAnalyzer, RunLoader) for programmatic post-processing.
Reproducibility tip (srtctl additionally prints this in dry-run): add an
id: block to your recipe — HF mannequin repo + revision, container
picture URI, and framework variations — so srtctl can confirm the runtime
matches your declaration at job begin.
"""))
print("
Tutorial full.")
We define the manufacturing workflow that we observe when transferring validated recipes from Colab to an actual SLURM-based GPU cluster. We evaluate the instructions used for setting setup, preflight validation, job submission, sweep execution, monitoring, dashboard evaluation, and experiment comparability. We additionally emphasize reproducibility by declaring mannequin revisions, container identities, and framework variations inside every benchmark recipe.
In conclusion, we constructed a whole understanding of how srtctl constructions, validates, expands, and prepares distributed LLM-serving benchmarks for execution on SLURM infrastructure. We moved from fundamental set up and repository inspection to superior disaggregated serving configurations, Cartesian parameter sweeps, programmatic schema entry, and benchmark-result evaluation. We additionally noticed how dry runs expose generated job artifacts and assist us detect configuration issues earlier than costly cluster sources are allotted. With this workflow in place, we are able to confidently switch our validated recipes to an actual NVIDIA GPU cluster, run preflight checks, submit benchmark jobs, monitor execution, evaluate experiment fingerprints, and analyze efficiency trade-offs in a reproducible method.
Check out the Full Code here. Also, be happy to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us
The put up Validating Distributed LLM Serving Benchmarks with NVIDIA srt-slurm, SLURM Recipes, Parameter Sweeps, and Pareto Analysis appeared first on MarkTechPost.
