Validating Estimates Using Distributed LLM with NVIDIA srt-slurm, SLURM Recipes, Parameter Sweeps, and Pareto Analysis
In this lesson, we explore srt-slurm for NVIDIA framework and learn how we use srtctl to convert our declarative YAML configuration into a reproducible workflow for the SLURM benchmark for distributed LLM deployment. We set up the project in Google Colab, explore its internal architecture, define the cluster configuration, built-in and custom recipes, and model the deployment of DeepSeek-R1 distributed encoding and decoding. We also generate parameter sweeps, interact with a typed Python API, validate extended configurations, and analyze simulated results for comparison using the Pareto frontier of throughput-versus-latency. Although Colab does not provide an actual SLURM environment, we use it as a development workshop to understand, verify, and optimize recipes for production scale before we move them to a real GPU cluster.
import os, sys, subprocess, textwrap, json, shutil, importlib
from pathlib import Path
def run(cmd, check=True, quiet=False):
"""Run a shell command, stream output."""
print(f"n$ {cmd}")
r = subprocess.run(cmd, shell=True, text=True, capture_output=True)
out = (r.stdout or "") + (r.stderr or "")
if not quiet:
print(out[-6000:])
if check and r.returncode != 0:
raise RuntimeError(f"Command failed ({r.returncode}): {cmd}")
return out
def section(title):
print("n" + "═"*78 + f"n {title}n" + "═"*78)
section("1. Install srt-slurm")
REPO = Path("/content/srt-slurm") if Path("/content").exists() else Path.cwd()/"srt-slurm"
if not REPO.exists():
run(f"git clone --depth 1 {REPO}", quiet=True)
run(f"{sys.executable} -m pip install -q -e {REPO}", quiet=True)
sys.path.insert(0, str(REPO / "src"))
importlib.invalidate_caches()
os.chdir(REPO)
run("srtctl --help")
We configure the Colab environment by importing the necessary modules and defining reusable helper functions for ordering and section formatting. We compile the NVIDIA srt-slurm repository, put it in editable mode, and expose its source directory to the Python runtime. Then we change to the repository directory and make sure that the srtctl command line interface is installed correctly.
section("2. Repository architecture")
print(textwrap.dedent("""
src/srtctl/
cli/ submit.py (apply/dry-run/preflight/monitor), do_sweep, interactive
core/ schema.py (typed config), sweep.py, slurm.py (sbatch gen),
validation.py, health.py, topology.py, fingerprint.py
backends/ sglang.py, trtllm.py, vllm.py, mocker.py ← engine adapters
frontends/ Dynamo / router frontends
templates/ Jinja2 → sbatch + orchestrator scripts
recipes/ ready-made benchmarks per platform (gb200-fp4, h100, b200-fp8,
qwen3-32b, dsv4-pro, mocker smoke tests, ...)
analysis/ srtlog (log parsers) + Streamlit dashboard (Pareto, latency...)
docs/ sweeps.md, profiling.md, analyzing.md, config-reference.md
"""))
for d in ["recipes", "docs"]:
print(f"{d}/ →", ", ".join(sorted(p.name for p in (REPO/d).iterdir()))[:300])
section("3. Cluster configuration (srtslurm.yaml)")
(REPO/"srtslurm.yaml").write_text(textwrap.dedent("""
cluster: "colab-demo"
default_account: "demo-account"
default_partition: "gpu"
default_time_limit: "01:00:00"
gpus_per_node: 4
use_gpus_per_node_directive: true
use_segment_sbatch_directive: true
containers:
dynamo-sglang: "/containers/dynamo-sglang.sqsh"
lmsysorg+sglang+v0.5.5.post2.sqsh: "/containers/sglang-v0.5.5.sqsh"
model_paths:
deepseek-r1: "/models/DeepSeek-R1"
"""))
print((REPO/"srtslurm.yaml").read_text())
We examine the repository structure to understand how srtctl organizes its command-line tools, schemas, backends, templates, recipes, and analysis components. We then create a local file srtslurm.yaml that contains the simulated cluster defaults, container aliases, GPU settings, and model methods. We use this configuration to resolve recipe references in Colab without needing access to the actual SLURM cluster.
section("4. Dry-run: mocker smoke test → generated sbatch script")
run("srtctl dry-run -f recipes/mocker/agg.yaml", check=False)
section("5. Custom disaggregated recipe (prefill/decode split)")
(REPO/"my-disagg.yaml").write_text(textwrap.dedent("""
name: "colab-disagg-demo"
model:
path: "deepseek-r1"
container: "lmsysorg+sglang+v0.5.5.post2.sqsh"
precision: "fp8"
resources:
gpu_type: "gb200"
gpus_per_node: 4
prefill_nodes: 1
decode_nodes: 2
prefill_workers: 1
decode_workers: 2
backend:
prefill_environment: { PYTHONUNBUFFERED: "1" }
decode_environment: { PYTHONUNBUFFERED: "1" }
sglang_config:
prefill:
served-model-name: "deepseek-ai/DeepSeek-R1"
model-path: "/model/"
trust-remote-code: true
kv-cache-dtype: "fp8_e4m3"
tensor-parallel-size: 4
disaggregation-mode: "prefill"
decode:
served-model-name: "deepseek-ai/DeepSeek-R1"
model-path: "/model/"
trust-remote-code: true
kv-cache-dtype: "fp8_e4m3"
tensor-parallel-size: 4
disaggregation-mode: "decode"
benchmark:
type: "sa-bench"
isl: 1024
osl: 1024
concurrencies: [64, 128, 256]
req_rate: "inf"
"""))
run("srtctl dry-run -f my-disagg.yaml", check=False)
We create a built-in mock recipe to test how srtctl validates the configuration and produces artifacts for SLURM deployments without using a real benchmark. We then describe an improved recipe for DeepSeek-R1 that divides the prefill and determines the workload across independent nodes and worker pools. We verify this configuration of SGLang in another way and test how the input parameters are translated into function scripts.
section("6. Parameter sweep (grid search) — dry-run + expansion on disk")
run("srtctl dry-run -f examples/example-sweep.yaml", check=False)
sweep_dirs = sorted((REPO/"dry-runs").glob("example-sweep_sweep_*"))
if sweep_dirs:
latest = sweep_dirs[-1]
print("Per-job configs generated by the sweep expander:")
for p in sorted(latest.rglob("config.yaml")):
print(" ", p.relative_to(REPO))
section("7. Programmatic use of srtctl's Python API")
import yaml
from srtctl.core.config import load_config
from srtctl.core.sweep import generate_sweep_configs, expand_template
from srtctl.core.schema import BenchmarkType, Precision, GpuType
cfg = load_config("my-disagg.yaml")
print(f"Loaded : {cfg.name}")
print(f"Model : {cfg.model.path} ({cfg.model.precision}) on {cfg.resources.gpu_type}")
print(f"Layout : {cfg.resources.prefill_nodes}P + {cfg.resources.decode_nodes}D nodes, "
f"{cfg.resources.gpus_per_node} GPUs/node")
print(f"Bench : {cfg.benchmark.type} isl={cfg.benchmark.isl} osl={cfg.benchmark.osl} "
f"concurrencies={cfg.benchmark.concurrencies}")
print(f"Enums : benchmarks={[b.value for b in BenchmarkType]}")
print(f" precisions={[p.value for p in Precision]}, gpus={[g.value for g in GpuType]}")
raw_sweep = yaml.safe_load(Path("examples/example-sweep.yaml").read_text())
jobs = generate_sweep_configs(raw_sweep)
print(f"nSweep expands to {len(jobs)} jobs:")
for job_cfg, params in jobs:
pf = job_cfg["backend"]["sglang_config"]["prefill"]
print(f" {params} → chunked-prefill-size={pf['chunked-prefill-size']}, "
f"max-total-tokens={pf['max-total-tokens']}")
print("nTemplate substitution:",
expand_template({"flag": "{x}", "n": "{y}"}, {"x": 4096, "y": 2}))
We model the parameter sweep and examine the configuration of each function created in its Cartesian search space. We load our custom recipe with a typed Python API and test its model, accuracy, GPU topology, benchmark settings, and supported calculations. We also programmatically extend sweep templates and verify how each parameter combination affects backend optimization.
section("8. Analysis: Pareto frontier from (simulated) benchmark results")
import numpy as np, matplotlib.pyplot as plt
rng = np.random.default_rng(0)
def simulate(variant, base_tps, base_itl):
rows = []
tps_gpu = base_tps * c / (c + 90) * rng.uniform(.97, 1.03)
itl = base_itl * (1 + c/220) * rng.uniform(.97, 1.03)
rows.append({"variant": variant, "concurrency": c,
"tok_s_gpu": tps_gpu, "itl_ms": itl})
return rows
results = simulate("chunked=4096", 260, 9.5) + simulate("chunked=8192", 300, 11.5)
print(json.dumps(results[:3], indent=2), "...")
plt.figure(figsize=(8, 5))
for variant in ("chunked=4096", "chunked=8192"):
pts = [(r["itl_ms"], r["tok_s_gpu"], r["concurrency"])
for r in results if r["variant"] == variant]
xs, ys, cs = zip(*pts)
plt.plot(xs, ys, "o-", label=variant)
for x, y, c in pts:
plt.annotate(str(c), (x, y), fontsize=7, xytext=(3, 3),
textcoords="offset points")
plt.xlabel("Inter-token latency (ms/token) → worse")
plt.ylabel("Throughput (tokens/s/GPU) → better")
plt.title("Pareto frontier: sweep variants (points labeled by concurrency)")
plt.legend(); plt.grid(alpha=.3); plt.tight_layout(); plt.show()
We simulate the benchmark observations of two variations of prefilling processed at all increasing synchronization levels. We calculate the proxy effect for each GPU and the inter-token latency values to reflect the congestion and latency growth commonly seen in distributed computing systems. We then visualize these results in a Pareto-style plot to compare performance and responsiveness across configurations.
section("9. Next steps on a real cluster")
print(textwrap.dedent("""
make setup ARCH=aarch64|x86_64
srtctl preflight -f my-disagg.yaml
srtctl apply -f my-disagg.yaml
srtctl apply -f sweep.yaml
srtctl monitor
uv run streamlit run analysis/dashboard/app.py
srtctl diff runA runB
Logs land in outputs/{JOB_ID}/logs/; analysis/srtlog parses them
(NodeAnalyzer, RunLoader) for programmatic post-processing.
Reproducibility tip (srtctl also prints this in dry-run): add an
identity: block to your recipe — HF model repo + revision, container
image URI, and framework versions — so srtctl can verify the runtime
matches your declaration at job start.
"""))
print("✅ Tutorial complete.")
We describe the manufacturing methodology we follow when transferring validated recipes from Colab to a real SLURM-based GPU cluster. We review the commands used for location setup, pre-flight verification, task deployment, sweeps, monitoring, dashboard analysis, and test comparisons. We also emphasize reproducibility by declaring model updates, container identities, and framework versions within each benchmark recipe.
In conclusion, we have built a comprehensive understanding of how srtctl builds, validates, extends, and configures benchmarks used by LLM for use in a SLURM infrastructure. We moved from basic installation and cache testing to advanced distributed configuration, Cartesian parameter sweeps, structured schema access, and benchmark results analysis. We also saw how dry running exposes generated job artifacts and helps us find maintenance issues before expensive cluster resources are allocated. With this workflow, we can confidently transfer our proven recipes to a real NVIDIA GPU cluster, perform pre-flight testing, submit benchmark jobs, monitor releases, compare test fingerprints, and analyze performance trade-offs in a reproducible manner.
Check it out Full Code here. Also, feel free to follow us Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to Our newspaper. Wait! are you on telegram? now you can join us on telegram too.
Need to work with us on developing your GitHub Repo OR Hug Face Page OR Product Release OR Webinar etc.? contact us
Sana Hassan, a consulting intern at Marktechpost and a dual graduate student at IIT Madras, is passionate about using technology and AI to address real-world challenges. With a deep interest in solving real-world problems, he brings a fresh perspective to the intersection of AI and real-life solutions.
