Research-Grade EdgeBench Analysis: AI Agent Benchmarking, Leaderboard Analytics, Scaling Laws, and Evaluation Metrics
In this tutorial, we discover EdgeBench as a sensible benchmark for evaluating superior AI brokers throughout various job classes, runtime environments, and interaction-time budgets. We start by downloading the dataset snapshot from Hugging Face, parsing the launched job specs, and analyzing the benchmark taxonomy, execution settings, web necessities, judging logic, and scoring metadata. We then extract the leaderboard information instantly from the repository README, standardize mannequin names, reshape task-level outcomes into an analysis-ready format, and examine efficiency throughout a number of time budgets. Finally, we match log-sigmoid scaling curves, measure category-level rating enhancements, examine the duties with the most important beneficial properties, and research how SForge rescale features rework uncooked analysis outputs into normalized benchmark scores.
!pip -q set up "huggingface_hub>=0.23" pandas numpy scipy matplotlib pyyaml
import os, glob, json, textwrap, warnings
import numpy as np, pandas as pd
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from huggingface_hub import snapshot_download
warnings.filterwarnings("ignore")
pd.set_option("show.max_colwidth", 90)
pd.set_option("show.width", 160)
REPO_ID = "ByteDance-Seed/EdgeBench"
TIME_BUDGETS = [2, 4, 6, 8, 10, 12]
MODELS = ["Claude Opus 4.8", "GPT-5.5", "GPT-5.4", "GLM-5.1", "DS-V4-Pro"]
def banner(t):
print("n" + "=" * 78 + f"n {t}n" + "=" * 78)
def canon_model(title):
n = title.decrease()
if "opus" in n:
return "Claude Opus 4.8"
if "5.5" in n:
return "GPT-5.5"
if "5.4" in n:
return "GPT-5.4"
if "glm" in n:
return "GLM-5.1"
if "ds-v4" in n or "deepseek" in n:
return "DS-V4-Pro"
return title
banner("1. DOWNLOADING DATASET SNAPSHOT")
local_dir = snapshot_download(repo_id=REPO_ID, repo_type="dataset")
print("Snapshot cached at:", local_dir)
We set up the required Python libraries, import the analytical instruments, and configure the pocket book show settings. We outline the dataset repository, interaction-time budgets, mannequin names, and helper features to format output and standardize mannequin labels. We then obtain the whole EdgeBench dataset snapshot from Hugging Face and retailer its native cache path for the rest of the workflow.
banner("2. LOADING TASK SPECIFICATIONS")
def flatten_task(d):
work, decide = d.get("work", {}) or {}, d.get("decide", {}) or {}
rescale = decide.get("rescale", {}) or {}
return {
"task_id": d.get("task_id"),
"title": d.get("title"),
"class": d.get("class"),
"base_image": d.get("base_image"),
"web": d.get("web"),
"game_mode": d.get("game_mode", False),
"cwd": d.get("cwd"),
"n_submit": len(d.get("submit_paths", []) or []),
"submit_paths": ", ".be a part of(d.get("submit_paths", []) or []) or "(interactive)",
"parser": decide.get("parser") or "(sport)",
"score_dir": decide.get("score_direction", "n/a"),
"choice": decide.get("choice"),
"eval_timeout": decide.get("eval_timeout"),
"rescale_kind": rescale.get("variety"),
"rescale": rescale,
"agent_query": work.get("agent_query", "")
}
data = []
for fp in sorted(glob.glob(os.path.be a part of(local_dir, "*.json"))):
strive:
with open(fp) as f:
data.append(flatten_task(json.load(f)))
besides Exception as e:
print(" ! skipped", os.path.basename(fp), "->", e)
df = pd.DataBody(data).dropna(subset=["task_id"]).reset_index(drop=True)
print(f"Loaded {len(df)} job specs.n")
print(df[["task_id", "category", "base_image", "internet", "rescale_kind"]].head(10).to_string(index=False))
banner("3. BENCHMARK TAXONOMY")
print("Tasks per class:n", df["category"].value_counts().to_string(), "n")
print("Runtime (base_image):n", df["base_image"].value_counts().to_string(), "n")
print("Judge rescale sorts:n", df["rescale_kind"].value_counts(dropna=False).to_string(), "n")
print(f"Tasks needing web: {int(df['internet'].sum())} | game_mode duties: {int(df['game_mode'].sum())}")
fig, ax = plt.subplots(1, 2, figsize=(13, 4.2))
df["category"].value_counts().plot.barh(ax=ax[0], coloration="#4C78A8")
ax[0].set_title("Released duties per class (51)")
ax[0].invert_yaxis()
df["base_image"].value_counts().plot.bar(ax=ax[1], coloration="#F58518")
ax[1].set_title("Runtime atmosphere")
ax[1].tick_params(axis="x", rotation=45)
plt.tight_layout()
plt.present()
banner("3b. ANATOMY OF ONE TASK")
s = df.iloc[0]
print(f"task_id: {s.task_id} | class: {s.class} | base_image: {s.base_image}")
print(f"decide parser: {s.parser} | rescale: {s.rescale_kind} -> {s.rescale}")
print("n--- agent_query (truncated) ---")
print(textwrap.fill(s.agent_query[:800], width=96))
We parse each job specification right into a structured desk containing its class, runtime picture, web entry, submission paths, decide configuration, and agent question. We summarize the benchmark taxonomy by counting duties throughout classes, execution environments, rescaling strategies, and sport modes. We additionally visualize these distributions and examine one consultant job to know how an EdgeBench analysis is outlined.
banner("4. PARSING THE LEADERBOARD")
readme = open(os.path.be a part of(local_dir, "README.md"), encoding="utf-8").learn()
def unescape(x):
return x.change("_", "_").change("", "").change("*", "").strip()
def to_float(x):
x = x.change("*", "").strip()
return np.nan if x in ("", "—", "-") else float(x)
def extract_md_tables(md):
tables, cur = [], []
for ln in md.splitlines():
s = ln.strip()
if s.startswith("|"):
cur.append([unescape(c) for c in s.strip("|").split("|")])
elif cur:
tables.append(cur)
cur = []
if cur:
tables.append(cur)
return [[r for r in t if not all(set(c) <= set("-: ") for c in r)] for t in tables if t]
tables = extract_md_tables(readme)
def parse_series(cell):
elements = cell.cut up("/")
if len(elements) != len(TIME_BUDGETS):
return None
strive:
return [to_float(p) for p in parts]
besides ValueError:
return None
long_rows = []
for tbl in tables:
head = tbl[0]
if head and head[0].decrease() == "job" and any("categ" in h.decrease() for h in head):
model_cols = [canon_model(m) for m in head[2:]]
for row in tbl[1:]:
if len(row) != len(head):
proceed
for mname, cell in zip(model_cols, row[2:]):
sequence = parse_series(cell)
if sequence is None:
proceed
for t, sc in zip(TIME_BUDGETS, sequence):
long_rows.append({
"job": row[0],
"class": row[1],
"mannequin": mname,
"hours": t,
"rating": sc
})
scores = pd.DataBody(long_rows)
print(
f"Parsed {scores['task'].nunique()} duties x "
f"{scores['model'].nunique()} fashions x "
f"{len(TIME_BUDGETS)} budgets = {len(scores)} cells."
)
agg_time, teams, cur = [], [], []
for tbl in tables:
head = tbl[0]
if head and "mannequin" in head[0].decrease() and any("@2h" in h for h in head):
cols = head[1:]
for row in tbl[1:]:
if len(row) == len(head):
rec = {"mannequin": canon_model(row[0])}
rec.replace({c: to_float(v) for c, v in zip(cols, row[1:])})
cur.append(rec)
teams.append(cur)
cur = []
agg51 = pd.DataBody(teams[1] if len(teams) > 1 else (teams[0] if teams else []))
if not agg51.empty:
print("nREADME combination (51-task subset):")
print(agg51.to_string(index=False))
We learn the repository README and extract its Markdown tables into structured Python data. We convert the task-level leaderboard right into a tidy dataset containing job, class, mannequin, interplay time, and rating values. We additionally parse the combination 51-task leaderboard desk to match the README abstract with our task-level calculations.
banner("5. LOG-SIGMOID SCALING LAW (match on per-task means -> sturdy)")
def log_sigmoid(t, lo, hello, okay, t0):
return lo + (hello - lo) / (1.0 + np.exp(-k * (np.log(t) - np.log(t0))))
def r2(y, yhat):
ssr = np.nansum((y - yhat) ** 2)
sst = np.nansum((y - np.nanmean(y)) ** 2)
return 1 - ssr / sst if sst > 0 else np.nan
agg = (
scores.groupby(["model", "hours"])["score"]
.imply()
.unstack("hours")
.reindex(index=MODELS)[TIME_BUDGETS]
)
print("Per-task imply by mannequin & hour:n", agg.spherical(2).to_string(), "n")
t_h = np.array(TIME_BUDGETS, float)
t_dense = np.linspace(2, 12, 200)
fig, ax = plt.subplots(figsize=(9, 5.5))
for coloration, mannequin in zip(plt.cm.tab10(np.linspace(0, 1, len(MODELS))), MODELS):
if mannequin not in agg.index:
proceed
y = agg.loc[model].values.astype(float)
strive:
popt, _ = curve_fit(
log_sigmoid,
t_h,
y,
p0=[y[0], y[-1] + 1, 2.0, 4.0],
bounds=([-50, -50, 1e-2, 0.5], [200, 200, 50, 50]),
maxfev=200000,
)
rr = r2(y, log_sigmoid(t_h, *popt))
ax.plot(
t_dense,
log_sigmoid(t_dense, *popt),
coloration=coloration,
lw=2,
label=f"{mannequin} (R²={rr:.3f})",
)
besides Exception as e:
print(" match failed for", mannequin, e)
ax.scatter(t_h, y, coloration=coloration, s=45, zorder=3)
ax.set_xlabel("Interaction time (hours)")
ax.set_ylabel("EdgeBench rating (51-task imply)")
ax.set_title("Log-sigmoid scaling of rating vs. interplay time")
ax.legend(fontsize=8)
ax.grid(alpha=0.3)
plt.tight_layout()
plt.present()
piv = scores.pivot_table(
index=["task", "category"],
columns=["model", "hours"],
values="rating",
aggfunc="imply",
)
high = MODELS[0]
uplift = (
piv[(top, 12)] - piv[(top, 2)]
).groupby("class").imply().sort_values(ascending=False)
print(f"nMean 2h→12h uplift for {high}, by class:n", uplift.spherical(2).to_string())
focus = (
piv[(top, 12)] - piv[(top, 2)]
).sort_values(ascending=False).head(5).index
fig, ax = plt.subplots(figsize=(9, 5))
for job, cat in focus:
ax.plot(
TIME_BUDGETS,
[piv.loc[(task, cat)][(top, h)] for h in TIME_BUDGETS],
marker="o",
label=job[:32],
)
ax.set_title(f"{high}: per-task studying trajectories (largest beneficial properties)")
ax.set_xlabel("Interaction time (hours)")
ax.set_ylabel("Task rating")
ax.legend(fontsize=8)
ax.grid(alpha=0.3)
plt.tight_layout()
plt.present()
We calculate the imply rating for every mannequin at each interaction-time price range and match a log-sigmoid scaling curve to the ensuing trajectories. We consider the goodness of every match utilizing the coefficient of willpower and visualize each the noticed scores and fitted curves. We then measure category-level rating beneficial properties and plot the person duties that profit most from longer interplay time.
banner("6. SForge SCORING 'RESCALE' FUNCTIONS")
def rescale_linear(uncooked, decrease, higher):
return float(np.clip((uncooked - decrease) / (higher - decrease) * 100.0, 0, 100))
for variety in ["linear", "piecewise_max"]:
ex = df[df["rescale_kind"] == variety]
if ex.empty:
proceed
row = ex.iloc[0]
rs = row["rescale"]
print(f"n[{kind}] instance: {row.task_id}n params: {rs}")
if variety == "linear" and "decrease" in rs and "higher" in rs:
for uncooked in [rs["lower"], (rs["lower"] + rs["upper"]) / 2, rs["upper"]]:
print(
f" uncooked={uncooked:>12.2f} -> "
f"{rescale_linear(uncooked, rs['lower'], rs['upper']):6.2f}"
)
else:
xs = [rs["baseline"], rs["rank30"], rs["rank1"], rs["super_anchor"]]
ys = [0.0, 70.0, 99.0, 100.0]
for uncooked in xs:
print(
f" uncooked={uncooked:>16.1f} -> "
f"{float(np.interp(uncooked, xs, ys)):6.2f} (illustrative)"
)
banner("DONE")
print("Real analysis runs by way of the SForge two-container harness:")
print(" https://github.com/ByteDance-Seed/EdgeBench | https://bytedance-seed.github.io/EdgeBench/")
We study the scoring rescale configurations utilized by the SForge judging system and implement the linear normalization system. We display how uncooked scores map to normalized benchmark scores for each linear and piecewise-style configurations. We end the tutorial by printing the official EdgeBench and SForge sources required for operating full evaluations.
In conclusion, we constructed a whole analytical workflow for understanding each the construction and the reported outcomes of EdgeBench. We moved past merely viewing a leaderboard by connecting job specs, runtime configurations, scoring guidelines, mannequin efficiency, and interaction-time scaling inside a single reproducible Colab pipeline. We additionally recognized the place extra interplay time yields the best efficiency enhancements and visualized how totally different fashions scale throughout the launched benchmark duties. It gives a technically grounded basis for decoding EdgeBench outcomes, evaluating agent capabilities, and making ready for deeper analysis utilizing the complete SForge execution harness.
Check out the Full Code here. Also, be at liberty to observe us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to associate with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us
The put up Research-Grade EdgeBench Analysis: AI Agent Benchmarking, Leaderboard Analytics, Scaling Laws, and Evaluation Metrics appeared first on MarkTechPost.
