|

Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills

ByRicardo

Anthropic has printed a new plugin evals workflow for Claude Code. The claude plugin eval command runs a plugin towards life like prompts, grades what Claude produced, and compares the outcome with a run the place the plugin will not be loaded. It solutions 3 questions plugin builders couldn’t beforehand measure: does the ability set off, does it survive an edit or a new mannequin, and does it beat a naked mannequin.

Deployable: Yes. It runs on Claude Code v2.1.269 or later towards any listing with a plugin.json or .claude-plugin/plugin.json manifest, or a skills-directory plugin. Every eval run and choose grader is a actual mannequin name billed to your plan or API account.

What a case appears like

An eval suite lives in an evals/ listing contained in the plugin. Each case is a subdirectory holding a immediate.md and a graders/ folder. The immediate physique goes to Claude precisely as written, and @path mentions usually are not expanded. Frontmatter on immediate.md can set max_turns (default 10), timeout_seconds (default 300), mannequin, tags, and allowed_tools.

Graders are markdown information whose frontmatter units a kind, an non-obligatory weight, and an non-obligatory arm. There are 6 sorts. Four value nothing as a result of they’re computed from the transcript and the information on disk: regex, tool_used, tool_order, and file_exists. Two name a choose mannequin and add to the invoice: llm, which scores the reply towards prose standards you write, and baseline, which compares it towards a reference reply.

claude plugin eval init reads the plugin, asks what a good outcome appears like, proposes circumstances and graders, tries them, and writes the information. In CI, --bare <identify> writes a clean template as an alternative.

The quantity that issues is Δ

By default each case runs twice: a with-arm the place the plugin is loaded and a without-arm the place it’s not. Their distinction, Δ, is what the plugin contributed. If a case scores 1.0 in each arms, the plugin will not be why it handed. The docs instance output reveals a single case at WITH 1.00, W/OUT 0.33, Δ +0.67 throughout 6 runs, costing an estimated $0.41 and taking 74 seconds. A grader marked with-only, sometimes tool_used: Skill, is reported as an indicator and excluded from the rating, because the without-arm has no ability to hearth.

Anthropic calls out the most typical first discovering: a Δ close to zero with the tool_used: Skill grader failing, which suggests Claude will not be selecting the ability on pure phrasing. That is the defect claude plugin validate can’t see, as a result of it checks manifest syntax and schema quite than conduct.

Results land beneath evals/outcomes/<timestamp>/report.html with per-grader verdicts and choose votes. Where the account helps it, the report can also be printed to claude.ai except --no-publish is ready.

Cost and CI

A set makes roughly circumstances × runs × arms agent runs, plus 3 quick choose calls per llm or baseline grader per run, and outcomes fluctuate between runs. The documented CI invocation is:

Copy Code
claude plugin eval . 
  --trust-plugin 
  --json outcomes.json 
  --threshold 0.8 
  --model claude-sonnet-5 
  --judge-model claude-haiku-4-5 
  --no-publish 
  --max-cost-usd 20

The runner wants a Claude Code set up and credentials comparable to ANTHROPIC_API_KEY. Without --trust-plugin, an untrusted checkout is refused with exit 1 when there is no such thing as a terminal. Report issues by no means change the exit code, and --json suppresses progress output.

Interactive explainer

Key Takeaways

  • claude plugin eval scores life like prompts with 6 grader sorts; 4 are free, llm and baseline invoice a choose mannequin.
  • Every case runs with and with out the plugin by default; Δ is the one quantity that proves the plugin did the work.
  • A Δ close to zero with a failing tool_used: Skill grader means the ability by no means triggers on pure phrasing.
  • --threshold, --max-cost-usd, and --trust-plugin flip it into a CI gate; usage-limit errors can pretend a regression.
  • Requires Claude Code v2.1.269+; claude plugin eval init writes the primary suite for you.

Check out the Technical details. Also, be at liberty to comply with us on Twitter and don’t neglect to be part of our 150k+ML SubReddit and Subscribe to (*6*). Wait! are you on telegram? now you can join us on telegram as well.

Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so on.? Connect with us

The submit Anthropic Adds Plugin Evals to Claude Code: 6 Grader Types, a No-Plugin Baseline, and a CI Gate for Skills appeared first on MarkTechPost.

Similar Posts