|

KwaiKAT Team Releases KAT-Coder-V2.5: An Agentic Coding Model Trained on 100,000+ Verifiable Repository Environments

ByRicardo

The KwaiKAT Team at Kuaishou has launched the KAT-Coder-V2.5. It is a coding mannequin skilled to function inside actual, executable repositories relatively than emit single-turn code. The served mannequin is accessible by StreamLake. An open-weight variant, KAT-Coder-V2.5-Dev, was launched individually on Hugging Face below Apache-2.0.

AutoBuilder: environments that really run the meant exams

The analysis frames a verifiable activity as a triplet. It wants a exact activity description, an executable repository surroundings, and a set of validation exams. A patch is appropriate provided that it passes all of them.

Tasks are mined from actual pull requests and commits, following the SWE-bench lineage. The merged code change provides a golden patch and the accompanying take a look at change provides a take a look at patch. Raw difficulty textual content is discarded as a specification. Instead, descriptions are regenerated into three components: an issue assertion grounded within the golden patch, necessities derived from the take a look at patch, and interface constraints inferred from each. A readability test then drops something ambiguous, incomplete, underspecified, or internally inconsistent.

AutoBuilder handles the surroundings facet. A construct agent analyzes the repository and writes a configuration script that installs dependencies and runs exams from a clear checkout. A verification agent executes that script in an remoted sandbox.

The acceptance rule is the fascinating half. Verification doesn’t learn exit codes or grep log patterns. It parses structured test-framework output, and accepts an surroundings solely when greater than 90% of anticipated exams are collected and go/fail outcomes reproduce throughout runs. Failures are fed again as structured info for iterative restore.

Combining a preconfigured base surroundings, build-system templates, and a retrievable library of distilled construct recipes raised the development success fee from 16.5% to 57.2%. The result’s over 100,000 verifiable environments spanning 12 languages. Git historical past, commit metadata, and different exploitable traces are stripped so brokers can’t learn the reference answer out of the repo.

Data Scaling Flywheel: filtering on course of

Filtering trajectories by closing take a look at success is deceptive. Some passing runs rely on hard-coding, mechanism bypassing, or test-oriented shortcuts. Some failing runs include beneficial search, localization, and restore conduct.

KwaiKAT addresses each instructions. For close to misses, focused process-level hints point out what to examine or confirm with out revealing the answer. That alone raises the go fee of beforehand zero-pass duties to roughly 20%. Because hinted trajectories include info unavailable at inference, the verified patch is then mounted and a hint-free trajectory is regenerated from the unique activity context. Only samples that go verification, present no trace leakage, and keep per the patch are retained.

For passing runs, rule-based gates take away invalid, unstable, or exploitative trajectories. A scoring stage then charges exploration, localization, pre-edit reasoning, specification constancy, repository conventions, patch minimality, verification high quality, restoration conduct, and honesty.

A 3rd mechanism targets harness overfitting. Tool names, argument conventions, output codecs, and immediate templates are randomized whereas performance is preserved. Because verification is anchored to check outcomes relatively than harness traces, one activity might be re-served below many harness configurations. Realistic perturbations are injected too: lacking dependencies, transient command failures, truncated outputs, and noisy logs.

Explore the pipeline and the numbers

Infrastructure failures capped rewards earlier than algorithmic limits

During KAT-Coder-V2 coaching, sluggish reward curves have been initially blamed on the RL algorithm. However, an audit revealed that ~16% of trajectories failed as a result of sandbox infrastructure points relatively than the mannequin coverage, with boundary misalignments generally emptying observations for ~40 steps and corrupting rewards.

Three infrastructure fixes adopted. First, an early-release picture eviction coverage lowered disk utilization from 95% to 60%, decreasing timeout-induced invalid rollouts from 6–7% to below 1%. Second, correcting surroundings variables throughout distant sandbox initialization stopped system overrides that flipped rewards on 6–7% of samples, chopping errors beneath 1%. Third, the Gateway Server bypassed mainstream chat endpoints—which triggered 40% token drift at a ~200-turn scale by re-applying apply_chat_template and re-tokenizing—and known as /generate immediately to make sure rollout token alignment.

Together, these updates diminished the sandbox suggestions error fee from roughly 16% to beneath 2% and lower coaching collapses by an order of magnitude.

Asymmetric PPO and a three-tier reward

The analysis staff selected PPO with GAE over critic-free trajectory strategies as a result of manufacturing harnesses cut up classes into structurally distinct samples, complicating group baselines.

Using asymmetric actor–critic, the Critic will get privileged coaching context (rewards, exams, protection, patches, metadata, future turns), whereas the Actor sees solely rollout state. The Critic and additional context are discarded at inference.

Rewards are three-tiered: Core Task Scores require all fail_to_pass and pass_to_pass exams to go; Standard Behavior Constraints penalize duplication, unhealthy device calls, and debug remnants; Failed Trajectory Incentives rating file retrieval by way of F2 and provides partial take a look at credit score.

Five specialists fuse by way of Multi-Teacher On-Policy Distillation utilizing reverse KL, an off-policy begin, and drift-aware truncation from Prune-OPD.

Results

Under a unified Claude Code harness, KAT-Coder-V2.5 leads its panel on PinchBench with 94.9, beating Opus 4.8 at 93.5. It locations second on SWE-Bench Pro (65.2 vs 69.2) and the inner KAT Code Bench (53.1 vs 57.3).

However, it lags on Terminal-Bench 2.1, inserting final with 60.7 behind GLM-5.1 (61.8) and Opus 4.8 (84.6). On SciCode, it scores 50.3, matching GLM-5.2.

Notably, the open-weight KAT-Coder-V2.5-Dev is a separate 35B-total / 3B-active MoE post-trained on Qwen3.6-35B-A3B utilizing 127K SFT examples, then RL. Evaluated on a separate in-house protocol, its outcomes should not akin to the principle flagship desk.

Key Takeaways

  • KwaiKAT treats agentic coding as an infrastructure downside, not a model-scale downside.
  • AutoBuilder lifted surroundings building success from 16.5% to 57.2%, yielding 100,000+ verifiable environments throughout 12 languages.
  • A sandbox audit discovered ~16% of RL trajectories failed due to the sandbox, not the coverage; fixes lower that to beneath 2%.
  • KAT-Coder-V2.5 tops PinchBench at 94.9 and ranks second on SWE-Bench Pro at 65.2, behind Opus 4.8.
  • The open-weight KAT-Coder-V2.5-Dev is a separate 35B-A3B MoE below Apache-2.0, with its personal benchmark numbers.

Check out the Paper, the Model Weight on Hugging Face, and the Product Page. All credit score for this analysis goes to the researchers of this undertaking.

The put up KwaiKAT Team Releases KAT-Coder-V2.5: An Agentic Coding Model Trained on 100,000+ Verifiable Repository Environments appeared first on MarkTechPost.

Similar Posts