The data behind early failure detection.

Where agent spend goes: a public SWE-bench audit

We measured every public mini-SWE-agent attempt on SWE-bench Verified across 41 leaderboard submissions, and attributed model spend to attempts that resolved and attempts that didn't.

On the subset with complete pricing, a failed attempt cost 1.51× a resolved one on average.

Failed runs are not just common. They are disproportionately expensive, because agents keep spending while they struggle.

Methods and limits

• Observational study of public benchmark runs. It measures waste; it does not measure First Light savings.
• Pricing statistics use defined subsets of attempts with usable cost records.
• Not all spend on unresolved attempts is recoverable: some failing runs are only knowable as failures at the end.

Source: github.com/amnry/swe-bench-token-waste ↗

Runs that look doomed usually aren't

The obvious fix for wasted spend is to kill runs that look like they are failing. We tested whether the usual warning signs predict failure.

Generic early predictors built on public trajectories did not beat the base rate at call 3 (Brier score 0.1667 vs. 0.1640).

Generic signals throw away good runs. The signals that predict failure are workflow-specific, which is why First Light learns them from each team's own outcome history.

Methods and limits

• Early-prediction tests used public coding-agent trajectories, without customer approval data.
• Our next study scales failure prediction across all 41 submissions (8,606 unresolved attempts) and tests a budget-cap decision against a fixed step cap.

Reported agent cost can't be taken at face value

Depending on how a toolchain's cost fields are interpreted, the reported cost of the same runs swung from −0.03% to +455%.

You cannot optimize cost per successful outcome until you can measure it. That is the first thing we compute for every design partner.

A drop-in runtime gate

A proxy that wraps any agent by changing one base URL, with no agent code changes. Decisions run cheapest first: deterministic rules, then a small model distilled from a frontier judge, then a human by SMS. Every decision is written to a hash-chained audit log, and mined failures retrain the model.

Methods and limits

• Tested end to end against a simulated Stripe environment with synthetic tasks and policies; 12 passing tests.
• This prototype demonstrates the integration pattern and decision order. It is not a measured cost result.

Pre-execution checks avoid work that caching can't

Six authored FFmpeg jobs tested whether checks before encoding could avoid work beyond an ordinary cached pipeline.

20.8% fewer executed encodes, with 6 of 6 technical QA passes in both arms. Measured local workflow time fell from 7.432 s to 6.708 s (−9.7%).

Methods and limits

• Authored fixtures, not customer video generation, approval or invoiced savings.
• Shows that deterministic checks can avoid operations before execution even after caching; does not establish model-based failure prediction or savings at production scale.

Task-aware model selection

A frozen group-routing policy selected between two small local models on 24 tool-use diagnostic tasks.

Methods and limits

• Two paired gains, zero paired losses; exact paired p = 0.50, so not statistically established.
• Total tokens were 4.5% higher for First Light.
• Unofficial diagnostic set, not a BFCL leaderboard run. Does not show a trained mid-run failure predictor or customer cost savings.