Why We're Changing Our Default Eval Model

We're changing the default solver model in our eval harness from Claude Sonnet 4.6 to GLM 5.1. This is the default we provide to everyone running evals on the platform. For most of the work the harness does, a frontier model gives you the strongest possible signal. However, that's more signal than the job needs and the difference is where eval budgets quietly leak. The question that decides how much you should be paying is whether a given eval run is measuring the model or measuring the skill.

The principle behind it: specify the model only when you care about the model. When your eval exists to answer "does this specific model ship well?", you have to run that exact model. When it exists to answer "does this skill improve agent behavior, and has anything regressed?", you don't need a specific model, you need a representative one.

We put this to the test on our own skill-evaluation harness and validated GLM 5.1 against Sonnet 4.6, the model it replaces as the default. We lost almost none of the signal skill authors rely on, and the eval bill went down. This post is the reasoning behind the switch, and a framework you can apply to your own eval stack.

Two questions, one eval harness

Our harness runs a large skill-evaluation suite: roughly 500 skills across about 850 tasks, each run twice, with the skill and without it. We score three things: instruction following (did the agent do what the skill tells it to do), task completion (did it reach the goal), and an overall blend weighted toward instruction following.

Lift is the difference between an agent's behavior with a skill and without it, and it's the number a skill author reads, because it isolates the skill's effect from the model's baseline.

Two models are in play on every run. The judge grades the trajectories; we keep it fixed and strong because the judge's grading on the rubric determines lift. The solver is the agent doing the task, and it's the free variable. Because each agentic trajectory is longer than a judging round, the solver dominates eval cost, so the practical question is whether we can swap the default solver for something cheaper without losing the lift signal.

To answer that, you need to know which of two questions your harness is answering.

The first is "does this specific model ship well?" If you are deciding which model goes into production, no proxy will do, because the model is the subject. The second is "does this skill change agent behavior, and not regress?" Here the model isn't the subject but an instrument for reading the skill, and an instrument only needs to be accurate enough to reproduce the signal you act on.

Most day-to-day skill development is the second question. You are iterating on a skill, watching whether the lift goes up, guarding against regressions. The specific solver underneath barely matters, as long as it tracks the frontier closely enough. The right default for that work is the cheapest model that faithfully reproduces the lift.

How to evaluate AI agents without paying frontier prices for every run

The obvious objection: a cheaper model is cheaper because it's worse, so won't the signal degrade with it? That depends on which signal. The absolute levels do degrade; the lift mostly doesn't.

We ran roughly 850 tasks across 500 skills head-to-head on both GLM 5.1 and Sonnet 4.6: same tasks, same judge, same with-and-without protocol. Then we correlated the per-skill lift.

At the skill level, across those 500 skills, the lift correlation was r = 0.72 (Spearman 0.69). If a skill lifts Sonnet, it often lifts GLM by a similar amount, and the correlation holds when you decompose it. This matters because a single headline number can hide a saturation artifact. Instruction-following lift, where almost all the signal lives (standard deviation 26), came in at r = 0.71. Task completion lift, which is small and near-saturated but carries the rare unlocks, came in at r = 0.74. The agreement is on each dimension and its magnitude.

For a screening tool, the number to watch is decision agreement. On the binary call every author actually makes, "does this skill help?", the two models agreed 88.5% of the time, and where they differ they differ in a safe direction: GLM is mildly conservative, with a mean lift of 22.3 against Sonnet's 24.3 and a regression slope around 0.76. It won't over-credit a skill, which is what you want for a regression guard.

For skill authors the takeaway is simple: the thing you act on, the sign and rough size of a skill's lift, reads the same on either model, so run the cheap one by default.

The limits of a cheap screen

The two models diverge on fine-grained, low-impact flagging. GLM catches roughly half the skills Sonnet rates as low-impact (under 5 points of lift), and on the rare outright-negative skills the overlap is smaller still. However, with only about two tasks per skill, plus the irreducible noise any LLM judge carries, the marginal cases are precisely where any two models disagree. The disagreement is concentrated where the evidence is thinnest, not spread across the confident calls.

This means that GLM is the cheap, fast screen you run constantly while developing skills and guarding against regressions. When a decision hinges on a single borderline skill or on which model you ship, you escalate to the model you care about. The screen narrows the field and the frontier model makes the final call. You're not trading accuracy for cost so much as spending accuracy where the decisions are and throughput everywhere else.

The cost story

Today, for our API pricing, the typical task is about 1.5x cheaper on GLM. GLM is cheaper on 83% of tasks and at least 1.5x cheaper on 52% of them, and on per-token API list price the gap widens to 2 to 3x. The one-line version: cheaper on the large majority of tasks, typically around 1.5x, and up to 2 to 3x per token.

Total eval spend is about 1.4x cheaper, roughly 28% lower, which is narrower than the per-task figure. The reason is a heavy cost tail: about 17% of tasks are runaway, chatty trajectories that loop or burn far more tokens than the median with one task alone reading 2.1M cached tokens. The aggregate gets pulled by that tail rather than by the typical task.

That tail is something we can optimize. The gap between the typical task at 1.5x and the aggregate at 1.4x comes from those runaway trajectories, and tightening turn and loop limits and how the harness drives long trajectories collapses the tail toward the median. That alone moves the aggregate toward the 1.5 to 2x the typical task already shows. This is cheaper today on most tasks, and on a cost curve we can keep pushing down.

How to apply this to your own stack

The principle generalizes well beyond our harness:

• Default your skill-development and regression evals to a cheap, SOTA-correlated solver. The volume of runs lives here.
• Pin the frontier model only for ship decisions and borderline single-skill calls. Here, a decision actually turns on the accuracy.

GLM 5.1 is now our default solver and is configurable in the eval runner. So before your next eval run, ask what that eval is actually answering: are you measuring the model, or measuring the skill? If it's the skill, what's the cheapest instrument that still moves when the skill moves?