A leaderboard you can trust must be able to say: this measurement was not valid.

What this measures, and what it does not

Conformance asks whether the server speaks MCP correctly. Fitness Report asks whether an agent can actually get the job done with it.

The word in that first sentence belongs to the official suite at github.com/modelcontextprotocol/conformance, which is a different project with a different job. Run it too. This board is about task outcomes.

MCP Atlas asks which model is best at using tools. Fitness Report asks which server is worth using.

The runner model is pinned into every score record and printed on every scored group. Two servers driven by different runner models are reported side by side but never read against each other, because the token accounting is not comparable across tokenizers.

Before anything is scored, validity gates run against our own generated task suite. When the eval itself is invalid we refuse to emit a score, and the refusal is published on the same board as the scores, in the slot a number would have occupied. A refusal is a result here, never a blank cell and never a missing row.

Gate order

Gates run cheapest first and stop at the first failure, so a broken eval never spends model budget: free structural checks, an answer-leak string check and the zero-token protocol probes, then cheap null-model baselines (no tools, stubbed-empty, random valid arguments), then the paid construct gate where a reference agent with full information must reach the answer key at a rate of at least 0.90, and only then the full drive. An indeterminate baseline halts exactly like a failure. On any gate failure the report carries no score field at all, plus the failing gate with its counts, threshold, alpha, p-value and reason string. Those are the strings printed on the board.

With a median of about 2.5 tools across the open server roster, a suite below 8 tasks is refused as INSUFFICIENT_SURFACE rather than published as a two-task 100%.

How to read a refusal

Naming the gate is not the same as explaining it. Each run record opens on the numbers the gate actually saw, drawn, and the three families read like this.

The null baseline killed it

Three null models run the same suite before any paid tier: one with no tools at all, one whose tool calls come back empty, and one calling tools with random valid arguments. The record shows each of those rates against the rate the agent reached with the real server, and the threshold between them. When a model with no tools passes most of the suite, the tasks were answerable from what the model already knew and the server was never in the loop. The finding is about our task generator, not about the server, and the record says so.

The surface was too thin

The record shows how many candidate tasks were generated, how many were admitted, the admission rate against its floor, and the suite size against the minimum of 8. Three fates are counted apart, because summing them would inflate the failure count: candidates rejected by admission, listed with the rule that rejected each one; candidates that passed admission and were trimmed as surplus, since the generator deliberately asks for more than it drives; and candidates deleted by the null screen below. Rows produced by the older generator, which did not serialise its per candidate reasons, carry that gap as a stated limitation instead of a breakdown.

Neither floor moves. A thin surface is fixed by generating tasks that survive admission. No threshold, ratio, floor or alpha on this page has ever been loosened to turn a refusal into a score, and none will be.

The construct gate failed

A reference agent that has been handed the answer key must still reach it through the server. When it cannot, the record prints its counts and then reads the tape: how many tool calls were recorded, how many came back as errors, and how many never received a matching protocol response at all. A server that advertises tools and then rejects every invocation before a response exists produces requests with no responses, and that is a finding about the server rather than about the agent. It is stated from the recorded counts, never assumed.

The generation time null screen, and the bias it introduces

Newer runs screen every candidate task before admission: the runner model is asked to answer it with no tools and no server, and a candidate it answers correctly is deleted before the suite is hashed and before any gate runs. The count deleted is published on each run record, and the deleted candidates are never counted as generator failures because nothing was wrong with them.

This is a bias and it points in one direction, so it is stated rather than buried: the run time null baseline on a screened suite measures the noise floor of a suite that was already purged of the tasks that gate looks for, which means it reads lower than it would on an unscreened suite. Nothing about the gate changed. Its 0.5 ratio, its 95th percentile and its halt behaviour are unchanged, the screen never removes a task from a gate denominator, and a screened suite can still be killed by the baseline. What it does mean is that a passed baseline on a screened row and a passed baseline on an unscreened row are not the same evidence, so each row prints which it is, and rows from different generator versions are never read against each other.

Every row therefore carries its task generator as a chip, scored or refused, and where a group holds more than one generator the group says so: rows built by one generator are only ever read against rows built by the same one. Records written before that field existed are chipped as generator not recorded rather than assumed to be the older version. Each run record states which screen and which extension protocol were registered for it, in the run's own words.

A suite the screen empties is its own finding rather than a thin surface. It refuses as DEGENERATE with the reason all_candidates_null_answerable and the screen ledger behind it, because "we generated enough tasks and a model answered them all with no server at all" is a statement about what the server adds, not about how many tools it has.

Extensions, and why EXTEND is not a second chance

The extension size and the maximum number of extensions are fixed in the pre-registration alongside n, before the first call, and both are written into the run record where you can read them. A gate that the registered sample cannot resolve either way is not re-run until it resolves: it consumes a registered extension batch, and after the last extension a gate that is still unresolved resolves to FAIL, which refuses the run as GATE_FAILED. Inside a run there is no optional stopping. A run completes its registered size or it is void, and a regenerated task suite is a new run under a new suite hash rather than another attempt at this one.

When a run did consume extensions, its record prints the whole sequence rather than the pooled number alone: what the gate stood at on the registered size, how many tasks each batch added and how many pooled trials they came to, what the pool moved to, and what that pooled count resolved to. Tasks and trials are printed as two numbers because they are two numbers: each task is driven at the registered reps, so a batch of six tasks can contribute eighteen trials to the pool, and one figure labelled both would misstate every batch.

That leaves one narrow use for EXTEND_EXHAUSTED: a pre-registration that bought no extension batch at all. With nothing to spend, a gate that resolves neither way is refused where it stands. So the outcome always means the same thing, in every era of this harness, which is that the gate resolved neither way and there was no batch available to buy. It has never meant that a gate was extended until it looked decisive.

Rows recorded before the extension protocol ran also carry EXTEND_EXHAUSTED, and they are not relabelled. Their records carry a zero policy and nothing else about the protocol: no registered batch size, no protocol statement on the gate that could have bought one, and no ledger. The arithmetic is the same as a pre-registration that buys none, and the record does not distinguish the two, so each such row says exactly that in its own record, read from its own fields rather than from a list of server names kept anywhere on this page. Their counts are untouched: they are what the gate measured on the day.

More than one run of the same server

A server can be driven again at any time, and several here have been. Each drive generates its own task suite under its own hash, so two runs of one server are two measurements and never two views of one. Every row therefore prints the run behind it, not just the server: the task generator as a chip, the suite hash, and the time the drive started. Where a server has more than one published run, each of its rows says which attempt it is and names the others, including where they sit in another group of the board.

Reruns are separate attempts, not a best of. The pre-registration binds what happens inside one run; it does not bind how many runs are attempted, and this page does not pretend otherwise. What it does instead is publish all of them: an earlier run is never replaced, hidden or folded into a later one, whatever its outcome was, and no row on this page was selected over another run of the same server.

What is not in these numbers

Every run uses two models. The runner drives the server and its turns are on the agent tape, so its tokens are counted and priced from the record. The judge synthesises the task suite and produces the destructiveness signal before the drive begins, and it writes to neither tape. A total summed from the tapes is therefore the runner's spend and not the run's.

So a dollar figure appears here only where the record supports it, labelled measured, and it is built from the runner's recorded tokens plus judge usage on the runs that record it. Everything known to sit outside that figure is named without a number: judge spend on runs that do not record it, the generation time null screen calls that are written to neither plane, and any model on a tape with no price on file. A flat per run assumption is not a measurement, and this page publishes none.

A figure is called a total only when nothing known is missing from it. Every other figure is labelled a floor and printed as at least, with the reason it is one. The reasons come from the record: judge usage the harness marked partial, a judge model in the usage breakdown with no price on file, judge calls that came back with no usage block at all, calls that threw before reporting any, and the same on the tape side. Each of those means dollars were spent that are not in the number, so the number is a lower bound and says so. A floor printed as a total is a wrong figure wearing a right label, and it is the one thing this block exists to stop. Where nothing in a record can support a figure at all, no figure is printed and the record says why.

Recorded on every run

The harness writes its method notes into every run it produces, so most of those notes are identical on every row and printing them twenty-two times would bury the notes that actually differ. The notes that a task generator records on every one of its runs are published once here, verbatim. Each run record shows only the notes that belong to that run, which are the ones carrying its own numbers, and says how many standing notes it also carries. Nothing is dropped and nothing is summarised: this is the same text, published once instead of twenty-two times.

Two documented divergences from evalgate

The gate math is ported from evalgate. It differs in exactly two places, and we own both here.

1. Construct denominator: oracle errors count

   evalgate silently shrinks the denominator when an oracle call throws, so three successes out of three completed calls reads as 100% even when forty calls errored. That is wrong for flaky remote servers, which is every remote server. Errors are gated on their own instead: errors / (n + errors) > 0.05 resolves the run to COMPROMISED. A compromised run is not a verdict about the server. It is a refusal to score, because the measurement did not complete.

2. Published PASS is symmetric with published FAIL

   A pass on this page additionally requires that the Wilson 95% lower bound sits at or above the threshold, or that the run's own n achieves the planned power against the detectable rate. Exact binomial power is sawtoothed in n, so clearing the planned sample size is not the same claim as being powered at it, and the run's actual n is what gets tested. Otherwise the verdict is downgraded to EXTEND and no pass is published. A small sample that happens to land above the line is not evidence, and publishing it while publishing failures at full rigour would be asymmetric.

Reading the interval

First-try success is drawn as its Wilson 95% interval on a shared 0 to 100 axis, with the point estimate marked inside it. The interval is the finding, not the point. There are no positions on this page: a row would only sit below another if that other interval lay entirely above it, and on the published runs no pair is separated that way, so the axis does the comparing and the scored group says out loud that nothing separates its rows. The intervals are computed once, in the run record. This page only renders the stored numbers, so there is no second copy of the statistics to drift.

Protocol probes

The probe columns are deterministic, cost zero tokens and run before anything else, so a refused run still reports them. Spec currency is the revision the connection actually negotiated. Protocol hygiene counts the probe checks that passed, including whether the server accepts an impossible protocol version, whether a mismatched method or name header is rejected, whether the modern server discovery method is implemented, whether cache hints are present and correctly scoped, and what the deprecation surface looks like. A probe that does not apply to a connection is reported as unknown, never folded into the pass count. A failed probe is a fact about the server, so it is coloured as a server fault and never as a refusal: a refusal is a decision about our own measurement.

Credential context

Every score is stamped with the credentials it was collected under: anonymous, free-key or owner-key. Tool surfaces differ by credential on real servers, so a number collected anonymously is not the same measurement as one collected with an owner key, and the two are labelled rather than merged.

Evidence

Every finding on this page links to the recorded session that justifies it. Each run publishes two tapes, the MCP wire plane and the agent plane, served with permissive cross-origin headers and an immutable cache, and the replay link opens both in the viewer in one merged timeline. Nothing is embedded or loaded until you click. There are no bare counts here: if a claim has no recording behind it, the row says so instead.

Prior art: MCPEval for model-driven task synthesis with machine-checkable predicates, evalgate for the validity gate math, mcp-tape for the tape format and pairing statistics. What is added here is the refusal, the signed replay behind every claim, and rewrite proposals tied to the recorded sessions they were derived from.