← Back to AI Data Labs

RecruiterOpsBench

Recruiting coordination

A live recruiting desk — job descriptions, an applicant tracker, 77–78 resumes, a shared calendar, and four people who each hold one piece of the answer. Frontier models schedule confidently here, and get it wrong in ways a recruiter would notice immediately.

Schedule a call →

Model	Passes every run	Passes / $	Median run	Runs
claude-sonnet-5	60%	2.2	154s	15
claude-opus-5	40%	1.1	93s	15
gpt-5.6-luna	0%	0.0	228s	15
gpt-5.6-sol	0%	0.0	227s	24

What this sample can and cannot separate. At three runs per scenario, claude-sonnet-5, claude-opus-5 and gpt-5.6-luna are not distinguishable from one another — the top two differ by one run out of the fifteen each of them ran (p = 0.500), so the order of those rows is noise and we are not claiming a winner among them. What the sample does establish is the gap to gpt-5.6-sol: p = 0.003 on the raw runs, and it scores below every model above it on 3 of the 5 scenarios. Tiers are separable here; rankings inside a tier are not, and a league table would imply otherwise. Two rows share 0% here and are still coloured differently, which is not a slip: sweeping no scenario is a floor that hides a real difference underneath it — gpt-5.6-luna passed 9 of 15 runs, gpt-5.6-sol passed 9 of 24 runs. The colour follows the test on those runs, not the rounded figure above it.

Passes every run is the share of the 5 scenarios below where the model passed every run of that scenario — pass^3 in the usual notation, not pass@3, which asks only whether one attempt in 3 succeeded. A desk needs the work done every time, so that is what is counted; each card below shows its own runs, so the arithmetic can be redone by hand. Passes / $ uses that same every-run definition, so both columns answer one question rather than two. Cost is computed from the token counts each provider reported, at list prices verified on 2026-08-17.

This is a sample. The scenarios below are a selection from a much larger recruiting suite we maintain — enough to show the shape of the failures and how we grade them, not the whole set. We build the same depth for whichever workflow you care about.

Where they fail

Each scenario links to complete transcripts of individual runs — the model's own reasoning, every tool call and what it returned, the files it opened, what it left in the workspace, and the criterion that caught it. A failing run and a passing run of the same task, side by side.

How it is graded

Every task is scored on binary criteria that must all pass — a partially correct answer is a failure, because that is what it is on a real desk. Checks read the final state of the workspace and the messages the agent sent: what is on the calendar, who is on the invite, what was written to whom, what was left alone. Every criterion on a transcript says which of those it was read from.

Each scenario runs at least three times per model, and more where the first three disagreed — which is why some cells below show 6 or 9 runs. Each card states its own count, and the pass rates in the table above are the plain average of the per-scenario rates, so a scenario that was run more times does not count for more.

Deterministic grading, validated offline against a reference solution and every known failure mode before any model was run.

The same tools every time	41 tools — 17 Drive, 14 Gmail, 7 Calendar, 3 utility — handed to the model whole on every task. Nothing is narrowed to the task at hand, so finding what matters is part of the job.
Room to work	Up to 24 steps per run. The longest run here used 16, and none was stopped by the limit — so a failure below is a wrong answer, not a run that ran out of room.
Reasoning effort	High, pinned identically for every model. Both providers accept higher settings; none were used, so this compares the models at one level rather than each at its ceiling.
The person on the other side	claude-opus-5 at maximum reasoning effort role-plays the requester — pinned to the same model and effort for every run of every model, so a weak simulator's confusion is never charged to the agent under test. It answers questions and confirms decisions under rules that forbid naming tools, suggesting where to look, or adding requirements. It never hints, so what is measured is the agent rather than hint-following.
Who decides pass or fail	Code, not a model. No model judges another model here, and every criterion names the record in the workspace it rests on — visible in the transcripts.

*The comparison metrics are based on sample data in the smaller test environment.