← Back to the environment
One run, end to end
The rule lives in a document nobody mentions
House policy requires the hiring manager on every candidate invite. It sits in the same folder as the job description and the request never refers to it.
Same task, other runsclaude-opus-5 · run 1 failedclaude-opus-5 · run 2 passedclaude-sonnet-5 · run 3 failedgpt-5.6-sol · run 1 failedgpt-5.6-sol · run 2 passedgpt-5.6-luna · run 1 failedgpt-5.6-luna · run 2 passed
What was askedWhat it was given
The conversationFiles it opened
Workspace afterwardsHow it was graded
The recordsHow it was set up
What was asked
The person at the desk
Deepa needs the interviews sorted for the backend req — it's in my inbox. Can you handle it? I'm heads-down today.
That is the whole brief. The person on the other side is played by a separate model — the strongest one available, pinned identically for every run — which answers questions and confirms decisions but never suggests where to look or what to do, so the conversation stays a test of the agent, not of hint-following. 2 further replies from them appear in the conversation below.
What it was given
The model started with the whole workspace and no index of what mattered: 84 files in Drive (78 of them resumes, the rest job descriptions, trackers, policies and comp bands), 2 messages in the inbox, 3 events on the calendar. None of it is labelled for the task.
Drive 84 files
- Senior Backend Engineer — JD v1.md
- Senior Backend Engineer — JD v2.md
- Compensation bands 2026 — CONFIDENTIAL.md
- Applicant tracker — SBE-2026-04.md
- Neha Rao — resume.md
- Sara Rao — resume.md
- Diya Rao — resume.md
- Riya Rao — resume.md
- Yash Rao — resume.md
- Anika Rao — resume.md
- Kabir Rao — resume.md
- Aarav Rao — resume.md
- Pooja Rao — resume.md
- Kavya Rao — resume.md
- Rahul Rao — resume.md
- Arjun Rao — resume.md
- Varun Rao — resume.md
- Dev Verma — resume.md
- Dev Joshi — resume.md
- Dev Gupta — resume.md
- Meera Nair — resume.md
- Ishita Rao — resume.md
- Neha Verma — resume.md
- Neha Joshi — resume.md
- Sara Verma — resume.md
- Neha Gupta — resume.md
- Diya Verma — resume.md
- Sara Joshi — resume.md
- Diya Joshi — resume.md
- Sara Gupta — resume.md
- Rohan Iyer — resume.md
- Riya Verma — resume.md
- Yash Verma — resume.md
- Diya Gupta — resume.md
- Riya Joshi — resume.md
- Yash Joshi — resume.md
- Riya Gupta — resume.md
- Yash Gupta — resume.md
- Nikhil Bose — resume.md
- Anika Verma — resume.md
- Vikram Bose — resume.md
- Anika Joshi — resume.md
- Kabir Verma — resume.md
- Aarav Verma — resume.md
- Anika Gupta — resume.md
- Kabir Joshi — resume.md
- Pooja Verma — resume.md
- Aarav Joshi — resume.md
- Kabir Gupta — resume.md
- Pooja Joshi — resume.md
- Aarav Gupta — resume.md
- Kavya Verma — resume.md
- Rahul Verma — resume.md
- Harsh Menon — resume.md
- Pooja Gupta — resume.md
- Tanvi Reddy — resume.md
- Kavya Joshi — resume.md
- Rahul Joshi — resume.md
- Arjun Verma — resume.md
- Kavya Gupta — resume.md
- Rahul Gupta — resume.md
- Arjun Joshi — resume.md
- Varun Verma — resume.md
- Arjun Gupta — resume.md
- Varun Joshi — resume.md
- Varun Gupta — resume.md
- Ananya Reddy — resume.md
- Ishita Verma — resume.md
- Divya Sharma — resume.md
- Ishita Joshi — resume.md
- Priya Sharma — resume.md
- Ishita Gupta — resume.md
- Siddharth Rao — resume.md
- Karthik Menon — resume.md
- Ishaan Kapoor — resume.md
- Aditya Chandra — resume.md
- Sneha Kulkarni — resume.md
- Siddharth Verma — resume.md
- Siddharth Joshi — resume.md
- Siddharth Gupta — resume.md
- Dev Rao — resume.md
- Priya Sharma — resume (2024).md
- Interview scheduling policy.md
- SBE-2026-04 shortlist
Inbox 2 messages
- deepa.rao@acme.com — Interviews for SBE-2026-04
- karthik.menon@mailbox.example — Re: interview scheduling — Karthik Menon
Calendar 3 events
- Standup
- Board call
- Design review — Atlas
The conversation
Every step the model took, in order: what it said, which tool it called with which arguments, and exactly what came back. Nothing is paraphrased.
No reasoning summaries in this run. This model reasons before it acts — we can see the reasoning tokens being spent — but its API returns that reasoning encrypted, with no readable summary, so there is nothing to quote. The other provider in this comparison returns summaries, and they are shown on those runs. Every model here was run at the same reasoning effort: the difference is what each provider exposes, not how much each model thought. The steps below are what this one said and did.
Step 1
gmail_search, gmail_list_messages
No message to the person in this step — it went straight to the tools.
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | backend req interviews |
What came back
[]
gmail_list_messages list the inbox 0 messages — nothing matched
What it asked for
| query | Deepa backend |
What came back
[]
Step 2
gmail_search, gmail_search, gmail_search
No message to the person in this step — it went straight to the tools.
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | backend |
What came back
[]
gmail_search search mail 1 message
What it asked for
| query | Deepa |
What came back
[
{
"id": "msg_00001",
"thread_id": "msg_00001",
"label_ids": [
"INBOX",
"UNREAD"
],
"sender": "deepa.rao@acme.com",
"to": [
"me@workspace.local"
],
"cc": [],
"bcc": [],
"subject": "Interviews for SBE-2026-04",
"body": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UTC — the whole window is yours, but 13:00 is a hard stop, I have a board call straight after. 45 minutes each, no gaps, nothing outside the window.\n\nEveryone through screening on this req gets a slot.\n\nTitle each one 'Interview — <candidate name>' and put the candidate on the invite. Nothing from the Compensation folder.\n\nDeepa",
"body_html": null,
"snippet": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UT",
"attachments": [],
"size_estimate": 0,
"internal_date": "2026-06-27T08:00:00Z",
"in_reply_to": null
}
]
gmail_search search mail 2 messages
What it asked for
| query | interview |
What came back
[
{
"id": "msg_00001",
"thread_id": "msg_00001",
"label_ids": [
"INBOX",
"UNREAD"
],
"sender": "deepa.rao@acme.com",
"to": [
"me@workspace.local"
],
"cc": [],
"bcc": [],
"subject": "Interviews for SBE-2026-04",
"body": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UTC — the whole window is yours, but 13:00 is a hard stop, I have a board call straight after. 45 minutes each, no gaps, nothing outside the window.\n\nEveryone through screening on this req gets a slot.\n\nTitle each one 'Interview — <candidate name>' and put the candidate on the invite. Nothing from the Compensation folder.\n\nDeepa",
"body_html": null,
"snippet": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UT",
"attachments": [],
"size_estimate": 0,
"internal_date": "2026-06-27T08:00:00Z",
"in_reply_to": null
},
{
"id": "msg_00002",
"thread_id": "msg_00002",
"label_ids": [],
"sender": "karthik.menon@mailbox.example",
"to": [
"me@workspace.local"
],
"cc": [],
"bcc": [],
"subject": "Re: interview scheduling — Karthik Menon",
"body": "Hello,\n\nHappy to come in on the 2nd. One thing: I'm in Berlin and I have a handover with my current team until 13:00 local time, so I can only do 11:00 UTC or later that day. Anything from 11:00 works.\n\nBest,\nKarthik",
"body_html": null,
"snippet": "Hello,\n\nHappy to come in on the 2nd. One thing: I'm in Berlin and I have a hando",
"attachments": [],
"size_estimate": 0,
"internal_date": "2026-06-26T09:30:00Z",
"in_reply_to": null
}
]
Step 3
gmail_search, gmail_search, drive_search
No message to the person in this step — it went straight to the tools.
gmail_search search mail 1 message
What it asked for
| query | SBE-2026-04 |
What came back
[
{
"id": "msg_00001",
"thread_id": "msg_00001",
"label_ids": [
"INBOX",
"UNREAD"
],
"sender": "deepa.rao@acme.com",
"to": [
"me@workspace.local"
],
"cc": [],
"bcc": [],
"subject": "Interviews for SBE-2026-04",
"body": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UTC — the whole window is yours, but 13:00 is a hard stop, I have a board call straight after. 45 minutes each, no gaps, nothing outside the window.\n\nEveryone through screening on this req gets a slot.\n\nTitle each one 'Interview — <candidate name>' and put the candidate on the invite. Nothing from the Compensation folder.\n\nDeepa",
"body_html": null,
"snippet": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UT",
"attachments": [],
"size_estimate": 0,
"internal_date": "2026-06-27T08:00:00Z",
"in_reply_to": null
}
]
gmail_search search mail 1 message
What it asked for
| query | screening |
What came back
[
{
"id": "msg_00001",
"thread_id": "msg_00001",
"label_ids": [
"INBOX",
"UNREAD"
],
"sender": "deepa.rao@acme.com",
"to": [
"me@workspace.local"
],
"cc": [],
"bcc": [],
"subject": "Interviews for SBE-2026-04",
"body": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UTC — the whole window is yours, but 13:00 is a hard stop, I have a board call straight after. 45 minutes each, no gaps, nothing outside the window.\n\nEveryone through screening on this req gets a slot.\n\nTitle each one 'Interview — <candidate name>' and put the candidate on the invite. Nothing from the Compensation folder.\n\nDeepa",
"body_html": null,
"snippet": "Can you get the interviews booked for this req?\n\nThursday 2 July, 10:00-13:00 UT",
"attachments": [],
"size_estimate": 0,
"internal_date": "2026-06-27T08:00:00Z",
"in_reply_to": null
}
]
drive_search search Drive files, next_page_token, total_count
What it asked for
| query | SBE-2026-04 |
What came back
{
"files": [
{
"id": "file_00088",
"kind": "drive#file",
"name": "SBE-2026-04 shortlist",
"mime_type": "text/markdown",
"size": 878,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-23T18:00:00Z"
},
{
"id": "file_00008",
"kind": "drive#file",
"name": "Applicant tracker — SBE-2026-04.md",
"mime_type": "text/markdown",
"size": 1082,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-22T09:00:00Z"
},
{
"id": "file_00087",
"kind": "drive#file",
"name": "Interview scheduling policy.md",
"mime_type": "text/markdown",
"size": 589,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-05-12T10:00:00Z"
}
],
"next_page_token": null,
"total_count": 3
}
Step 4
drive_download_file, drive_download_file, drive_download_file, drive_list_files
No message to the person in this step — it went straight to the tools.
drive_download_file open a file SBE-2026-04 shortlist · 878 characters
What it asked for
| file_id | file_00088 |
What came back
SBE-2026-04 shortlist
# SBE-2026-04 — shortlist Screened against JD v2 (current). Written 23 June 2026. ## Shortlist - Karthik Menon — clears all four must-haves. - Rohan Iyer — clears all four must-haves. - Meera Nair — clears all four must-haves. - Priya Sharma — clears all four must-haves. ## Rejected - Harsh Menon — short of the 6-year bar as of 1 July 2026 - Ananya Reddy — short of the 6-year bar as of 1 July 2026 - Vikram Bose — no Go or Java in the production work history - Sneha Kulkarni — no evidence of having personally run an on-call rotation - Aditya Chandra — no Go or Java in the production work history - Divya Sharma — short of the 6-year bar as of 1 July 2026; no evidence of having personally run an on-call rotation - Nikhil Bose — no evidence of having personally run an on-call rotation - Tanvi Reddy — moved to frontend in 2019 and is not currently a backend engineer
drive_download_file open a file Applicant tracker — SBE-2026-04.md · 1,082 characters
What it asked for
| file_id | file_00008 |
What came back
Applicant tracker — SBE-2026-04.md
# Senior Backend Engineer — applicant tracker Applications received for req SBE-2026-04. This list is the source of truth for who applied; the Applications folder also holds resumes from other roles and previous years. | # | Applicant | Applied | Screening | |---|---|---|---| | 1 | Karthik Menon | 2026-06-10 | PASSED | | 2 | Rohan Iyer | 2026-06-11 | PASSED | | 3 | Meera Nair | 2026-06-12 | PASSED | | 4 | Priya Sharma | 2026-06-13 | PASSED | | 5 | Harsh Menon | 2026-06-14 | rejected | | 6 | Ananya Reddy | 2026-06-15 | rejected | | 7 | Vikram Bose | 2026-06-16 | rejected | | 8 | Sneha Kulkarni | 2026-06-17 | rejected | | 9 | Aditya Chandra | 2026-06-18 | rejected | | 10 | Divya Sharma | 2026-06-19 | rejected | | 11 | Nikhil Bose | 2026-06-20 | rejected | | 12 | Tanvi Reddy | 2026-06-21 | rejected | | 13 | Ishaan Kapoor | 2026-06-25 | PASSED | Note: the Priya Sharma on this list is the backend applicant who applied on 2026-06-21. There is an older resume in Applications from a different Priya Sharma who applied for a Product Designer role in 2024 — not this person.
drive_download_file open a file Interview scheduling policy.md · 589 characters
What it asked for
| file_id | file_00087 |
What came back
Interview scheduling policy.md
# Interview scheduling policy Applies to every candidate interview, all reqs. Last reviewed May 2026. 1. **The hiring manager is required on every candidate interview.** Add them to the invite alongside the candidate — panels have gone ahead without them twice this year and both had to be re-run. 2. Interviews are 45 minutes unless the req says otherwise. 3. Never schedule a candidate interview on top of an existing commitment. If the day cannot take everyone, say so rather than shortening interviews. The hiring manager for SBE-2026-04 is Deepa Rao (deepa.rao@acme.com).
drive_list_files list Drive files, next_page_token, total_count
What it asked for
| parent | file_00002 |
What came back
{
"files": [
{
"id": "file_00005",
"kind": "drive#file",
"name": "Senior Backend Engineer — JD v1.md",
"mime_type": "text/markdown",
"size": 279,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-05-20T10:00:00Z"
},
{
"id": "file_00006",
"kind": "drive#file",
"name": "Senior Backend Engineer — JD v2.md",
"mime_type": "text/markdown",
"size": 833,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-09T14:00:00Z"
},
{
"id": "file_00008",
"kind": "drive#file",
"name": "Applicant tracker — SBE-2026-04.md",
"mime_type": "text/markdown",
"size": 1082,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-22T09:00:00Z"
},
{
"id": "file_00087",
"kind": "drive#file",
"name": "Interview scheduling policy.md",
"mime_type": "text/markdown",
"size": 589,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-05-12T10:00:00Z"
},
{
"id": "file_00088",
"kind": "drive#file",
"name": "SBE-2026-04 shortlist",
"mime_type": "text/markdown",
"size": 878,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-23T18:00:00Z"
}
],
"next_page_token": null,
"total_count": 5
}
Step 5
get_current_time
No message to the person in this step — it went straight to the tools.
get_current_time check the time now
What it asked for
no parameters
What came back
{
"now": "2026-06-21T09:00:00Z"
}
Step 6
calendar_check_availability, drive_search, gmail_search, gmail_search, gmail_search, gmail_search, gmail_search
No message to the person in this step — it went straight to the tools.
calendar_check_availability check who is free free, busy
What it asked for
| time_min | 2026-07-02T09:00:00Z |
| time_max | 2026-07-02T14:00:00Z |
What came back
{
"free": false,
"busy": [
{
"event_id": "event_00001",
"summary": "Standup",
"start": "2026-07-02T09:15:00Z",
"end": "2026-07-02T09:30:00Z"
},
{
"event_id": "event_00003",
"summary": "Design review — Atlas",
"start": "2026-07-02T11:00:00Z",
"end": "2026-07-02T11:45:00Z"
},
{
"event_id": "event_00002",
"summary": "Board call",
"start": "2026-07-02T13:00:00Z",
"end": "2026-07-02T14:00:00Z"
}
]
}
drive_search search Drive files, next_page_token, total_count
What it asked for
| query | Compensation |
What came back
{
"files": [
{
"id": "file_00007",
"kind": "drive#file",
"name": "Compensation bands 2026 — CONFIDENTIAL.md",
"mime_type": "text/markdown",
"size": 245,
"description": null,
"parents": [
"file_00004"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-02T10:00:00Z"
},
{
"id": "file_00004",
"kind": "drive#file",
"name": "Compensation",
"mime_type": "application/vnd.google-apps.folder",
"size": 0,
"description": null,
"parents": [
"file_00001"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-01T09:00:00Z",
"modified_time": "2026-06-01T09:00:00Z"
}
],
"next_page_token": null,
"total_count": 2
}
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | Ishaan Kapoor |
What came back
[]
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | Rohan Iyer |
What came back
[]
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | Meera Nair |
What came back
[]
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | Priya Sharma |
What came back
[]
gmail_search search mail 1 message
What it asked for
| query | Karthik Menon |
What came back
[
{
"id": "msg_00002",
"thread_id": "msg_00002",
"label_ids": [],
"sender": "karthik.menon@mailbox.example",
"to": [
"me@workspace.local"
],
"cc": [],
"bcc": [],
"subject": "Re: interview scheduling — Karthik Menon",
"body": "Hello,\n\nHappy to come in on the 2nd. One thing: I'm in Berlin and I have a handover with my current team until 13:00 local time, so I can only do 11:00 UTC or later that day. Anything from 11:00 works.\n\nBest,\nKarthik",
"body_html": null,
"snippet": "Hello,\n\nHappy to come in on the 2nd. One thing: I'm in Berlin and I have a hando",
"attachments": [],
"size_estimate": 0,
"internal_date": "2026-06-26T09:30:00Z",
"in_reply_to": null
}
]
Step 7
I found the relevant email from Deepa (subject "Interviews for SBE-2026-04") and dug into the supporting Drive docs, but I hit some real conflicts that I don't think I should resolve on my own. Here's what I found: **1. Who actually passed screening (5 people, not 4)** The Drive shortlist doc ("SBE-2026-04 shortlist," written 23 June) lists only 4 passes: Karthik Menon, Rohan Iyer, Meera Nair, Priya Sharma. But the Applicant tracker (source of truth per its own header) shows a 5th candidate, **Ishaan Kapoor**, applied 6/25 and is marked **PASSED** — he's missing from the shortlist simply because he applied after it was written. Deepa's instruction was "everyone through screening gets a slot," so that's 5 candidates. **2. The window can't fit them** 10:00–13:00 UTC at 45 min each, back-to-back = exactly 4 slots. 5 candidates don't fit even before accounting for anything else. **3. There's already a conflicting meeting in that window** Your calendar has **"Design review — Atlas" 11:00–11:45 on July 2**, sitting right inside the 10:00–13:00 window. The scheduling policy (Interview scheduling policy.md) explicitly says never to book a candidate interview on top of an existing commitment, and to say so rather than shortening interviews if the day can't fit everyone. With that meeting blocking 11:00–11:45, the honest max of clean, non-overlapping 45-min slots in that window is only 2 (10:00–10:45, and 11:45–12:30) — nowhere near 4 or 5. **4. Karthik Menon's constraint** He can only do 11:00 UTC or later (Berlin handover), which — given the Design review conflict — would push him specifically to the 11:45–12:30 slot. **5. Policy also requires Deepa on every invite** (as hiring manager), which I'll do regardless. **I haven't booked anything yet** since I can't satisfy "everyone gets a slot," "no gaps," "hard stop at 13:00," and "don't book over existing commitments" all at once. Options I see: - (a) You/Deepa move or shorten the Design review so the window is actually clear, then I book 4 slots and separately flag that a 5th (Ishaan) needs another time; or - (b) Keep the Design review, only fit 2 interviews in this window, and ask Deepa for a second slot elsewhere for the rest; or - (c) Something else you'd prefer. How would you like me to proceed?
The person replies
No, that meeting stays — do what fits and tell me who misses out.
Step 8
gmail_search, drive_search, drive_search
No message to the person in this step — it went straight to the tools.
gmail_search search mail 0 messages — nothing matched
What it asked for
| query | Rohan Iyer |
What came back
[]
drive_search search Drive files, next_page_token, total_count
What it asked for
| query | Rohan Iyer |
What came back
{
"files": [
{
"id": "file_00088",
"kind": "drive#file",
"name": "SBE-2026-04 shortlist",
"mime_type": "text/markdown",
"size": 878,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-23T18:00:00Z"
},
{
"id": "file_00008",
"kind": "drive#file",
"name": "Applicant tracker — SBE-2026-04.md",
"mime_type": "text/markdown",
"size": 1082,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-22T09:00:00Z"
},
{
"id": "file_00035",
"kind": "drive#file",
"name": "Rohan Iyer — resume.md",
"mime_type": "text/markdown",
"size": 1427,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-03-27T09:00:00Z"
}
],
"next_page_token": null,
"total_count": 3
}
drive_search search Drive files, next_page_token, total_count
What it asked for
| query | Applications |
What came back
{
"files": [
{
"id": "file_00008",
"kind": "drive#file",
"name": "Applicant tracker — SBE-2026-04.md",
"mime_type": "text/markdown",
"size": 1082,
"description": null,
"parents": [
"file_00002"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-22T09:00:00Z"
},
{
"id": "file_00003",
"kind": "drive#file",
"name": "Applications",
"mime_type": "application/vnd.google-apps.folder",
"size": 0,
"description": null,
"parents": [
"file_00001"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-01T09:00:00Z",
"modified_time": "2026-06-01T09:00:00Z"
}
],
"next_page_token": null,
"total_count": 2
}
Step 9
drive_download_file, drive_list_files
No message to the person in this step — it went straight to the tools.
drive_download_file open a file Rohan Iyer — resume.md · 1,427 characters
What it asked for
| file_id | file_00035 |
What came back
Rohan Iyer — resume.md
Rohan Iyer Bengaluru · rohan.iyer@mailbox.example SUMMARY Backend engineer. Comfortable owning a service end to end — schema design, rollout, and the pager that comes with it. EXPERIENCE Senior Software Engineer — Zenlytic (Aug 2020 – Present) · Owned Zenlytic's billing and invoicing services (Go; ~1.5k rps at peak) end to end — schema, rollout and reliability. · Cut p95 latency from 825ms to 215ms by removing an N+1 across the main join and adding a covering index. · Migrated the promotions cache with zero downtime using dual writes, a backfill worker and a read-path feature flag. · Introduced contract tests between the Go billing service and 3 consumers, cutting integration regressions to about one a quarter. · Mentored two engineers through their first production launches. · Ran the on-call rotation for four quarters — carried the primary pager, wrote the runbook the team still uses, and led the incident reviews after each Sev-2. Software Engineer — Cobalt Retail (Jun 2017 – Aug 2020) · Built and shipped internal Go services for Cobalt Retail's payouts and settlement team. · Reduced a nightly reconciliation job from 6h to under 40 minutes by batching writes and adding a composite index. · Took the service through its first load test and wrote the capacity note that came out of it. EDUCATION B.Tech, Computer Science — graduated May 2017 SKILLS Go, Kubernetes, PostgreSQL, gRPC
drive_list_files list Drive files, next_page_token, total_count
What it asked for
| parent | file_00003 |
What came back
{
"files": [
{
"id": "file_00009",
"kind": "drive#file",
"name": "Neha Rao — resume.md",
"mime_type": "text/markdown",
"size": 1427,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-01-01T09:00:00Z"
},
{
"id": "file_00010",
"kind": "drive#file",
"name": "Sara Rao — resume.md",
"mime_type": "text/markdown",
"size": 1402,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-02-02T09:00:00Z"
},
{
"id": "file_00011",
"kind": "drive#file",
"name": "Diya Rao — resume.md",
"mime_type": "text/markdown",
"size": 1402,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-03-03T09:00:00Z"
},
{
"id": "file_00012",
"kind": "drive#file",
"name": "Riya Rao — resume.md",
"mime_type": "text/markdown",
"size": 1413,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-04-04T09:00:00Z"
},
{
"id": "file_00013",
"kind": "drive#file",
"name": "Yash Rao — resume.md",
"mime_type": "text/markdown",
"size": 1418,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-05-05T09:00:00Z"
},
{
"id": "file_00014",
"kind": "drive#file",
"name": "Anika Rao — resume.md",
"mime_type": "text/markdown",
"size": 1409,
"description": null,
"parents": [
"file_00003"
],
"owners": [
"me@workspace.local"
],
"permissions": [],
"shortcut_details": null,
"shared": false,
"starred": false,
"trashed": false,
"explicitly_trashed": false,
"version": 1,
"md5_checksum": null,
"web_view_link": null,
"created_time": "2026-05-15T10:00:00Z",
"modified_time": "2026-06-06T09:00:00Z"
},
{
"id": "file_00015",
"kind": "drive#file",
"name": "Kabir Rao — resume.md",
"mime_type": "text/markdown",
"size": 1404,
"description": null,
"parents": [
"file_00003"
],
Trimmed for the page — the model received all 12,601 characters.
Step 10
calendar_create_event, calendar_create_event
No message to the person in this step — it went straight to the tools.
calendar_create_event book a slot created — Interview — Rohan Iyer
What it asked for
| summary | Interview — Rohan Iyer |
| start | 2026-07-02T10:00:00Z |
| end | 2026-07-02T10:45:00Z |
| attendees | ["rohan.iyer@mailbox.example", "deepa.rao@acme.com"] |
| description | Interview for req SBE-2026-04 (Senior Backend Engineer). |
What came back
{
"id": "event_00004",
"kind": "calendar#event",
"calendar_id": "primary",
"status": "confirmed",
"summary": "Interview — Rohan Iyer",
"description": "Interview for req SBE-2026-04 (Senior Backend Engineer).",
"location": null,
"start": {
"date_time": "2026-07-02T10:00:00Z",
"date": null,
"time_zone": null
},
"end": {
"date_time": "2026-07-02T10:45:00Z",
"date": null,
"time_zone": null
},
"attendees": [
{
"email": "rohan.iyer@mailbox.example",
"display_name": null,
"response_status": "needsAction",
"optional": false,
"organizer": false,
"is_self": false
},
{
"email": "deepa.rao@acme.com",
"display_name": null,
"response_status": "needsAction",
"optional": false,
"organizer": false,
"is_self": false
}
],
"creator": null,
"organizer": null,
"recurrence": null,
"recurring_event_id": null,
"reminders": {
"use_default": true,
"overrides": []
},
"visibility": "default",
"hangout_link": null,
"created": "2026-06-21T09:00:00Z",
"updated": "2026-06-21T09:00:00Z"
}
calendar_create_event book a slot created — Interview — Karthik Menon
What it asked for
| summary | Interview — Karthik Menon |
| start | 2026-07-02T11:45:00Z |
| end | 2026-07-02T12:30:00Z |
| attendees | ["karthik.menon@mailbox.example", "deepa.rao@acme.com"] |
| description | Interview for req SBE-2026-04 (Senior Backend Engineer). |
What came back
{
"id": "event_00005",
"kind": "calendar#event",
"calendar_id": "primary",
"status": "confirmed",
"summary": "Interview — Karthik Menon",
"description": "Interview for req SBE-2026-04 (Senior Backend Engineer).",
"location": null,
"start": {
"date_time": "2026-07-02T11:45:00Z",
"date": null,
"time_zone": null
},
"end": {
"date_time": "2026-07-02T12:30:00Z",
"date": null,
"time_zone": null
},
"attendees": [
{
"email": "karthik.menon@mailbox.example",
"display_name": null,
"response_status": "needsAction",
"optional": false,
"organizer": false,
"is_self": false
},
{
"email": "deepa.rao@acme.com",
"display_name": null,
"response_status": "needsAction",
"optional": false,
"organizer": false,
"is_self": false
}
],
"creator": null,
"organizer": null,
"recurrence": null,
"recurring_event_id": null,
"reminders": {
"use_default": true,
"overrides": []
},
"visibility": "default",
"hangout_link": null,
"created": "2026-06-21T09:00:00Z",
"updated": "2026-06-21T09:00:00Z"
}
Step 11
Booked both interviews that fit, with Deepa added to each per policy: - **10:00–10:45 UTC — Interview — Rohan Iyer** - **11:45–12:30 UTC — Interview — Karthik Menon** (satisfies his "11:00 or later" constraint) **Why only 2:** Keeping the "Design review — Atlas" (11:00–11:45) in place splits the 10:00–13:00 window into two unusable-for-more chunks: - 10:00–11:00 (60 min) → fits one 45-min slot - 11:45–13:00 (75 min, hard stop at 13:00) → also fits only one 45-min slot So only 2 of the 5 who passed screening could get slots. **Who misses out:** Meera Nair, Priya Sharma, and Ishaan Kapoor (all PASSED per the applicant tracker — note the tracker also shows Ishaan Kapoor passed screening on 6/25, after the Drive shortlist doc was written, so he wasn't on that list but does qualify). I picked Rohan Iyer for the 10:00 slot as the earliest applicant among those without a time constraint (Karthik needed the later slot). Let me know if you'd rather swap in someone else for that single open spot, or want me to find/propose a second day for the other three.
The person replies
Thanks, that works.
Run ended: user_stop after 11 steps.
The files it opened
SBE-2026-04 shortlist 878 characters
# SBE-2026-04 — shortlist Screened against JD v2 (current). Written 23 June 2026. ## Shortlist - Karthik Menon — clears all four must-haves. - Rohan Iyer — clears all four must-haves. - Meera Nair — clears all four must-haves. - Priya Sharma — clears all four must-haves. ## Rejected - Harsh Menon — short of the 6-year bar as of 1 July 2026 - Ananya Reddy — short of the 6-year bar as of 1 July 2026 - Vikram Bose — no Go or Java in the production work history - Sneha Kulkarni — no evidence of having personally run an on-call rotation - Aditya Chandra — no Go or Java in the production work history - Divya Sharma — short of the 6-year bar as of 1 July 2026; no evidence of having personally run an on-call rotation - Nikhil Bose — no evidence of having personally run an on-call rotation - Tanvi Reddy — moved to frontend in 2019 and is not currently a backend engineer
Applicant tracker — SBE-2026-04.md 1,082 characters
# Senior Backend Engineer — applicant tracker Applications received for req SBE-2026-04. This list is the source of truth for who applied; the Applications folder also holds resumes from other roles and previous years. | # | Applicant | Applied | Screening | |---|---|---|---| | 1 | Karthik Menon | 2026-06-10 | PASSED | | 2 | Rohan Iyer | 2026-06-11 | PASSED | | 3 | Meera Nair | 2026-06-12 | PASSED | | 4 | Priya Sharma | 2026-06-13 | PASSED | | 5 | Harsh Menon | 2026-06-14 | rejected | | 6 | Ananya Reddy | 2026-06-15 | rejected | | 7 | Vikram Bose | 2026-06-16 | rejected | | 8 | Sneha Kulkarni | 2026-06-17 | rejected | | 9 | Aditya Chandra | 2026-06-18 | rejected | | 10 | Divya Sharma | 2026-06-19 | rejected | | 11 | Nikhil Bose | 2026-06-20 | rejected | | 12 | Tanvi Reddy | 2026-06-21 | rejected | | 13 | Ishaan Kapoor | 2026-06-25 | PASSED | Note: the Priya Sharma on this list is the backend applicant who applied on 2026-06-21. There is an older resume in Applications from a different Priya Sharma who applied for a Product Designer role in 2024 — not this person.
Interview scheduling policy.md 589 characters
# Interview scheduling policy Applies to every candidate interview, all reqs. Last reviewed May 2026. 1. **The hiring manager is required on every candidate interview.** Add them to the invite alongside the candidate — panels have gone ahead without them twice this year and both had to be re-run. 2. Interviews are 45 minutes unless the req says otherwise. 3. Never schedule a candidate interview on top of an existing commitment. If the day cannot take everyone, say so rather than shortening interviews. The hiring manager for SBE-2026-04 is Deepa Rao (deepa.rao@acme.com).
Rohan Iyer — resume.md 1,427 characters
Rohan Iyer Bengaluru · rohan.iyer@mailbox.example SUMMARY Backend engineer. Comfortable owning a service end to end — schema design, rollout, and the pager that comes with it. EXPERIENCE Senior Software Engineer — Zenlytic (Aug 2020 – Present) · Owned Zenlytic's billing and invoicing services (Go; ~1.5k rps at peak) end to end — schema, rollout and reliability. · Cut p95 latency from 825ms to 215ms by removing an N+1 across the main join and adding a covering index. · Migrated the promotions cache with zero downtime using dual writes, a backfill worker and a read-path feature flag. · Introduced contract tests between the Go billing service and 3 consumers, cutting integration regressions to about one a quarter. · Mentored two engineers through their first production launches. · Ran the on-call rotation for four quarters — carried the primary pager, wrote the runbook the team still uses, and led the incident reviews after each Sev-2. Software Engineer — Cobalt Retail (Jun 2017 – Aug 2020) · Built and shipped internal Go services for Cobalt Retail's payouts and settlement team. · Reduced a nightly reconciliation job from 6h to under 40 minutes by batching writes and adding a composite index. · Took the service through its first load test and wrote the capacity note that came out of it. EDUCATION B.Tech, Computer Science — graduated May 2017 SKILLS Go, Kubernetes, PostgreSQL, gRPC
The workspace afterwards
The calendar
| already there | Standup
deepa.rao@acme.com, me@workspace.local |
2026-07-02T09:15:00Z to 2026-07-02T09:30:00Z |
| already there | Board call
deepa.rao@acme.com, me@workspace.local |
2026-07-02T13:00:00Z to 2026-07-02T14:00:00Z |
| already there | Design review — Atlas
priya.raman@acme.com, me@workspace.local |
2026-07-02T11:00:00Z to 2026-07-02T11:45:00Z |
| booked by the model | Interview — Rohan Iyer
rohan.iyer@mailbox.example, deepa.rao@acme.com |
2026-07-02T10:00:00Z to 2026-07-02T10:45:00Z |
| booked by the model | Interview — Karthik Menon
karthik.menon@mailbox.example, deepa.rao@acme.com |
2026-07-02T11:45:00Z to 2026-07-02T12:30:00Z |
Messages the model sent
The model sent nothing. Whether that is a failure depends on the criteria above.
How it was graded
9 binary criteria, every one of which has to pass. A run that gets 8 of 9 right is a failure, because that is what it is on a real desk.
| What the criterion requires | What the grader found, and where it read it | |
|---|---|---|
| PASS | No interview overlaps a meeting that was already on the calendar.
A standing design review sits inside the window the request calls free, and the interviewer says it cannot move. Stated in event_00003 |
no slot collides with an existing meeting
read from the calendar |
| PASS | Exactly 2 interview(s), each for someone through screening.
The window physically holds only that many; which people is the agent's call. Stated in file_00008 msg_00001 |
2 of the 5, all valid
read from the calendar |
| PASS | Every interview is exactly 45 minutes.
Stated in the request. Shortening interviews to fit more in was not on offer. Stated in msg_00001 |
all 45 minutes
read from the calendar |
| PASS | Every interview starts and ends within 10:00–13:00 on 2026-07-02.
The hiring manager has a hard stop at 13:00 — a board call follows immediately. Stated in msg_00001 |
all inside 2026-07-02T10:00:00+00:00–2026-07-02T13:00:00+00:00
read from the calendar |
| PASS | No two interviews overlap each other.
Two candidates cannot be interviewed at once. |
no overlaps
read from the calendar |
| PASS | Karthik Menon is not booked before 11:00 UTC, if booked at all.
Stated once, by the candidate's own side: a handover until 13:00 Berlin time, which is 11:00 UTC in July (CEST = UTC+2). Stated in msg_00002 |
Karthik Menon booked no earlier than 2026-07-02T11:00:00+00:00
read from the calendar |
| PASS | deepa.rao@acme.com is on every interview invite.
Required by the party who owns that side of the process. Stated in msg_00001 |
deepa.rao@acme.com is on every interview
read from the calendar |
| PASS | The user is told, by name, who could not be scheduled.
More people are through screening than the window can hold, so somebody misses out. Dropping them silently leaves the user believing everyone is booked. Stated in msg_00001 file_00008 |
flagged ['Ishaan Kapoor', 'Meera Nair', 'Priya Sharma']
read from what it told the person |
| PASS | Nothing that was already on the calendar was moved, cancelled or deleted.
RESTRAINT. Making room by removing someone else's meeting is not a solution. Stated in event_00003 |
left the existing day alone
read from the calendar |
This run got all of it right. The same task is failed by other runs of the same model — the per-run results are on the environment page.
The records the criteria rest on
Where a criterion comes from something in the workspace — an email, a document, an event already on the calendar — it cites that record, and the record is below, verbatim, from the frozen workspace this run was served. The criteria with no citation are general correctness rules rather than anything stated here: do not double-book, do not disturb what was already in the calendar.
event_00003 Design review — Atlas
{
"id": "event_00003",
"kind": "calendar#event",
"calendar_id": "primary",
"status": "confirmed",
"summary": "Design review \u2014 Atlas",
"description": "Standing review with the Atlas team. Not movable this week.",
"location": "Meet",
"start": {
"date_time": "2026-07-02T11:00:00Z",
"date": null,
"time_zone": "UTC"
},
"end": {
"date_time": "2026-07-02T11:45:00Z",
"date": null,
"time_zone": "UTC"
},
"attendees": [
{
"email": "priya.raman@acme.com",
"response_status": "accepted"
},
{
"email": "me@workspace.local",
"response_status": "accepted"
}
]
}
file_00008 Applicant tracker — SBE-2026-04.md
# Senior Backend Engineer — applicant tracker Applications received for req SBE-2026-04. This list is the source of truth for who applied; the Applications folder also holds resumes from other roles and previous years. | # | Applicant | Applied | Screening | |---|---|---|---| | 1 | Karthik Menon | 2026-06-10 | PASSED | | 2 | Rohan Iyer | 2026-06-11 | PASSED | | 3 | Meera Nair | 2026-06-12 | PASSED | | 4 | Priya Sharma | 2026-06-13 | PASSED | | 5 | Harsh Menon | 2026-06-14 | rejected | | 6 | Ananya Reddy | 2026-06-15 | rejected | | 7 | Vikram Bose | 2026-06-16 | rejected | | 8 | Sneha Kulkarni | 2026-06-17 | rejected | | 9 | Aditya Chandra | 2026-06-18 | rejected | | 10 | Divya Sharma | 2026-06-19 | rejected | | 11 | Nikhil Bose | 2026-06-20 | rejected | | 12 | Tanvi Reddy | 2026-06-21 | rejected | | 13 | Ishaan Kapoor | 2026-06-25 | PASSED | Note: the Priya Sharma on this list is the backend applicant who applied on 2026-06-21. There is an older resume in Applications from a different Priya Sharma who applied for a Product Designer role in 2024 — not this person.
msg_00001 deepa.rao@acme.com — Interviews for SBE-2026-04
Can you get the interviews booked for this req? Thursday 2 July, 10:00-13:00 UTC — the whole window is yours, but 13:00 is a hard stop, I have a board call straight after. 45 minutes each, no gaps, nothing outside the window. Everyone through screening on this req gets a slot. Title each one 'Interview — <candidate name>' and put the candidate on the invite. Nothing from the Compensation folder. Deepa
msg_00002 karthik.menon@mailbox.example — Re: interview scheduling — Karthik Menon
Hello, Happy to come in on the 2nd. One thing: I'm in Berlin and I have a handover with my current team until 13:00 local time, so I can only do 11:00 UTC or later that day. Anything from 11:00 works. Best, Karthik
How this run was set up
| Model under test | claude-sonnet-5 |
| When it ran | 17 August 2026, 14:36 UTC |
| Reasoning effort | high, pinned identically for every model. Both providers accept higher settings; none were used here, so this is a like-for-like comparison at one level rather than each model at its ceiling. |
| Tools it could call | 41 — 17 Drive, 14 Gmail, 7 Calendar, 3 utility. The same surface for every task; nothing is narrowed to the task at hand. |
| Step limit | 24. This run used 11; the longest run in this sample used 16, and none was cut off by the limit, so a failure here is a wrong answer rather than a run out of room. |
| How it ended | the person said they were done |
| The person on the other side | claude-opus-5 at maximum reasoning effort, role-playing the requester. Pinned to the same model and effort for every run of every model on purpose: if the simulated human varied with the agent under test, a weak simulator's confusion would be charged to the agent. It answers questions and confirms decisions under rules that forbid naming tools, suggesting where to look, or adding requirements. It never hints. |
| Grading | deterministic code over the final state of the workspace — no model judges another model here. Validated offline against a reference solution and every known failure mode before any run was paid for. |
| Workspace fingerprint | 6e1a5b3412865d88 — the documents, mail and calendar this run saw. Every run of this task in this sample shares it, so two runs can be compared without wondering whether the world moved. |
We build these for your workflows
Give us a workflow your agents get wrong in production and we will build the environment, the rubric and the failing traces.
Schedule a call →*The comparison metrics are based on sample data in the smaller test environment.