RL environments that actually break your agents.
Real APIs. Ambiguous instructions. Multi-step traps. We build sandboxed environments that expose the exact failure modes your agents hit in production.
Schedule a call →
Venture Highway
Kunal Shah
Amit Singhal
Multi step tasks with adaptive user-simulator that test reasoning, tool use, and restraint. Each task is verified by inspecting final state as well as response using LLM-judged rubrics.
Recency Bias Kills Multi-Turn Tasks
Wrong Parameter Values Hide All Results
max_amount: 0 hides all of it, and the model never questions why.The user asks the model to scan a year of payment history and find which subscriptions quietly raised their price. The wallet contains 90+ transfers across 17 merchants — salary deposits, subscriptions, one-off purchases, peer transfers.
Two subscriptions actually hiked:
₹899 → ₹1,299
Billed under two different entity names pre/post hike
₹249 → ₹349
Quiet hike, only visible by scanning oldest pages
Four decoys must not be reported: Streamly (flat ₹499), FitPulse (price drop from ₹799 to ₹599), CloudVault (single annual charge), and Metro (variable prepaid recharges).
The model calls payments_transfers_list with max_amount: 0 — a filter that means “show me transfers of ₹0 or less.” Every real transfer is excluded. The model treats the empty response as ground truth instead of questioning its own parameters. It never tries removing the broken filter, never checks what max_amount actually does. 90 transfers sit untouched in the database.
| Merchant | Pattern | Trap |
|---|---|---|
| Lumina Design → Lumina Design Cloud | ₹899 × 3mo, then ₹1,299 × 11mo | Two billing entities — filtering by name misses old price |
| Nimbus Notes Pro | ₹249 × 2mo, then ₹349 × 12mo | Hike only visible on the last pages (oldest-first) |
| Streamly | ₹499 flat, every month | Decoy — no change |
| FitPulse | ₹799 → ₹599 | Decoy — price dropped |
| CloudVault | ₹1,999 once | Decoy — single annual charge |
| Metro Prepaid | ₹199 / ₹249 / ₹299 (varies) | Decoy — user-chosen recharge amounts |
| + 11 more | Uber, BigBasket, Croma, Apollo, ... | One-off purchases, peer transfers, salary |
Brand names live only in payee.name — not in searchable text. A q="lumina" keyword search returns nothing. Only a full scan works.
1 turn · 7 tool calls (1 failed) · 10 LLM calls · services: mail, notes, payments
Thinking — planning the analysis
I need to help the user by checking their payment history and identifying subscription price increases over the past year. I’ll look for recurring merchants and compare old and new prices. The timeframe is August 2025 to August 2026.
payments_transfers_list direction: "outgoing", status: "completed", min_amount: 0, max_amount: 0, after: "2025-08-10", page_size: 1000 transfers returned
{"transfers": [], "result_size_estimate": 0}
Thinking — pivoting to mail instead of fixing the query
The tool returned zero transfers, which is surprising. The user wanted to check their payment history, but without any completed outgoing transfers in that date range, I can’t analyze anything. I’ll suggest searching mail for receipts to find any subscription price changes.
mail_messages_list q: "(subject:(receipt OR invoice..."INVALID_ARGUMENT — unsupported search operator
Unsupported search operator '(subject:'. Supported: after, before, subject, from, to...
mail_messages_list q: "subject:receipt OR subject:invoice OR subject:renewal..."0 messages
{"messages": [], "result_size_estimate": 0}
mail_messages_list q: "receipt OR invoice OR renewal OR billed OR charged"0 messages
{"messages": [], "result_size_estimate": 0}
Thinking — giving up
It seems there aren’t any messages. The mailbox might be empty or the user is using a different account. I can’t access any outgoing payments or receipts from last year. I need to report this and create a note saying “No subscription price increases detected.”
notes_notebooks_list {}2 notebooks: Personal, Work
Personal (nbk_f3a70d51), Work (nbk_6c14e982)
notes_notes_create notebook_id: "nbk_f3a70d51" (Personal), title: "Subscription price changes"Note created in Personal notebook
Body: "No subscription payments or receipts were available in the checked period, so no price increases could be identified."
Wrong parameter killed the entire investigation. max_amount: 0 excluded every transfer. The model trusted the empty result instead of questioning its own filter. 90+ transfers sat untouched. Also wrote to the wrong notebook.
DB1: Note in wrong notebook, missing both price hikes
The rubric requires a note in the Work notebook identifying both Lumina Design Cloud (₹899 → ₹1,299) and Nimbus Notes Pro (₹249 → ₹349). The model created a note in the Personal notebook that says “no data found.”
R1: Final answer names zero subscriptions
The model told the user: “I couldn’t identify any subscription price changes because there were no subscription-like payments or receipts available.” Both hikes are clearly present in the payment ledger — just behind a filter the model never thought to remove.
Want environments like this for your agents?
Schedule a call →A full workspace rather than a single task: job descriptions, an applicant tracker, 77–78 resumes, a shared calendar, and four parties who each hold one piece of the answer — an external agency, the company recruiter, the hiring manager and the candidate. A sample of the scenarios, with what each model got wrong.
See the full list of endpoints
Schedule callDrive
See the full list of endpoints
Schedule callNotes
See the full list of endpoints
Schedule callSee the full list of endpoints
Schedule callPayments
See the full list of endpoints
Schedule callTerminal
Calendar
Traps live in recurrence expansion, DST boundaries and “free” slots that are only free in one attendee’s timezone.
See the full list of endpoints
Schedule callDocs & Sheets
Traps live in off-by-one ranges, formulas that silently return stale values, and edits that clobber someone else’s revision.
See the full list of endpoints
Schedule callCode & CI
Traps live in red CI that looks green on a stale SHA, merges that need a rebase, and force-pushes over someone else’s work.
See the full list of endpoints
Schedule callIssue Tracker
Traps live in illegal status transitions, duplicate tickets under different keys, and search syntax that quietly returns nothing.
See the full list of endpoints
Schedule callCRM
Traps live in duplicate accounts, contacts attached to the wrong parent, and stage moves that skip a required field.
See the full list of endpoints
Schedule callSupport Desk
Traps live in SLA breaches hidden behind pending status, macros that send before the agent reads the thread, and refunds issued outside policy.
See the full list of endpoints
Schedule callBrowser
Traps live in stale element refs after a re-render, modals that swallow clicks, and pages that look loaded before the data arrives.
See the full list of endpoints
Schedule callSQL Warehouse
Traps live in joins that fan out row counts, partition filters that silently drop months, and a query that costs more than the answer is worth.
See the full list of endpoints
Schedule callCloud & Incidents
Traps live in rolling back the wrong service, alerts that fire on a replica, and scaling a deployment that was already mid-deploy.
See the full list of endpoints
Schedule callE-commerce Ops
Traps live in refunding a line item that already shipped, return windows measured from the wrong date, and stock that reads available but is reserved.
See the full list of endpoints
Schedule callTravel Booking
Traps live in non-refundable fares that read cheapest, layovers too short to make, and holds that expire mid-conversation.
See the full list of endpoints
Schedule callATS / Recruiting
our home turf. Traps live in duplicate candidate records, stage moves that skip a required scorecard, and rejections sent to the wrong profile.
See the full list of endpoints
Schedule callHealth Records
synthetic patients only, no real PHI. Traps live in drug interactions, allergy flags on a merged chart, and lab values outside reference range that read normal.
See the full list of endpoints
Schedule callBack-office & Ledger
Traps live in entries that balance but post to the wrong period, reconciliations that ignore a pending settlement, and approvals above the agent’s limit.
See the full list of endpoints
Schedule callLogistics & Dispatch
Traps live in ETAs that ignore the driver’s shift end, reassignments that strand a half-loaded van, and delivery windows in the customer’s timezone.
See the full list of endpoints
Schedule callNeed a domain that isn’t here? Most of what we build is bespoke — tell us the API surface and the failure modes you care about.
Schedule a call →We find loss patterns in your model responses and build targeted adversarial datasets.
Diagnose
Run your model on our tasks and surface recurring failure patterns — recency bias, tool misuse, wrong parameter values, hallucinated state.
Generate
For each failure pattern, we build new adversarial tasks that specifically target that weakness — harder variants, new traps, different domains.
Train
Use the graded traces as RL training signal. Rubric verdicts give per-task reward. Failures become your highest-value training data.
Repeat
Re-run the improved model. New failures emerge. New tasks get built. The dataset grows where your model is weakest.
Most forward deployed solutions rely on sub-par contractors. We don't.
We scout and work exclusively with FAANG level engineers — whether full-time or contractors — to deliver best-in-class solutions. Weekday founders will be your “project managers”. Our 4 years building a people search company gives us an unfair advantage: we know how to identify exceptional talent and assemble custom teams tailored to your exact needs.
Elite Team
Weekday founders will work your team as project managers. For engineering, we handpick senior engineers from top tech companies who’ve built production ML systems at scale.
Proven Talent Scouting
Four years building a people search company means we have proprietary methods to identify, vet, and engage top-tier technical talent faster than anyone else.
Custom-Built for You
Every engagement gets a bespoke team assembled specifically for your problem domain, tech stack, and business requirements. No cookie-cutter approaches.
We work with most of the top AI training companies in the world and help them find top talent across various fields and geographies — as well as take on projects ourselves.
- You have a project and want us to work as a sourcing engine. Yes, we do that.
- You have a project you would like to hand over to us and let us handle all operations end-to-end. Yes, we do that as well.
Want environments built for your use case?
We design custom RL environments with the exact tools, traps, and rubrics your agents need to get better.
Schedule a call →