One coding task, one ML-numerics task, one hardware task. Each has a Docker environment, a deterministic grader with a hidden answer key, and a reference solution that scores 1. We ran Opus 5.5, Opus 5.0, Fable 5.1 and Muse Wonder against them for two hours each. Every run scored 0, and the same items failed across models.
Each task is graded all-or-nothing over its hidden items. "Wrong" is the number of items the grader rejected. Full per-run logs ship with the tasks.
| Task | Model | Runs | Outcome |
|---|---|---|---|
| never-learned-replies310 hidden plans | Opus 5.5 | 3 | fail · 4, 20 and 12 / 310 wrong |
| Opus 5.0 | 1 | fail · 16 / 310 wrong | |
| Fable 5.1 | 2 | fail · hit the 2 h limit, 57 and 125 / 310 wrong | |
| attention-mask-fill-signoff40 deployments | Opus 5.5 | 2 | fail · 10 / 40 wrong, both runs |
| Opus 5.0 | 1 | fail · 10 / 40 wrong | |
| Muse Wonder | 1 | fail · 10 / 40 wrong | |
| Muse Wonder Full | 1 | fail · 12 / 40 wrong | |
| ram-boot-image256 hidden queries | Opus 5.5 | 2 | fail · 38 and 9 / 256 wrong |
| Fable 5.1 | 2 | fail · 9 and 38 / 256 wrong |
Setup for every run: the task's own container with the agent added, the prompt as the only instruction, no network except the model's API, 2 hours of agent time. Tasks are in Terminal-Bench / Harbor format.
Coding / ML infra. A nightly fine-tuning job trains on an endless chat log. From the sources of TRL, transformers and tokenizers alone (not installed, not runnable), decide which replies some schedule can never learn. Exact integers and fractions over 310 held-out plans.
ML numerics. For 40 CPU transformer deployments, give the exact interval of additive-mask fill values that keeps every output finite and the leak within tolerance, from what the pinned PyTorch kernels actually compute in fp16 and bf16.
Hardware / EDA. From the Yosys 0.70 source alone (no synthesis tool, no compiler), predict for RAMs and ROMs coded in many Verilog styles which cells four FPGA flows create and the exact INIT contents of each. 256 hidden queries, hash-verified.
Excerpts from the grader logs, as shipped. Expected values are hidden here. The tasks come with the full logs and the model's deliverable for every run.
hidden data verified (310 plans) in 0.6 s driver ready (9 package files) plan p023 (density): FAIL wrong answer (expected [hidden], got q186085086086086076086/77777777777777777) plan p037 (density): FAIL wrong answer (expected [hidden], got q158730/1111111) plan p167 (density): FAIL wrong answer (expected [hidden], got q461760033189/11111111111111) ... first_risk: 70 plans, 0 failed risk_count: 80 plans, 0 failed count: 80 plans, 0 failed density: 80 plans, 4 failed reward: 0
FAIL xc7023: wrong class: got FF, expected [hidden] FAIL xc7030: wrong class: got FF, expected [hidden] FAIL xc7035: wrong cells, class, init: got FF/0 cells, expected [hidden] FAIL ice40013: wrong class: got FF, expected [hidden] FAIL ice40047: wrong class: got FF, expected [hidden] ... xc7 items 64, failed 3 ice40 items 64, failed 3 ecp5 items 64, failed 2 gw1n items 64, failed 1 items 256, failed 9; 5.1 s reward: 0
Base image by digest, every source and wheel by SHA-256. The answer is a function of fixed bytes, and the reference solution derives it from those bytes alone.
The grader never enters the image. It runs on a hash-checked runtime, and the agent's code runs as an unprivileged child. Package changes or tampering cannot change the grade.
The library or tool that decides the answer is not installed and cannot be built. No network. Guessing, bounding or conservative answers fail.
These three are tuned as frontier-breaking evals. For training we deliver tasks inside a target pass-rate band, measured on the model you choose, with per-item or dense reward if you prefer.
We send the full package by private link: Docker environments, graders with answer keys, reference solutions, and every model run with its grader log and deliverable. One email with your name and lab is enough. We can also build one task to your spec as a trial.
FasterLabeling