FasterLabelingEnvironment
Docker image pinned by digest, every source and wheel by SHA-256. The prompt is the only instruction. No network except the model's API.
For data vendors and brokers
Coding RL environments, built to order for the labs you sell to.
You hold the customer relationship. We build the hard coding tasks: a Docker environment, a deterministic grader, an oracle solution, and the grader logs that show which frontier models fail. Delivered in your format, calibrated to the pass rate your customer asks for, non-exclusive by default.
What ships
A finished task, not a task idea.
Every environment arrives as software the lab runs itself. Nothing in it needs us to be present.
Grader
A program with a hidden answer key, never an AI judge. It runs on a hash-checked runtime outside the image, so package changes or tampering cannot change the grade.
Oracle solution
A reference solution that scores 1, derived from the pinned inputs alone. It proves the task is solvable and the key is right.
Evidence folder
For every model run: the grader log, the clean verifier log, the reward, and the model's own deliverable. Your customer can audit the failure before buying.
Format: Terminal-Bench / Harbor by default, NeMo Gym on request, or your own harness. Domains so far: software engineering on real libraries, ML infrastructure, inference numerics, hardware and EDA, terminal work.
Evidence
Frontier runs on our public showcase set.
Three tasks on the samples page, graded all-or-nothing over hidden items. Each frontier model got two hours of agent time per run in the task's own container. The sets we send to vendors are separate from this showcase and have not been shown to any lab.
| Task | Model | Runs | Outcome |
|---|---|---|---|
| never-learned-replies310 hidden plans | Opus 5.5 | 3 | fail · 4, 20 and 12 / 310 wrong |
| Opus 5.0 | 1 | fail · 16 / 310 wrong | |
| Fable 5.1 | 2 | fail · hit the 2 h limit, 57 and 125 / 310 wrong | |
| attention-mask-fill-signoff40 deployments | Opus 5.5 | 2 | fail · 10 / 40 wrong, both runs |
| Opus 5.0 | 1 | fail · 10 / 40 wrong | |
| ram-boot-image256 hidden queries | Opus 5.5 | 2 | fail · 38 and 9 / 256 wrong |
| Fable 5.1 | 2 | fail · 9 and 38 / 256 wrong |
hidden data verified (310 plans) in 0.6 s driver ready (9 package files) plan p023 (density): FAIL wrong answer (expected [hidden], got q186085086086086076086/77777777777777777) plan p037 (density): FAIL wrong answer (expected [hidden], got q158730/1111111) ... density: 80 plans, 4 failed reward: 0
FAIL xc7023: wrong class: got FF, expected [hidden] FAIL xc7035: wrong cells, class, init: got FF/0 cells, expected [hidden] FAIL ice40013: wrong class: got FF, expected [hidden] ... items 256, failed 9; 5.1 s reward: 0
Terms
Built to order. Non-exclusive by default.
Your customer, your name
We supply you. The lab sees your deliverable in your format. We do not contact your customers.
Calibrated to the order
You name the domain and the model that should fail, or the pass-rate band for training. We measure on that model and ship the runs.
Non-exclusive, or fresh
Tasks are licensed non-exclusively by default. When a customer wants exclusivity, we build fresh tasks for that customer alone.
Clean provenance
No personal data, no client code, all rights ours. The only third-party code is unmodified open-source releases, pinned by hash and downloaded at image build time.
Start with an evaluation set.
Tell us which labs you supply and which domain they are buying. We send a two-task evaluation set by private link, with grader logs and the model's deliverables, so your team can judge the work before any order. Then a pilot batch built to your spec.
- Format
- Terminal-Bench / Harbor, NeMo Gym, or yours
- Based
- United States