A taskset is only as good as its grader. This one was built grader-first: every task has rules precise enough to check from the outside, and the verifier only ever touches what a user could.
The shape of a task
Every task shares one contract, then adds its own rules:
- One standalone HTML file: inline CSS and JavaScript, no libraries, no network, no build step.
- Working state, not a mockup: real calculations and state transitions from given initial data.
- Usable at 360 px and 1280 px without page-level horizontal scrolling.
- Keyboard-only and accessible: visible labels, accessible names, visible focus, specific errors tied to their fields, results announced through a status region, no blocking browser dialogs.
- Modals done properly: named, focus moved in and trapped, Escape closes, focus restored.
The task rules are exact enough to grade. For example, a parcel tariff task fixes the base price, the per-500 g step, the zone multipliers and the express surcharge, and asks for each component and the total. A room-booking task fixes opening hours, 15-minute steps, capacities and the rule that intervals are start-inclusive and end-exclusive, so adjacent bookings are allowed.
How the verifier grades
- It drives the app in Chromium through rendered controls and the visible DOM only. It never imports or calls the app's functions.
- It checks behaviour at both viewport widths.
- It classifies failures into three kinds, so a broken run is never scored as a wrong answer: - binding: the verifier couldn't find a control it needs; - infrastructure: the browser or runtime failed; - assertion: the app did the wrong thing.
- Each output is fingerprinted with SHA-256, so a report always points at the exact file it graded.
Facts
| Item | Detail |
|---|---|
| Tasks | 50 web-app tasks under one shared contract |
| Verifier | Playwright and Chromium, about 380 lines; covers a first set of 5 tasks so far |
| Runs | Dry runs and runtime evaluation on Modal (Sep 2026) |
| Status | A working prototype of the shape, not a product for sale |
What you can buy
Tasksets in this shape, written by a person to your spec:
- tasks with exact, checkable rules in your domain;
- a programmatic grader that works through the interface, not the code;
- failure classes that keep infrastructure noise out of the score;
- delivered as HUD tasksets with their environment, or as a repository with a runner.
FAQ
What makes a task gradable?
Rules exact enough that a program can decide pass or fail from the outside: fixed initial data, fixed arithmetic and rounding, and explicit edge cases such as adjacent bookings or zero rows.
Why grade through the UI instead of unit tests?
Because the model is free to structure its code however it likes. Grading through rendered controls checks what a user would experience, and works for any implementation.
Are the 50 tasks for sale?
No. The prompts were drafted with an assistant, so I don't license them. I write tasksets like this by hand, to order.