MOPD: GRPO training with LLM-as-a-judge and typed answers
MOPD aligns Qwen3.5-0.8B with TRL's GRPOTrainer, using an LLM judge that answers typed questions (a choice, a yes/no-style decision or a score on fixed levels) rather than writing free-text grades. Training problems are generated by DeepSeek and cached append-only. The implementation and tests are complete, but no training has been run yet, so there are no results to report.
"LLM as a judge" usually means asking a model to grade an answer in prose and then parsing a number out of it. MOPD asks the judge typed questions and gets typed answers back, so the reward is computed from a structured decision, not from text.
How it works
- Problems are generated by DeepSeek V4.1 Flash from a versioned template, validated, and stored in an append-only cache with stable content ids. Each problem is one of three kinds: a choice between candidates, a single decision, or a score on ordered levels.
- The student (Qwen3.5-0.8B) must answer in strict JSON. Parsing failures are categorised, not silently dropped.
- The judge gets a state and a set of typed questions: the reference question for the problem's kind, auxiliary checks on each sample with explicit tolerances, and a comparison across the group of samples.
- Rewards combine imitation of the judge's direct typed answer with the auxiliary checks, and feed TRL's
GRPOTraineras group-relative rewards. - Evaluation runs on a frozen fixture set: agreement with the judge, malformed-output rate, robustness to candidate order, Brier score, latency and cost.
Training is set up to run on Modal H100s with persistent volumes.
Why typed judge answers instead of free-text grades
- No parsing guesswork. The reward reads a typed field, so a judge that rambles cannot produce an accidental score.
- Checkable invariants. Every request and response pair is validated, so a malformed judge answer fails loudly instead of becoming noise in the reward.
- Order robustness is measured. Shuffling candidates should not change the decision, and the evaluation checks that it doesn't.
Facts
| Item | Detail |
|---|---|
| Student | Qwen3.5-0.8B |
| Trainer | TRL GRPOTrainer |
| Problem generator | DeepSeek V4.1 Flash through OpenRouter |
| Clients | Mock, replay and live, so tests and dry runs make no network calls |
| History | 39 commits; tests and fixtures throughout |
| Status | Implementation complete. No training has been run |
What you can buy
- The codebase, licensed non-exclusively from a clean export once it is listed.
- Judge-based reward pipelines built to your spec: typed judge questions, invariants, frozen evaluation sets and GRPO wiring.
FAQ
What does "LLM as a judge" mean here?
A second model decides how good the student's answer is, and that decision becomes the training reward. In MOPD the judge answers typed questions, so the reward comes from a structured field rather than from parsing prose.
Has MOPD been trained?
Not yet. The implementation and tests are complete, and no training run has been made, so there are no accuracy or alignment results to report.
Does running the tests call a paid API?
No. Tests and dry runs use mock and replay clients, and make no live model calls.
License it, or have me build an environment in the same style.