install step is needed once: later modules reuse judges written in earlier ones, and exec:java resolves those from the local repository.
Run every module from the repository root — the demos resolve test-workspace relative to where Maven was launched.
No module needs an API key or a live model endpoint.
The first build downloads Maven dependencies; prepare the caches before attempting an offline run.
PetClinic case study
The configured PetClinic path uses Agent Eval 0.18.0 from Maven Central. Open the normal tutorial checkout at2ad2b03c85e63e00324d18f7f57e61a3b0ad4b3f and
CONFIGURED-RUN.md and
the presenter runbook
for setup and commands; --isolated is optional.
Each generated roster invokes its configured backend once; an agent backend can internally call
models and tools several times. Structured controls execute once per item. The model-backed demos
replay unchanged archived answers. Module 01 runs a real build with prepared caches.
The case-study guide explains requirements, runtime setup,
evidence, complete retained results and separate application policy. Producer AssertJ regression
tests assert the archived populations; explicit merge gates intentionally reject EARS52 and RFC13.
Prepared Judge and Requirement-first AssertJ examples exercise the same public API.
Module 06 remains unimplemented. The unchanged DRY-RUN.md is historical 0.17 evidence.
The fundamentals subject
Every fundamentals module judges the same change.test-workspace is a small Maven project whose HelloController delegates to a service and returns a record — the codebase’s idiom.
An agent was asked to expose a sales report endpoint and added ReportController, which opens its own JDBC connection, concatenates SQL from request parameters, builds JSON by hand, swallows the exception, and closes nothing.
It compiles. It is in the right package. It is named correctly. It has the requested method.
The fundamentals sequence
The modules are ordered by the evaluation problem each one creates, not by the next API.
A twelfth module,
judge-junit, is a small bridge turning an Agent Judge outcome into a useful JUnit assertion failure.
Three kinds of oracle
The sequence exists to make one distinction concrete. Known. An exact or deterministic answer exists — compilation succeeds, tests pass, a file exists, API compatibility holds, coverage clears a threshold. Use ordinary Java tooling or a deterministic judge. Modules 01–03. Derived. No single expected value exists, but several objective facts settle the question — a class is where the layout says it should be, a patch touched only allowed files, coverage improved without deleting tests. Still deterministic, just not one assertion. Module 05. Judgment. The property itself requires interpretation — does this fit the conventions of this codebase, did the review identify the important risks, is the proposed API sensible, did the implementation satisfy the user’s intent. This is where a model-backed judge earns its existence. Modules 01 and 07.The fundamentals live path
Four modules, roughly ten minutes, in directory order:What to observe
Module 01 keeps four criteria in plainassertTrue on purpose.
Agent Judge appears only for the criterion that has no expected value, and the judgment it returns carries one Check per criterion rather than an average — five failures out of six do not become 0.17, because a number cannot name the criterion that binds.
Module 03 separates three things a score normally collapses: the measurement (100.00% → 36.00%), the normalized score (0.0, which floors and so cannot distinguish a 1-point drop from a 64-point one), and the status (FAIL, against a bar written down with its derivation).
Pointed at a workspace where no build ran, the judge returns ERROR — not FAIL, and not 0.0.
Module 06 is the distinction the library most needs stated.
Five criteria are not five opinions about one thing, so 3 of 5 = 0.60 is the wrong rule.
It also counts invocations to show that Judges.allOf short-circuits after 3 of 5 judges, and runs the conjunction over an empty list to show that allMatch returns true there.
Module 08 is the case where a jury is correct: three reviewers independently estimating one property, disagreeing two to one.
Majority returns FAIL; consensus returns ABSTAIN, because a split vote is indeterminate rather than negative.
Module 09 runs one panel under all four error policies.
Three of the four report PASS for a run in which a required criterion was never evaluated.
It also distinguishes ABSTAIN (an applicable question left undecided) from NOT_APPLICABLE (a declared exclusion); an undeclared exclusion in a jury becomes ERROR.
Modules 10 and 11 execute real adapter code with deterministic framework objects, so bridge construction and metadata extraction stay verifiable offline.
Credential-free model-backed judges
Modules 01 and 07 use a fixtureJudgeModel that supplies fixed replies.
The source says so rather than dressing the fixture up as a recording: a fixture cannot tell you a judge is right, only that the wiring around it is.
Everything else on the path is real — the prompt is rendered from the actual JudgmentContext, the classifier parses free text into a Judgment with Checks, and swapping the model for SpringAiJudgeModel is a one-line change.
An unrecorded subject returns ERROR rather than a default PASS.
A stand-in that approves what it has not seen is a judge that cannot fail.
Framework versions
Continue
Writing Custom Judges
Design an oracle before choosing a mechanism
Jury System
Requirements versus estimates