Skip to main content
This page’s examples describe released Agent Judge 0.17.0 and the fundamentals reactor. For configured Agent Eval 0.18 execution, use the requirements walkthrough and current API reference.
A JUnit test has an oracle. A judge is the oracle.
You wrote expected in assertEquals(expected, actual) because you knew the answer, so the check was checked — by a human, at authoring time. A judge has no expected. It produces one. Nothing checks the checker. That asymmetry is the reason this page starts with design rather than with extension points:

​
Six questions before you write code

​
1. What one claim does this judge evaluate?

If you cannot state it in a sentence, it is two judges. “Is this good?” cannot be debugged. “Does every critical finding cite a file and line?” can.

​
2. What evidence would convince you?

Write the evidence down before the implementation. It becomes the judge’s Check list, and the list is what anyone reads when the judgment is a FAIL.

​
3. Is the oracle known, derived, or judgment?

Most criteria that feel like they need a model do not.

​
4. Choose the least interpretive instrument that can reliably answer it

Agent Judge does not replace JUnit, AssertJ, ArchUnit, Checkstyle, JaCoCo, JApiCmp, or the compiler. Reach for a model only for what is left over after those.

​
5. Keep distinct failure criteria visible

Aggregate at read time, never at write time. An aggregate is cheap to recompute from parts. Parts cannot be recovered from an aggregate.
Every per-item result computed on the way to a verdict is evidence and must be persisted. That is what Check is for. A judgment carrying its parts can be re-aggregated under a different rule for free; one carrying only a number needs a re-run of the agent, not just the judge. Report the binding item — which criterion, which entry, which input decided the verdict. A mean cannot produce it, and it is the part anyone acts on.

​
6. Only now, pick the mechanism

Start with a lambda. A plain lambda has no discoverable identity, so wrap it with Judges.named as soon as anything but the call site reads its result.

​
Return a status-first result

For a Boolean policy, return Judgment.pass(reasoning), Judgment.fail(reasoning), or begin with Judgment.verdict(boolean) when you also need checks or metadata. Do not store 1.0 or 0.0 merely to repeat the status. PASS already carries that fact, and storing it twice creates two places that can disagree. For a real measurement, begin with Judgment.scored(normalizedScore) and declare the passing threshold. For a categorical policy, set a label in addition to the required status. Use Judgment.abstain(reasoning) when the question applies but remains undecided, including when required evidence is missing. Use Judgment.notApplicable(reasoning) when the question should not have been asked, and declare the exclusion condition in advance. Use Judgment.error(reasoning) when evaluation could not complete — logging the originating exception where it was caught.

​
Scores: measure, never invent

A judge may report a number it computed. It must not report a number it made up. Do not ask a model-backed judge for a score. “Conformance: 0.85” has no procedure behind it, and nobody can say what would have made it 0.86. It looks like a measurement and is an impression wearing one. Let the model assess, and let the code count and decide. Splitting those two jobs is measurable. A judge asked to invent its criteria, apply them, and state a verdict returned both PASS and FAIL on identical input across four runs. Supplying the criteria and doing the rollup in code produced an identical criteria set, verdict, and binding criterion across five consecutive runs. Five runs is a small sample, and it is a measurement rather than an impression. A criteria-based judge does have a legitimate number, and it is a count. If the judge assessed five named criteria and four were met, then “4 of 5” is a real measurement: computed from parts the judge recorded, reproducible, and auditable back to the criterion that failed. Record each criterion as a Check and derive the count when you read it rather than storing the total. An aggregate is cheap to recompute from its parts, and parts cannot be recovered from an aggregate. The part you lose is the one you act on, which is the criterion that decided the verdict.

​
The count reports, it does not gate

Keep the gate conjunctive. Four of five is not eighty percent of done, and a strong criterion must not pay for a weak one. The count earns its place elsewhere. It is the only thing that shows movement between runs. PASS and FAIL cannot tell “one gap remaining” from “ignored the pattern entirely”, so with a Boolean alone you cannot tell whether a change to your agent improved anything. Four of five becoming five of five is a signal. FAIL becoming PASS is the same signal arriving too late to be useful.

​
A mean can pass a zero

The failure is not theoretical. A judge that averages per-criterion scores and takes its verdict from the average will pass work that missed a criterion entirely, whenever enough sibling criteria score well enough to pay for it. If your rubric names a per-criterion acceptability line, a mean contradicts your own rubric. Two things make this hard to see from inside. The number looks like evidence, so nobody asks what it averaged. And a test written against the buggy behavior will assert the pass as correct, so the fix arrives looking like a regression.
ERROR is not a low score. It is not a number at all. A judge that could not run has not rejected the subject; scoring it zero blames the subject for a missing input, and averaging that zero into anything is worse.

​
Where the pass mark comes from

0.5 is not an answer, and neither is “it looked about right.” In order of preference:
  1. The rubric’s own level semantics. If the rubric says “raise a concern below 2”, the bar is 2. It is already written down.
  2. Standard setting (Angoff, Bookmark) — a mature field for deriving a cut score from expert judgement about content.
  3. An explicitly provisional number, marked as such at the call site, with a date.
Put the derivation on the constant, not on the method that uses it. That is where the next person changing the value will be standing.
Bar-fitting. Setting the threshold just under what a known-good run scored is fitting the bar to the data it must judge. It will always pass, and you will never know.

​
Degeneracies to check before trusting a judge

Most of these cost nothing to run.
  • Enumerate the inputs on which it cannot return FAIL. Not “can it fail?” — which inputs cannot. Empty strings, ., an absolute path, ../escape, and a directory where a file was expected are the usual five.
  • allMatch / anyMatch over a possibly-empty stream. allMatch over empty is true: a check that passes because it examined nothing.
  • State how many references were checked and how many resolved. "0 of 0 references resolved" must not render as "all references resolve".
  • Path escape. resolve() returns an absolute argument unchanged, so a path can leave the workspace. That is a misconfigured judge — ERROR, not FAIL.
  • Checks vs. status. A judgment carrying a failing Check must not be PASS.
  • Dated assets. A pinned model id is a dated asset. Scan for retired ones.

​
Test the five outcome paths

Assert status(), never pass(). pass() is false for FAIL, ABSTAIN, NOT_APPLICABLE and ERROR alike, so a test written on it cannot tell a rejection from a judge that never ran. For a judge that declares an exclusion condition, test both the applicable and excluded cases; an undeclared exclusion must not be accepted.
Write the verdict tests from the criterion, never from the code. A test written from the code’s behaviour cannot disagree with the code.
Include a negative control: feed the judge a subject you know is bad and assert FAIL. A judge with only one reachable verdict is a constant, and a suite that never watches the other one cannot tell the difference.
When you make a judge more honest — returning ABSTAIN or ERROR where it previously returned a score — fix its consumer in the same change. A consumer branching on !pass() will record the honest abstention as a rejected subject: the judge stops lying and the system starts blaming the subject.

​
Compiled examples

​
Compose a model-backed judge

ModelBackedJudge keeps three concerns independent.
  1. JudgePromptTemplate renders evaluation inputs from a context.
  2. JudgeModel invokes a backend.
  3. JudgmentClassifier converts the response into a normalized judgment.
LabelJudgmentClassifier requires an explicit label-to-status policy. Unknown labels produce a reasoned ABSTAIN with the raw output in metadata and neither a label nor a score — not a guess, and not a default PASS. Production adapters include SpringAiJudgeModel and AgentClientJudgeModel.

​
Keep metadata portable

Judgment metadata may contain only portable primitives, arrays, and string-keyed objects. Convert SDK objects to stable fields before attaching them, and keep live exceptions and framework-native objects in logs or JudgmentContext, not in the result.