Why a Jury?
A single judge gives you a single score. A jury gives you diagnostic information — when something fails, you know where in the stack it failed and why. The experiment driver uses a cascaded jury with three tiers. Each tier is more expensive than the last. ACascadedJury runs a later tier only when the earlier tier’s policy lets it continue; a SimpleJury runs every judge.
The Three Tiers
1
Tier 1: Deterministic
Zero-cost, instant, binary. Checks facts that are unambiguously right or wrong.Examples: Does the project compile? Does
java -version report the right version? Are all javax.* imports replaced with jakarta.*?Cost: Free (no LLM calls)2
Tier 2: Structural
Compares the agent’s output against the reference implementation at a structural level — AST diffs, import sets, annotation changes, POM dependency trees.Examples: Are the same imports present? Do method signatures match? Are the right dependencies in the POM?Cost: Free (structural comparison, no LLM)
3
Tier 3: Semantic
LLM-powered evaluation for questions that can’t be answered structurally. Uses criteria extracted from the execution plan to judge whether the agent’s approach was sound.Examples: Is the error handling strategy appropriate? Does the migration preserve business logic semantics?Cost: LLM tokens per item
Wiring a Simple Jury
Start with a single Tier 1 judge:Wiring a Multi-Tier Jury
Add judges from each tier with weights:FileExistsJudge is the custom judge from Getting Started. This block compiles against 0.9.0.
Weights shape the vote; they do not order execution. To stop before an expensive tier, use CascadedJury.
Stopping early with a cascade
A cascade runs its next tier only when the earlier tier’s policy lets it continue. Both blocks compile against their version:MajorityVotingStrategy is in io.github.markpollack.judge.voting, RoutingRule in io.github.markpollack.judge.verdict, and TierPolicy is gone; plan is the item’s plan text.
Writing a Custom Judge
ImplementJudgeWithMetadata, a Judge that also describes itself. This compiles against 0.9.0; it is named MavenCompileJudge so it does not shadow Agent Eval’s own BuildSuccessJudge:
Judge interface
JudgmentContext provides
Agent Eval 0.17.0’s
JudgmentContext is (goal, workspace, executionTime, startedAt, agentOutput, status, error, metadata). It has no reference-path accessor.
Judgment fields
Agent Eval 0.17.0 keeps outcome, normalized score, and label as independent facts. A Boolean outcome
uses
Judgment.pass(...) or Judgment.fail(...); it does not duplicate the same fact in a score
object. ABSTAIN, NOT_APPLICABLE, and ERROR contribute no effective score.
Agent Eval 0.18.0 and later have no JudgmentContext; see the main-only note.
On main (0.10.0-SNAPSHOT, not yet released) a jury is built per item from
AgentJudgingInput, judges implement Judgment judge(), Judgment lives in judge.judgment, and voting strategies in judge.voting. See the main tab of Getting Started. None of this is in 0.9.0.Related
Four-Tier Jury Methodology
The evaluation framework behind experiment scoring
Creating Experiments
Dataset design, variant ladders, configuration