Jury System - Pollack AI Lab

Task, Agent, Verifier

Agent Bench follows a Terminal-Bench-inspired separation:

A task defines instructions, optional workspace material, setup/post-processing, and metadata.
An agent changes the prepared workspace.
A verifier grades the result.

The verifier is an Agent Judge Judge or jury created by JudgeFactory. Agent Bench preserves Agent Judge’s Judgment, jury, tier-policy, and voting semantics. The 0.6.0 release retains deterministic judges, simple and cascaded juries, and the agents module’s Agent Client-backed LLM judge.

Cascaded Tiers

Policy	Behavior
`REJECT_ON_ANY_FAIL`	Stop the cascade when any check fails
`ACCEPT_ON_ALL_PASS`	Continue when all checks pass
`FINAL_TIER`	Use the last tier’s aggregated result as the verdict

jury:
  tiers:
    - name: build
      policy: REJECT_ON_ANY_FAIL
      checks:
        - type: maven-build
          goals: [clean, test]
    - name: test-quality
      policy: FINAL_TIER
      checks:
        - type: test-quality-llm
          prompt: prompts/judge-practice-adherence.txt
          model: claude-sonnet-4-6

Registered Judge Types

Type	Implementation	Purpose
`file-exists`	Agent Judge	Verify a workspace path exists
`file-content`	Agent Judge	Compare file content exactly or by containment
`maven-build`	Agent Judge `BuildSuccessJudge`	Run Maven goals and require success
`coverage-preservation`	Agent Judge	Require coverage not to regress from the baseline
`coverage-improvement`	Agent Judge	Require coverage to meet a threshold
`test-quality-llm`	`agent-bench-agents`	Score practice adherence with a real Agent Client-backed LLM judge

The core CLI uses deterministic Agent Judge implementations and an abstaining LLM registration for end-to-end execution. The agents-module CLI registers TestQualityJudge for LLM-graded run and resume operations.

Custom Judges

Custom judges implement Agent Judge’s Judge interface and can be registered without changing the benchmark format:

public final class MyJudge implements Judge {
    @Override
    public Judgment judge(JudgmentContext context) {
        return Judgment.builder()
            .pass()
            .reasoning("Workspace satisfies the custom rule")
            .build();
    }
}

JudgeFactory factory = new JudgeFactory();
factory.register("my-check", config -> new MyJudge());

Benchmark and Task YAML

schema: bench.benchmark.v1
name: my-benchmark
version: "1.0"
default-timeout: PT10M
jury:
  checks:
    - type: file-exists
      path: expected.txt

schema: bench.task.v1
id: my-task
difficulty: easy
instruction: |
  Create expected.txt.
timeout: PT10M
metadata:
  key: value
setup:
  - "command before the agent"
post:
  - "command after the agent"

Setup, post-processing, configured agents, and process-based judges execute locally with the invoking user’s permissions. A task workspace is not a security boundary; externally isolate untrusted definitions or agents.