This guide describes the Agent Eval 0.18.0 API. Maven coordinates and Java packages still use
agent-judge and io.github.markpollack.judge.EarsRequirement.from(path, revision) and Rfc2119Requirement.from(path, revision) read the selected document during setup and retain its SHA-256 source reference and native item IDs.
Select your roster before executing. The PetClinic path keeps UC6-AC7 through UC6-AC12 (six), the full UC6 document (52), and the architectural MUST roster (13).
Configure the runtime
EvalModel specializes EvalRuntime<EvalModelRequest,EvalModelResponse> for generated answers.
SpringAiEvalModel adapts Spring AI; AgentClientEvalModel adapts a native agent client.
Configure credentials, workspace, permissions, tools, deadlines, and native options on the backend before constructing the Judge or Jury.
An investigative harness needs actual investigation capability. A prepared-only harness can declare withInputs(GeneratedInput.PREPARED_EVIDENCE); requesting investigation then fails before calls.
Request roles, options, and metadata must be transmitted honestly or refused before invocation. SDK capture does not promise raw HTTP bytes. Cancellation propagates.
Invocation counts
A generated investigative roster enters the configured backend once for the selected whole roster. That backend may make several internal model or tool calls. One Agent Eval invocation does not imply one HTTP request, turn, or tool operation. The root Verdict owns the shared Invocation; each requirement result references its ID rather than repeating its usage. Structured roster runtimes execute separately for each item: EARS6 means six executions, EARS52 means 52, and RFC13 means 13. Jev uses this typed path; its per-item counts must not be described as one generated batch call.InvocationRecords.of(verdict) resolves retained owners and counts each ID once, including lower invocations on refused originals.
Unknown usage or cost stays absent; an omitted field is not zero.
Run a prepared example
This complete example uses synthetic answers and citations to test plumbing. It makes no model call and establishes no claim about the real application. Create arequirements-demo directory. Save the POM below there, and save the Java block as
requirements-demo/src/main/java/RequirementsDemo.java. The dependencies resolve from Maven Central.
requirements-demo directory with Java 21 and Maven 3.9 or later.
The first run downloads the dependencies and plugins; the example uses deterministic responses and performs no live inference.
-o option to run offline.
Expected output:
compositeAttempts.
One ABSTAIN leaves this roster INCONCLUSIVE. A verified violation produces FAIL; omission, malformed identity, or unjustified N/A prevents a passing audit.
Applicability must be declared before execution. An excluded original remains visible with its treatment.
A native roster establishes required coverage; an ordinary voting panel combines opinions. AllEligiblePassStrategy is an opinion rule, not a substitute for roster coverage.
Requirement-first assertions
assertThat(actualRequirement).judgedBy(recipe).withEvidence(actualEvidence).isSatisfied() passes the actual Requirement into typed construction.
Ready Judge/Jury assertions use isPassed(); evaluate() returns the complete reusable EvaluationResult.
Repeated terminals on the same live stage cache the result or thrown failure. Calling judge() or vote() separately is a fresh execution.
Retained assertions inspect Verdict/EvaluationResult and have no withPolicy terminal. Explicitly call Evaluations.apply(retainedVerdict, policy) only when requesting a new policy decision.
Continue with retained results and the executable PetClinic path.