Getting Started with Agent Judge

Agent Judge evaluates an agent execution after it runs. A Judge receives a JudgmentContext and returns a Judgment whose required status is independent from its optional normalized score and optional label.

Prerequisites

Java 21 or newer
Maven 3.9 or the Maven Wrapper
Credentials only when you choose a live LLM-backed judge

The canonical tutorial modules use deterministic inputs and require no credentials.

Add the dependency

<dependency>
  <groupId>io.github.markpollack</groupId>
  <artifactId>agent-judge-core</artifactId>
  <version>0.15.0</version>
</dependency>

All ten artifacts use the same group and version.

Artifact	Add it for
`agent-judge-core`	Core contracts, status-first results, file checks, and juries
`agent-judge-ai-core`	Framework-neutral model-backed judges
`agent-judge-exec`	Command, build, and coverage checks
`agent-judge-file`	Semantic file comparison
`agent-judge-llm`	Spring AI as a judging backend
`agent-judge-rag`	RAG quality judges
`agent-judge-spring-ai`	Evaluate Spring AI output
`agent-judge-langchain4j`	Evaluate LangChain4j output
`agent-judge-koog`	Evaluate Koog output
`agent-judge-agent-client`	Evaluate or judge through AgentClient

The framework bridge dependencies are optional at the application boundary. Your application supplies the framework runtime version it uses.

Build a context and run one judge

This excerpt is compiled as tutorial module 01.

Path workspace = Path.of("test-workspace");

JudgmentContext context = JudgmentContext.builder()
    .goal("Add a HelloController class")
    .workspace(workspace)
    .status(ExecutionStatus.SUCCESS)
    .startedAt(Instant.now())
    .executionTime(Duration.ofSeconds(5))
    .build();

Judge fileJudge = new FileExistsJudge(
    "src/main/java/com/example/HelloController.java");

Judgment result = fileJudge.judge(context);

Read result.status() first. result.score() and result.label() may be null because a Boolean decision does not duplicate its outcome as a stored number or category. Use result.effectiveScore() only when an aggregation algorithm deliberately needs a numeric view of PASS or FAIL.

Understand result portability

A result can carry strings, booleans, interoperable integers, finite numbers, arrays, and string-keyed objects in metadata. Agent Judge recursively copies and freezes those values. Live exceptions, SDK responses, Duration objects, enums, and other Java identities do not belong in result metadata. JudgmentContext is an in-process input and may hold framework-native objects needed by a judge. Judgment is the portable output boundary.

Run the tutorial

Clone https://github.com/markpollack/agent-judge-tutorial.git, enter the checkout, run ./mvnw clean test, and then run ./mvnw exec:java -pl module-01-single-judge. The tutorial contains ten modules and progresses through composition, juries, custom judges, model-backed evaluation, Koog, and LangChain4j.

Tutorial

Run every compiled sample

Design Philosophy

Learn the normalized result invariants

Built-in Judges

Choose a judge family

Jury System

Aggregate multiple judgments

Projects

AgentWorks

Supporting Projects

Migration

Getting Started with Agent Judge

Prerequisites

Add the dependency

Build a context and run one judge

Understand result portability

Run the tutorial

Next

Tutorial

Design Philosophy

Built-in Judges

Jury System

​Prerequisites

​Add the dependency

​Build a context and run one judge

​Understand result portability

​Run the tutorial

​Next

Tutorial

Design Philosophy

Built-in Judges

Jury System

Prerequisites

Add the dependency

Build a context and run one judge

Understand result portability

Run the tutorial

Next