Skip to main content
This page builds one experiment end to end: a one-item dataset, an agent, a jury, and the result. The agent here edits the workspace directly so the example runs offline; a real invoker calls an agent at that point. Both programs below were compiled and run, each with a negative control.

Prerequisites

  • Java 21+
  • Maven (the project includes ./mvnw)

Concepts

Dataset

A collection of items, each with a task description, “before” source state, and “reference” solution

AgentInvoker

Your agent — anything that takes a prompt + workspace and produces a result

Jury

One or more judges that score the agent’s output

AgentExperiment

Orchestrates: load items → invoke agent → evaluate → persist results

Step 1: Create a Dataset

A dataset is a directory with a manifest and per-item directories:
dataset.json — the manifest:
item.json — per-item metadata:
The before/ directory is the starting state. The reference/ directory is the correct answer. The agent never sees the reference — it’s used by judges for comparison. The two Person.java files: my-dataset/items/RENAME-001/before/src/main/java/com/example/Person.java
my-dataset/items/RENAME-001/reference/src/main/java/com/example/Person.java is the same file with fullName for name and getFullName for getName. The dataset format is the same on 0.9.0 and on main.

Step 2: The agent, the jury and the run

pom.xml — experiment-core 0.9.0 from Maven Central:
src/main/java/com/example/experiment/GettingStarted.java — in 0.9.0 one jury serves every item, and a judge reads the item’s workspace from its JudgmentContext:
Compile and run:
Output:
With an agent that changes nothing, the same program prints RENAME-001 passed=false and Passed 0 of 1 scored items.

Using a real agent

Replace the lambda that edits Person.java with your agent.
  • 0.9.0: for Claude Code, use ClaudeSdkInvoker from the experiment-claude module (0.9.0; removed on main).
  • main: for Claude Code or Codex, use AgentClientInvoker from experiment-agent-client, which takes its options from Agent Client.
The agent-experiment-template is pinned to 0.6.0 and is not yet updated for these versions.

Compare variants

Same dataset, a different configuration: build one AgentExperiment per configuration and run each.
configB can add .knowledgeBaseDir(Path.of("knowledge")) and a prompt template that includes {{knowledgeRefs}}. Compare the two with DefaultComparisonEngine; see Creating Experiments.

What’s Next

Creating Experiments

Dataset design, variant ladders, and filter strategies

Building a Jury

Deterministic, structural and semantic judges, and cascades