Prerequisites
- Java 21+
- Maven (the project includes
./mvnw)
Concepts
Dataset
A collection of items, each with a task description, “before” source state, and “reference” solution
AgentInvoker
Your agent — anything that takes a prompt + workspace and produces a result
Jury
One or more judges that score the agent’s output
AgentExperiment
Orchestrates: load items → invoke agent → evaluate → persist results
Step 1: Create a Dataset
A dataset is a directory with a manifest and per-item directories:before/ directory is the starting state. The reference/ directory is the correct answer. The agent never sees the reference — it’s used by judges for comparison.
The two Person.java files:
my-dataset/items/RENAME-001/before/src/main/java/com/example/Person.java
my-dataset/items/RENAME-001/reference/src/main/java/com/example/Person.java is the same file with fullName for name and getFullName for getName.
The dataset format is the same on 0.9.0 and on main.
Step 2: The agent, the jury and the run
- 0.9.0 (released)
- main (0.10.0-SNAPSHOT, not yet released)
pom.xml — experiment-core 0.9.0 from Maven Central:src/main/java/com/example/experiment/GettingStarted.java — in 0.9.0 one jury serves every item, and a judge reads the item’s workspace from its JudgmentContext:RENAME-001 passed=false and Passed 0 of 1 scored items.Using a real agent
Replace the lambda that editsPerson.java with your agent.
- 0.9.0: for Claude Code, use
ClaudeSdkInvokerfrom theexperiment-claudemodule (0.9.0; removed on main). - main: for Claude Code or Codex, use
AgentClientInvokerfromexperiment-agent-client, which takes its options from Agent Client.
Compare variants
Same dataset, a different configuration: build oneAgentExperiment per configuration and run each.
configB can add .knowledgeBaseDir(Path.of("knowledge")) and a prompt template that includes {{knowledgeRefs}}. Compare the two with DefaultComparisonEngine; see Creating Experiments.
What’s Next
Creating Experiments
Dataset design, variant ladders, and filter strategies
Building a Jury
Deterministic, structural and semantic judges, and cascades