Agent Experiment

What’s New → Current release: 0.7.1 · Java 21 · three modules

Overview

Agent Experiment provides a repeatable lifecycle for agent evaluations: load versioned fixture or Git-backed datasets, provision isolated workspaces, invoke an agent, judge results with Agent Judge juries, persist structured evidence, and compare variants. experiment-core has no direct agent-SDK integration. experiment-claude adds Claude Code SDK invocation, planning, and semantic evaluation, while experiment-workflow adapts typed Agent Workflow executions and journaled step costs to the common AgentInvoker contract. The framework records source and dataset revisions, dirty state, configuration, knowledge hashes, workspaces, results, costs/tokens, and journals. Exact replay still depends on callers pinning the external model, CLI, tools, network services, and other nondeterministic inputs.

Architecture

Dataset

Git-managed fixture datasets with items, before/reference snapshots, and version tracking

AgentExperiment

Orchestrates the full loop: load items, invoke agent, judge, aggregate, persist

Comparison Engine

Compare runs across variants with per-judge deltas, regression detection, and summary statistics

Sessions & Sweeps

Group variant results into sessions, group sessions into sweeps for multi-run analysis

ReEvaluator

Post-hoc re-scoring of stored results without re-invoking agents

JudgeExperiment

Run a judge as the system under test against labeled datasets

Modules

Module	Description	Key Dependencies
`experiment-core`	Datasets, workspace provisioning, runner, journals, comparison, results, storage	Agent Judge 0.15.1, Agent Journal 1.8.2, Jackson
`experiment-claude`	Claude SDK invoker, plan generator, semantic judge	Claude Code SDK 1.5.1
`experiment-workflow`	Typed workflow invokers with journal and cost adaptation	Agent Workflow 0.12.1, Agent Client 0.29.3

Documentation

Getting Started

Run your first experiment: dataset, agent, jury, variant comparison

Creating Experiments

Design datasets, configure variants, wire custom judges

Jury System

Build cascaded juries for tiered evaluation

API Reference

Core types, runner, comparison, storage, diagnostics

Quick Start

<dependency>
    <groupId>io.github.markpollack</groupId>
    <artifactId>experiment-core</artifactId>
    <version>0.7.1</version>
</dependency>

ExperimentConfig config = ExperimentConfig.builder()
    .experimentName("my-experiment")
    .datasetDir(Path.of("datasets/my-benchmark"))
    .model("sonnet")
    .promptTemplate("Your task: {{task}}")
    .perItemTimeout(Duration.ofMinutes(10))
    .build();

DatasetManager dm = new FileSystemDatasetManager();
ResultStore store = new FileSystemResultStore(resultsDir);

AgentExperiment experiment = new AgentExperiment(dm, jury, store, config);
ExperimentResult result = experiment.run(invoker);
// result.passRate(), result.totalCostUsd(), result.items()

The public template provides a complete starting point with variant configuration, workflow invokers, default-on journal capture, and a credential-free smoke test.

Release and Compatibility

Version 0.6.0 moves runtime judging to Agent Judge 0.14.0. Stored results use Agent Experiment-owned RecordedJudgment and RecordedVerdict projections. Existing 0.5 / Agent Judge 0.13 result files load automatically and preserve normalized outcomes, reasoning, checks, labels, metadata, and available composite evidence. Re-saving writes the new format; obsolete range bounds, categorical allow-lists, and unnamed legacy composite identity cannot be reconstructed losslessly. Standalone consumers resolve Agent Journal/Capture 1.8.2, Jackson 2.22.2, and Jackson 3.2.2 without an AgentWorks BOM. The parent artifact publishes one aggregate CycloneDX 1.6 JSON SBOM, and the stable release includes signed binaries, sources, and Javadocs with the project BSL text.

Role in the Lab

Agent Experiment is the execution layer that ties the other AgentWorks projects together:

Agent Judge — Jury scores every item
Agent Journal — Traces captured during invocation
Agent Sandbox — Isolated execution environments
Agent Bench — Benchmark datasets consumed by experiments

Public experiments built on this lifecycle include:

Source

GitHub

Source code (BSL 1.1) — three modules, 544 tests

Projects

AgentWorks

Supporting Projects

Migration

Overview

Architecture

Dataset

AgentExperiment

Comparison Engine

Sessions & Sweeps

ReEvaluator

JudgeExperiment

Modules

Documentation

Getting Started

Creating Experiments

Jury System

API Reference

Quick Start

Release and Compatibility

Role in the Lab

Source

GitHub

​Overview

​Architecture

Dataset

AgentExperiment

Comparison Engine

Sessions & Sweeps

ReEvaluator

JudgeExperiment

​Modules

​Documentation

Getting Started

Creating Experiments

Jury System

API Reference

​Quick Start

​Release and Compatibility

​Role in the Lab

​Source

GitHub

Overview

Architecture

Modules

Documentation

Quick Start

Release and Compatibility

Role in the Lab

Source