Getting Started with Agent Bench

Prerequisites

Java 21
An AI coding agent with a CLI for agent-driven runs

Clone the repository, select the 0.6.0 release tag, and run its verified build:

git clone https://github.com/markpollack/agent-bench.git
cd agent-bench
git checkout v0.6.0
./mvnw clean verify

Benchmark setup, post-processing, and configured agent commands run as local host processes with your permissions. A workspace directory organizes files; it does not isolate them. For untrusted benchmark definitions or agents, supply your own disposable VM, CI runner, or other externally managed isolation.

Pattern 1: End-to-End `run`

Create an agent configuration:

# agents/my-agent.yaml
command: claude --print --dangerously-skip-permissions "Read INSTRUCTION.md and follow the instructions precisely."
timeout: PT5M

Run the deterministic hello-world benchmark through the core CLI:

./mvnw -q -pl agent-bench-core exec:java \
  -Dexec.args="run --benchmark hello-world --agent agents/my-agent.yaml"

The lifecycle is provide → setup → agent → post-processing → grade. Results are written under runs/<uuid>/, including aggregate metadata and the preserved task workspace. For a jury containing test-quality-llm, use the agents-module entry point. Its BenchApp wires the real Agent Client-backed judge:

./mvnw -q -pl agent-bench-agents exec:java \
  -Dexec.args="run --benchmark code-coverage --agent agents/my-agent.yaml"

That path may invoke a model provider and requires the corresponding CLI credentials and runtime.

Pattern 2: `provide`, External Agent, `grade`

The split workflow lets another system own agent execution:

./mvnw -q -pl agent-bench-core exec:java \
  -Dexec.args="provide --benchmark hello-world --task hello-world --workspace /tmp/hello-bench"

cd /tmp/hello-bench
your-agent "Read INSTRUCTION.md and complete the task"

cd /path/to/agent-bench
./mvnw -q -pl agent-bench-core exec:java \
  -Dexec.args="grade --benchmark hello-world --task hello-world --workspace /tmp/hello-bench"

provide writes INSTRUCTION.md and .bench-context.yaml; the external agent changes the workspace; grade uses the benchmark’s Agent Judge configuration. The split CLI path is intended for deterministic juries. LLM-backed judging is wired for agents-module run and resume.

Pattern 3: Java API

Applications can discover definitions and materialize the same Agent Judge jury directly:

BenchmarkCatalog catalog = new BenchmarkCatalog(Path.of("benchmarks"));
Benchmark benchmark = catalog.discover().stream()
    .filter(candidate -> candidate.name().equals("hello-world"))
    .findFirst()
    .orElseThrow();

JudgeFactory factory = new JudgeFactory();
Judge judge = factory.createFromConfig(benchmark.juryConfig());
Judgment judgment = judge.judge(
    JudgmentContext.builder().workspace(Path.of("/tmp/hello-bench")).build());

RunCommand, ProvideCommand, GradeCommand, CompareCommand, and the benchmark/result records are also available for programmatic orchestration.

Result Layout

runs/<uuid>/
  result.json
  run-metadata.json
  bench.lock
  tasks/
    <task-id>/
      result.json
      workspace/

Agent Configuration

Commands, timeouts, journals, and the trust boundary

CLI Reference

Commands and entry-point differences

Jury System

Agent Judge tiers, policies, and custom judges

Projects

AgentWorks

Supporting Projects

Migration

Getting Started with Agent Bench

Prerequisites

Pattern 1: End-to-End `run`

Pattern 2: `provide`, External Agent, `grade`

Pattern 3: Java API

Result Layout

Agent Configuration

CLI Reference

Jury System

​Prerequisites

​Pattern 1: End-to-End run

​Pattern 2: provide, External Agent, grade

​Pattern 3: Java API

​Result Layout

Agent Configuration

CLI Reference

Jury System

Prerequisites

Pattern 1: End-to-End `run`

Pattern 2: `provide`, External Agent, `grade`

Pattern 3: Java API

Result Layout