> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Bench

> Benchmarking suite for Java-centric AI agents on real-world software engineering tasks

**[What's New →](/docs/agent-bench/whats-new)**

<Info>
  This project has moved from the `spring-ai-community` GitHub organization to
  `markpollack`. New releases are published under the Maven groupId
  `io.github.markpollack`, and Java packages now use the `io.github.markpollack`
  namespace. If you previously used `org.springaicommunity`, update your
  dependency coordinates and imports to the current values shown below.
</Info>

## Overview

Agent Bench measures AI coding agents on real enterprise development tasks.
Bring your agent, point it at a benchmark, and get graded results.

The filesystem is the contract.
Any CLI tool that reads `INSTRUCTION.md` and modifies the workspace can participate --- no SDK integration required.

## How It Works

<Steps>
  <Step title="Configure your agent">
    Write a 2-line YAML file with your agent's command and timeout.
  </Step>

  <Step title="Run a benchmark">
    `bench run` sets up the workspace, invokes your agent, and grades the result.
  </Step>

  <Step title="Review results">
    JSON results with accuracy, pass\@k, per-tier jury scores, cost, and duration.
  </Step>
</Steps>

## Architecture

<CardGroup cols={2}>
  <Card title="Benchmark Catalog" icon="list">
    YAML-defined benchmarks with tasks, setup scripts, and difficulty levels
  </Card>

  <Card title="Agent Invocation" icon="robot">
    Your agent runs as a subprocess in an isolated workspace --- any executable works
  </Card>

  <Card title="Jury System" icon="scale-balanced">
    Cascaded tiers of judges from [Agent Judge](/projects/agent-judge) --- deterministic and LLM-based
  </Card>

  <Card title="Result Model" icon="chart-bar">
    TrialResult per task, BenchmarkResult aggregate, pass\@k, resume support
  </Card>
</CardGroup>

## Quick Start

```yaml theme={null}
# agents/my-agent.yaml
command: claude --print --dangerously-skip-permissions "Read INSTRUCTION.md and follow the instructions."
timeout: PT10M
```

```bash theme={null}
git clone https://github.com/markpollack/agent-bench.git
cd agent-bench
./mvnw clean install -DskipTests

# Run hello-world benchmark with your agent
./mvnw exec:java -pl agent-bench-core \
  -Dexec.args="run --benchmark hello-world --agent agents/my-agent.yaml"
```

## Benchmarks

| Benchmark         | Difficulty | What it measures                                          |
| ----------------- | ---------- | --------------------------------------------------------- |
| **hello-world**   | Easy       | File creation, basic agent infrastructure                 |
| **code-coverage** | Medium     | JUnit test generation, coverage uplift on Spring projects |

## Documentation

<CardGroup cols={2}>
  <Card title="Getting Started" icon="rocket" href="/docs/agent-bench/getting-started">
    Test your agent in 5 minutes
  </Card>

  <Card title="Agent Configuration" icon="gear" href="/docs/agent-bench/agent-config">
    How to configure any CLI tool as a benchmark agent
  </Card>

  <Card title="Jury System" icon="scale-balanced" href="/docs/agent-bench/jury-system">
    Cascaded tiers, built-in judges, custom judge types
  </Card>

  <Card title="CLI Reference" icon="terminal" href="/docs/agent-bench/cli-reference">
    All commands: run, resume, compare, list, grade
  </Card>
</CardGroup>

## Role in the Lab

* Uses [Agent Judge](/projects/agent-judge) for YAML-configured jury grading
* Validated against [Code Coverage v2](/experiments/code-coverage-v2) experiment results
* Agent-agnostic: Claude Code, Gemini, Amazon Q, or any CLI tool

## Source

<Card title="GitHub" icon="github" href="https://github.com/markpollack/agent-bench">
  Source code --- two modules, 211 tests
</Card>