Agent Bench

What’s New →

The latest public release is 0.6.1 under the io.github.markpollack Maven group and Java package namespace.

Overview

Agent Bench measures AI coding agents on repeatable software-engineering tasks. Inspired by Terminal-Bench, each benchmark separates the task definition, the agent that changes a workspace, and the verifier that grades the result. Agent Bench uses Agent Judge to materialize deterministic checks and cascaded juries rather than reimplementing grading. The filesystem is the contract: any CLI agent that reads INSTRUCTION.md and modifies the workspace can participate without an SDK integration.

Maven Artifacts

Use the core module for the agent-neutral CLI and Java API:

<dependency>
  <groupId>io.github.markpollack</groupId>
  <artifactId>agent-bench-core</artifactId>
  <version>0.6.1</version>
</dependency>

Use the agents module when BenchApp and the real Agent Client-backed LLM judge are required:

<dependency>
  <groupId>io.github.markpollack</groupId>
  <artifactId>agent-bench-agents</artifactId>
  <version>0.6.1</version>
</dependency>

Supported Usage Patterns

End-to-end run

run prepares a workspace, executes setup and the configured agent, runs post-processing, and grades the result.

Split workflow

provide prepares the files, an external agent operates on them, and grade evaluates the resulting workspace.

Java API

Use the benchmark catalog, command classes, JudgeFactory, Agent Judge types, and result model directly from Java.

Execution and Trust Boundary

Benchmark setup scripts, post-processing scripts, and configured agent commands execute as local host processes with the permissions of the user who starts Agent Bench. A workspace directory organizes inputs and results, but it is not a security boundary. Agent Bench does not provide container execution or provision isolation. Run untrusted benchmark definitions or agents only inside your own disposable VM, CI runner, or other externally managed isolation.

CLI Entry Points

agent-bench-core runs BenchMain, the agent-neutral CLI with deterministic Agent Judge checks.
agent-bench-agents runs BenchApp, which retains the core commands and wires the real Agent Client-backed LLM judge for LLM-graded run and resume operations.

Quick Start from Source

git clone https://github.com/markpollack/agent-bench.git
cd agent-bench
git checkout v0.6.1
./mvnw clean verify

./mvnw -q -pl agent-bench-core exec:java \
  -Dexec.args="run --benchmark hello-world --agent agents/my-agent.yaml"

Benchmarks

Benchmark	Status	What it measures
hello-world	Working	File creation and basic agent infrastructure
code-coverage	Working judges; full agent run requires provider access	JUnit test generation, coverage uplift, and LLM-scored practice adherence
spring-boot-upgrade	Candidate	Spring modernization; the public jury is a build quick-check and the baseline-aware grader is owner-operated externally

Documentation

Getting Started

Run end-to-end, split, and Java API workflows

Agent Configuration

Configure a CLI tool and understand its host permissions

Jury System

Agent Judge tiers, built-in judges, and custom judge types

CLI Reference

Core and agents-module commands

Source

GitHub

Released 0.6.1 source --- two published modules

Projects

AgentWorks

Supporting Projects

Migration

Overview

Maven Artifacts

Supported Usage Patterns

End-to-end run

Split workflow

Java API

Execution and Trust Boundary

CLI Entry Points

Quick Start from Source

Benchmarks

Documentation

Getting Started

Agent Configuration

Jury System

CLI Reference

Source

GitHub

​Overview

​Maven Artifacts

​Supported Usage Patterns

End-to-end run

Split workflow

Java API

​Execution and Trust Boundary

​CLI Entry Points

​Quick Start from Source

​Benchmarks

​Documentation

Getting Started

Agent Configuration

Jury System

CLI Reference

​Source

GitHub

Overview

Maven Artifacts

Supported Usage Patterns

Execution and Trust Boundary

CLI Entry Points

Quick Start from Source

Benchmarks

Documentation

Source