Entry Points
The 0.6.0 release publishes both CLI entry points. Run them from av0.6.0 source checkout:
BenchMain registers Agent Judge’s deterministic file, build, and coverage judges. During run
and resume, its test-quality-llm registration abstains because no model-backed judge is present.
BenchApp replaces that registration with TestQualityJudge, backed by Agent Client and the Claude
agent implementation. Other commands delegate to the core CLI.
All benchmark setup, post-processing, and configured agent commands execute as local host processes
with the invoking user’s permissions. Workspaces do not provide isolation.
Discovery Commands
list
tasks
--benchmark is required.
End-to-End Commands
run
Prepares each selected task, runs setup scripts and the configured agent, runs post-processing, and
grades the result.
resume
Resumes runs/<uuid>, preserving completed trial results and running missing trials.
--run-id is required.
compare
--runs accepts one or more run-directory paths and prints accuracy, pass@k, cost, duration, and
trial-count comparisons.
Split Workflow
provide
Creates the workspace, copies an optional template, and writes INSTRUCTION.md and
.bench-context.yaml.
grade
Evaluates an existing workspace with the benchmark’s deterministic Agent Judge configuration.
--benchmark and --workspace; --task is optional only for a single-task
benchmark. The built-in split CLI is for deterministic juries. Use agents-module run or resume
when the jury includes the real test-quality-llm judge.
Output Structure
FailureMode values, including agent timeout/error, setup or
grade errors, build/test failures, context exhaustion, and unknown failures.