Skip to main content
This reference describes 0.9.0. On main (0.10.0-SNAPSHOT, not yet released): AgentExperiment takes a jury factory, ReEvaluationContextFactory.create returns Optional<AgentJudgingInput>, JudgeExperiment takes candidate(Function<DatasetItem, Judge>) and replicateCount(int) and no longer has input(...), WorkflowInvoker.buildWorkflow returns a WorkflowInvoker.Plan, and result totals are nullable (null means unknown). None of this is in 0.9.0.

ExperimentConfig

AgentInvoker

Single-method interface — implement this to plug in any agent:
Contract:
  • Blocking: returns when agent completes, times out, or fails
  • Thread-safe: callable from multiple threads
  • NOT responsible for: timeout enforcement, workspace setup, result tracking

Template Invokers

experiment-workflow provides AbstractTemplateAgentInvoker, WorkflowAgentInvoker and WorkflowInvoker<S>; the agent-experiment-template adds TemplateAgentInvoker and TwoPhaseTemplateAgentInvoker on top of them. Choose based on your orchestration needs: AbstractTemplateAgentInvoker provides:
  • Pre/post invoke hooks (preInvoke, postInvoke)
  • Knowledge file injection into workspaces
  • Phase capture collection
WorkflowAgentInvoker wraps a single ClaudeStep in a Workflow with journal integration wired automatically — step events are recorded as WorkflowStepEvent entries via WorkflowJournal. The experiment name is pulled from context.metadata("experimentId"). WorkflowInvoker<S> is the base for multi-step workflows with typed state. Journal and cost tracking are built in. Subclasses implement three methods:

InvocationContext

What the runner passes to your agent:

InvocationResult

What your agent returns:

ExecutionDetail

Marker interface that decouples shared experiment infrastructure (ComparisonEngine, ResultStore, VerdictExtractor) from domain-specific per-item execution details.
ItemResult.executionDetail() returns @Nullable ExecutionDetail. Consumers use instanceof pattern matching to access domain-specific fields:

Dataset Format

dataset.json

item.json

Directory layout

ItemFilter

ResultStore

Both implement:

ExperimentResult


Re-Evaluation

Re-evaluate stored experiment results with a different jury without re-invoking the system under test.

ReEvaluationContextFactory

Functional interface that reconstructs a JudgmentContext from a stored ItemResult:
Returns Optional.empty() when re-evaluation is not possible (failed item, missing execution detail, workspace not preserved).

AgentReEvaluationContextFactory

Default implementation for agent experiment results. Pattern-matches on InvocationResult to reconstruct the context:
Maps TerminalStatus to ExecutionStatus (COMPLETED → SUCCESS, TIMEOUT → TIMEOUT, ERROR → FAILED). Preserves original costUsd and totalTokens.

ReEvaluator

Orchestrates post-hoc re-scoring of stored experiment results:
Re-evaluated results carry metadata: reEvaluated=true, systemReinvoked=false, originalCostUsd, reEvaluationJury, originalTimestamp. Skipped items carry reEvaluationSkipped=true with a reason.

Judge Experiment

Run a judge as the system under test against a labeled dataset to measure agreement rate.

JudgeScorer

Functional interface that scores a candidate judge’s Judgment against the expected label:

JudgeScoringInput

JudgeScorerResult

JudgeScorers

Built-in scoring implementations:

JudgeExecutionDetail

Domain evidence preserved for each item:
Unscored results carry null; see JudgeScorerResult.unscored. Runtime candidates return Agent Eval 0.17 Judgment values, which Agent Experiment converts to its RecordedJudgment persistence form. Results written before 0.9.0 remain readable and are not rewritten.

JudgeExperiment

Builder-based experiment runner where the system under test is a Judge:
Takes List<DatasetItem> directly — judge datasets do not require filesystem loading.

JudgeExperimentResult

JudgeDisagreement


Modules