> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Experiment API Reference

> Configuration, dataset format, invoker contract, and result model

## ExperimentConfig

```java theme={null}
ExperimentConfig.builder()
    .experimentName("name")                    // Required: experiment identifier
    .datasetDir(Path.of("dataset"))            // Required: dataset directory
    .model("sonnet")                           // Required: LLM model
    .promptTemplate("{{task}}")                // Required: prompt with placeholders
    .perItemTimeout(Duration.ofMinutes(2))     // Required: per-item timeout
    .itemFilter(ItemFilter.bucket("A"))        // Optional: filter items
    .knowledgeBaseDir(Path.of("kb"))           // Optional: KB root
    .outputDir(Path.of("results"))             // Optional: persist workspaces
    .experimentTimeout(Duration.ofHours(1))    // Optional: overall timeout
    .metadata(Map.of("key", "value"))          // Optional: arbitrary metadata
    .baselineId("abc123")                      // Optional: comparison baseline
    .build();
```

## AgentInvoker

Single-method interface — implement this to plug in any agent:

```java theme={null}
public interface AgentInvoker {
    InvocationResult invoke(InvocationContext context)
        throws AgentInvocationException;
}
```

**Contract**:

* Blocking: returns when agent completes, times out, or fails
* Thread-safe: callable from multiple threads
* NOT responsible for: timeout enforcement, workspace setup, result tracking

### Template Invokers

The [agent-experiment-template](https://github.com/markpollack/agent-experiment-template) provides a hierarchy of ready-made invokers. Choose based on your orchestration needs:

| Class | Extends | Use When |
| - | - | - |
| `TemplateAgentInvoker` | `AbstractTemplateAgentInvoker` | Single-phase agent via `AgentClient` (rename to `{Domain}AgentInvoker`) |
| `TwoPhaseTemplateAgentInvoker` | `AbstractTemplateAgentInvoker` | Two-phase explore + act via `ClaudeSyncClient` |
| `WorkflowAgentInvoker` | `AbstractTemplateAgentInvoker` | Single-step workflow — simplest workflow entry point |
| `WorkflowInvoker<S>` | `AgentInvoker` (direct) | Multi-step typed workflow with cost tracking |

**`AbstractTemplateAgentInvoker`** provides:

* Pre/post invoke hooks (`preInvoke`, `postInvoke`)
* Knowledge file injection into workspaces
* Phase capture collection

**`WorkflowAgentInvoker`** wraps a single `ClaudeStep` in a `Workflow` with journal integration wired automatically — step events are recorded as `WorkflowStepEvent` entries via `WorkflowJournal`. The experiment name is pulled from `context.metadata("experimentId")`.

**`WorkflowInvoker<S>`** is the base for multi-step workflows with typed state. Journal and cost tracking are built in. Subclasses implement three methods:

```java theme={null}
public class MyWorkflow extends WorkflowInvoker<MyState> {

    @Override
    protected String workflowName() { return "my-experiment"; }

    @Override
    protected Workflow<Object, MyState> buildWorkflow(
            InvocationContext ctx, WorkflowExecutor executor) {
        return Workflow.<Object, MyState>define(workflowName())
                .withExecutor(executor)   // journal-backed
                .step(analyzeStep)
                .step(fixStep)
                .build();
    }

    @Override
    protected MyState buildInitialState(InvocationContext ctx) {
        return new MyState(ctx.workspacePath());
    }
}
```

## InvocationContext

What the runner passes to your agent:

| Field | Type | Description |
| - | - | - |
| `workspacePath` | `Path` | Directory where agent operates |
| `prompt` | `String` | Fully constructed prompt |
| `systemPrompt` | `String` | Optional additional system instructions |
| `model` | `String` | Model identifier |
| `timeout` | `Duration` | Timeout hint |
| `metadata` | `Map` | Pass-through (experimentId, itemId, etc.) |
| `runDir` | `Path` | Optional directory for trace artifacts |

## InvocationResult

What your agent returns:

```java theme={null}
// Success
InvocationResult.completed(phases, inputTokens, outputTokens,
    thinkingTokens, totalCostUsd, durationMs, sessionId, metadata);

// Timeout
InvocationResult.timeout(durationMs, metadata, errorMessage);

// Error
InvocationResult.error(errorMessage, metadata);
```

| Field | Type | Description |
| - | - | - |
| `success` | `boolean` | Agent completed without error |
| `status` | `TerminalStatus` | `COMPLETED`, `ERROR`, `TIMEOUT` |
| `inputTokens` | `int` | Total input tokens consumed |
| `outputTokens` | `int` | Total output tokens produced |
| `totalCostUsd` | `double` | Estimated cost |
| `durationMs` | `long` | Wall-clock execution time |

## ExecutionDetail

Marker interface that decouples shared experiment infrastructure (ComparisonEngine, ResultStore, VerdictExtractor) from domain-specific per-item execution details.

```java theme={null}
public interface ExecutionDetail {
    // Marker — shared infrastructure stores but never interprets
}
```

| Implementation | Used By | Contains |
| - | - | - |
| `InvocationResult` | `AgentExperiment` | Agent invocation output, tokens, cost, phases |
| `JudgeExecutionDetail` | `JudgeExperiment` | Candidate judgment, expected label, scorer result |

`ItemResult.executionDetail()` returns `@Nullable ExecutionDetail`. Consumers use `instanceof` pattern matching to access domain-specific fields:

```java theme={null}
if (item.executionDetail() instanceof InvocationResult inv) {
    System.out.println("Cost: $" + inv.totalCostUsd());
}
```

## Dataset Format

### dataset.json

```json theme={null}
{
  "schemaVersion": 1,
  "name": "dataset-name",
  "version": "1.0.0",
  "description": "What this dataset tests",
  "items": [
    {
      "id": "ITEM-001",
      "slug": "short-description",
      "path": "items/ITEM-001",
      "bucket": "A",
      "taskType": "task-type",
      "status": "active"
    }
  ]
}
```

### item.json

```json theme={null}
{
  "schemaVersion": 1,
  "id": "ITEM-001",
  "slug": "short-description",
  "developerTask": "Natural language task description",
  "taskType": "task-type",
  "bucket": "A",
  "noChange": false,
  "knowledgeRefs": ["path/to/kb-entry.md"],
  "tags": ["tag1", "tag2"],
  "status": "active"
}
```

### Directory layout

```
dataset/
├── dataset.json
└── items/
    └── ITEM-001/
        ├── item.json
        ├── before/          # Starting state
        │   └── src/...
        └── reference/       # Correct result
            └── src/...
```

## ItemFilter

```java theme={null}
ItemFilter.all()                     // No filtering
ItemFilter.bucket("A")               // Single bucket
ItemFilter.tags("rename", "simple")  // By tags
ItemFilter.id("ITEM-001")           // Single item
```

## ResultStore

| Implementation | Use case |
| - | - |
| `FileSystemResultStore(path)` | Production — persists to disk |
| `InMemoryResultStore()` | Testing — HashMap-backed |

Both implement:

```java theme={null}
void save(ExperimentResult result);
Optional<ExperimentResult> load(String id);
List<ExperimentResult> listByName(String experimentName);
Optional<ExperimentResult> mostRecent(String experimentName);
```

## ExperimentResult

| Method | Type | Description |
| - | - | - |
| `experimentId()` | `String` | Unique run ID |
| `experimentName()` | `String` | Experiment name from config |
| `items()` | `List<ItemResult>` | Per-item results |
| `passCount()` | `int` | Items that passed all judges |
| `failCount()` | `int` | Items that failed |
| `passRate()` | `double` | Pass count / total (0.0–1.0) |

***

## Re-Evaluation

Re-evaluate stored experiment results with a different jury without re-invoking the system under test.

### ReEvaluationContextFactory

Functional interface that reconstructs a `JudgmentContext` from a stored `ItemResult`:

```java theme={null}
@FunctionalInterface
public interface ReEvaluationContextFactory {
    Optional<JudgmentContext> create(ItemResult item);
}
```

Returns `Optional.empty()` when re-evaluation is not possible (failed item, missing execution detail, workspace not preserved).

### AgentReEvaluationContextFactory

Default implementation for agent experiment results. Pattern-matches on `InvocationResult` to reconstruct the context:

```java theme={null}
ReEvaluationContextFactory factory =
    AgentReEvaluationContextFactory.defaults();
```

Maps `TerminalStatus` to `ExecutionStatus` (`COMPLETED` → `SUCCESS`, `TIMEOUT` → `TIMEOUT`, `ERROR` → `FAILED`). Preserves original `costUsd` and `totalTokens`.

### ReEvaluator

Orchestrates post-hoc re-scoring of stored experiment results:

```java theme={null}
ReEvaluator reEvaluator = ReEvaluator.builder()
    .contextFactory(AgentReEvaluationContextFactory.defaults())
    .resultStore(store)
    .build();

ExperimentResult reScored = reEvaluator.reEvaluate(originalResult, newJury);
```

| Method | Description |
| - | - |
| `reEvaluate(ExperimentResult, Jury)` | Re-score a loaded result with a new jury |
| `reEvaluate(String experimentId, Jury)` | Load by ID, then re-score |
| `agentDefaults(ResultStore)` | Convenience factory with `AgentReEvaluationContextFactory` |

Re-evaluated results carry metadata: `reEvaluated=true`, `systemReinvoked=false`, `originalCostUsd`, `reEvaluationJury`, `originalTimestamp`. Skipped items carry `reEvaluationSkipped=true` with a reason.

***

## Judge Experiment

Run a judge as the system under test against a labeled dataset to measure agreement rate.

### JudgeScorer

Functional interface that scores a candidate judge's `Judgment` against the expected label:

```java theme={null}
@FunctionalInterface
public interface JudgeScorer {
    JudgeScorerResult score(JudgeScoringInput input);
}
```

### JudgeScoringInput

```java theme={null}
public record JudgeScoringInput(
    DatasetItem item,        // dataset item (for item-level context)
    Judgment actual,         // candidate judge's judgment
    String expectedLabel     // expected label from dataset
)
```

### JudgeScorerResult

```java theme={null}
public record JudgeScorerResult(
    boolean match,           // judge agreed with expected label
    double score,            // normalized agreement score [0, 1]
    String reasoning         // explanation of match/mismatch
)
```

### JudgeScorers

Built-in scoring implementations:

| Factory Method | Scoring Rule |
| - | - |
| `exactVerdictMatch()` | PASS/FAIL must exactly match expected `"PASS"`/`"FAIL"` label |
| `exactCategoryMatch()` | `CategoricalScore` value must match expected label (case-insensitive) |
| `numericalTolerance(double)` | `NumericalScore` within tolerance of expected numeric value |

### JudgeExecutionDetail

Domain evidence preserved for each item:

```java theme={null}
public record JudgeExecutionDetail(
    RecordedJudgment candidateJudgment,
    String expectedLabel,
    JudgeScorerResult scorerResult
) implements ExecutionDetail
```

Runtime candidates still return Agent Judge 0.14 `Judgment` values. Agent Experiment converts them
immediately to its normalized `RecordedJudgment` persistence boundary. Existing 0.5 / Agent Judge 0.13
result files are upgraded on read; saving them writes the normalized 0.6 format.

### JudgeExperiment

Builder-based experiment runner where the system under test is a `Judge`:

```java theme={null}
JudgeExperimentResult result = JudgeExperiment.builder()
    .name("correctness-judge-calibration")
    .candidate(myCorrectnessJudge)
    .items(labeledItems)
    .input(item -> buildContextFromItem(item))
    .expected(item -> item.metadata().get("expectedVerdict"))
    .scorer(JudgeScorers.exactVerdictMatch())
    .resultStore(store)
    .build()
    .run();
```

| Builder Method | Required | Description |
| - | - | - |
| `name(String)` | Yes | Experiment name |
| `candidate(Judge)` | Yes | Judge to evaluate |
| `items(List<DatasetItem>)` | Yes | Labeled dataset items |
| `input(Function<DatasetItem, JudgmentContext>)` | Yes | Builds context from item |
| `expected(Function<DatasetItem, String>)` | Yes | Extracts expected label from item |
| `scorer(JudgeScorer)` | Yes | Scoring strategy |
| `resultStore(ResultStore)` | Yes | Persistence |
| `datasetVersion(String)` | No | Defaults to `"1.0.0"` |

Takes `List<DatasetItem>` directly — judge datasets do not require filesystem loading.

### JudgeExperimentResult

```java theme={null}
public record JudgeExperimentResult(
    ExperimentResult experimentResult,
    double agreementRate,
    List<JudgeDisagreement> disagreements
)
```

| Method | Description |
| - | - |
| `agreementRate()` | Fraction of items where judge agreed with expected label |
| `disagreements()` | Items where judge disagreed |
| `from(ExperimentResult)` | Create from an `ExperimentResult` containing `JudgeExecutionDetail` items |
| `asExperimentResult()` | Unwrap for `ComparisonEngine` and `ResultStore` compatibility |

### JudgeDisagreement

```java theme={null}
public record JudgeDisagreement(
    String itemId,
    JudgeExecutionDetail detail
)
```

***

## Modules

```xml theme={null}
<!-- Core: runner, dataset, jury, stores, diagnostics -->
<dependency>
    <groupId>io.github.markpollack</groupId>
    <artifactId>experiment-core</artifactId>
    <version>0.6.0</version>
</dependency>

<!-- Claude SDK integration (ClaudeSdkInvoker, SemanticDiffJudge) -->
<dependency>
    <groupId>io.github.markpollack</groupId>
    <artifactId>experiment-claude</artifactId>
    <version>0.6.0</version>
</dependency>

<!-- Agent Workflow integration (typed workflow invokers and journal adaptation) -->
<dependency>
    <groupId>io.github.markpollack</groupId>
    <artifactId>experiment-workflow</artifactId>
    <version>0.6.0</version>
</dependency>
```