> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Experiment API Reference

> Configuration, dataset format, invoker contract, and result model

## ExperimentConfig

```java theme={null}
ExperimentConfig.builder()
    .experimentName("name")                    // Required: experiment identifier
    .datasetDir(Path.of("dataset"))            // Required: dataset directory
    .model("sonnet")                           // Required: LLM model
    .promptTemplate("{{task}}")                // Required: prompt with placeholders
    .perItemTimeout(Duration.ofMinutes(2))     // Required: per-item timeout
    .itemFilter(ItemFilter.bucket("A"))        // Optional: filter items
    .knowledgeBaseDir(Path.of("kb"))           // Optional: KB root
    .outputDir(Path.of("results"))             // Optional: persist workspaces
    .experimentTimeout(Duration.ofHours(1))    // Optional: overall timeout
    .metadata(Map.of("key", "value"))          // Optional: arbitrary metadata
    .baselineId("abc123")                      // Optional: comparison baseline
    .build();
```

## AgentInvoker

Single-method interface — implement this to plug in any agent:

```java theme={null}
public interface AgentInvoker {
    InvocationResult invoke(InvocationContext context)
        throws AgentInvocationException;
}
```

**Contract**:

* Blocking: returns when agent completes, times out, or fails
* Thread-safe: callable from multiple threads
* NOT responsible for: timeout enforcement, workspace setup, result tracking

### Template Invokers

The [agent-experiment-template](https://github.com/markpollack/agent-experiment-template) provides a hierarchy of ready-made invokers. Choose based on your orchestration needs:

| Class                          | Extends                        | Use When                                                                |
| ------------------------------ | ------------------------------ | ----------------------------------------------------------------------- |
| `TemplateAgentInvoker`         | `AbstractTemplateAgentInvoker` | Single-phase agent via `AgentClient` (rename to `{Domain}AgentInvoker`) |
| `TwoPhaseTemplateAgentInvoker` | `AbstractTemplateAgentInvoker` | Two-phase explore + act via `ClaudeSyncClient`                          |
| `WorkflowAgentInvoker`         | `AbstractTemplateAgentInvoker` | Single-step workflow — simplest workflow entry point                    |
| `WorkflowInvoker<S>`           | `AgentInvoker` (direct)        | Multi-step typed workflow with cost tracking                            |

**`AbstractTemplateAgentInvoker`** provides:

* Pre/post invoke hooks (`preInvoke`, `postInvoke`)
* Knowledge file injection into workspaces
* Phase capture collection

**`WorkflowAgentInvoker`** wraps a single `ClaudeStep` in a `Workflow` with journal integration wired automatically — step events are recorded as `WorkflowStepEvent` entries via `WorkflowJournal`. The experiment name is pulled from `context.metadata("experimentId")`.

**`WorkflowInvoker<S>`** is the base for multi-step workflows with typed state. Journal and cost tracking are built in. Subclasses implement three methods:

```java theme={null}
public class MyWorkflow extends WorkflowInvoker<MyState> {

    @Override
    protected String workflowName() { return "my-experiment"; }

    @Override
    protected Workflow<Object, MyState> buildWorkflow(
            InvocationContext ctx, WorkflowExecutor executor) {
        return Workflow.<Object, MyState>define(workflowName())
                .withExecutor(executor)   // journal-backed
                .step(analyzeStep)
                .step(fixStep)
                .build();
    }

    @Override
    protected MyState buildInitialState(InvocationContext ctx) {
        return new MyState(ctx.workspacePath());
    }
}
```

## InvocationContext

What the runner passes to your agent:

| Field           | Type       | Description                               |
| --------------- | ---------- | ----------------------------------------- |
| `workspacePath` | `Path`     | Directory where agent operates            |
| `prompt`        | `String`   | Fully constructed prompt                  |
| `systemPrompt`  | `String`   | Optional additional system instructions   |
| `model`         | `String`   | Model identifier                          |
| `timeout`       | `Duration` | Timeout hint                              |
| `metadata`      | `Map`      | Pass-through (experimentId, itemId, etc.) |
| `runDir`        | `Path`     | Optional directory for trace artifacts    |

## InvocationResult

What your agent returns:

```java theme={null}
// Success
InvocationResult.completed(phases, inputTokens, outputTokens,
    thinkingTokens, totalCostUsd, durationMs, sessionId, metadata);

// Timeout
InvocationResult.timeout(durationMs, metadata, errorMessage);

// Error
InvocationResult.error(errorMessage, metadata);
```

| Field          | Type             | Description                     |
| -------------- | ---------------- | ------------------------------- |
| `success`      | `boolean`        | Agent completed without error   |
| `status`       | `TerminalStatus` | `COMPLETED`, `ERROR`, `TIMEOUT` |
| `inputTokens`  | `int`            | Total input tokens consumed     |
| `outputTokens` | `int`            | Total output tokens produced    |
| `totalCostUsd` | `double`         | Estimated cost                  |
| `durationMs`   | `long`           | Wall-clock execution time       |

## ExecutionDetail

Marker interface that decouples shared experiment infrastructure (ComparisonEngine, ResultStore, VerdictExtractor) from domain-specific per-item execution details.

```java theme={null}
public interface ExecutionDetail {
    // Marker — shared infrastructure stores but never interprets
}
```

| Implementation         | Used By           | Contains                                          |
| ---------------------- | ----------------- | ------------------------------------------------- |
| `InvocationResult`     | `AgentExperiment` | Agent invocation output, tokens, cost, phases     |
| `JudgeExecutionDetail` | `JudgeExperiment` | Candidate judgment, expected label, scorer result |

`ItemResult.executionDetail()` returns `@Nullable ExecutionDetail`. Consumers use `instanceof` pattern matching to access domain-specific fields:

```java theme={null}
if (item.executionDetail() instanceof InvocationResult inv) {
    System.out.println("Cost: $" + inv.totalCostUsd());
}
```

## Dataset Format

### dataset.json

```json theme={null}
{
  "schemaVersion": 1,
  "name": "dataset-name",
  "version": "1.0.0",
  "description": "What this dataset tests",
  "items": [
    {
      "id": "ITEM-001",
      "slug": "short-description",
      "path": "items/ITEM-001",
      "bucket": "A",
      "taskType": "task-type",
      "status": "active"
    }
  ]
}
```

### item.json

```json theme={null}
{
  "schemaVersion": 1,
  "id": "ITEM-001",
  "slug": "short-description",
  "developerTask": "Natural language task description",
  "taskType": "task-type",
  "bucket": "A",
  "noChange": false,
  "knowledgeRefs": ["path/to/kb-entry.md"],
  "tags": ["tag1", "tag2"],
  "status": "active"
}
```

### Directory layout

```
dataset/
├── dataset.json
└── items/
    └── ITEM-001/
        ├── item.json
        ├── before/          # Starting state
        │   └── src/...
        └── reference/       # Correct result
            └── src/...
```

## ItemFilter

```java theme={null}
ItemFilter.all()                     // No filtering
ItemFilter.bucket("A")               // Single bucket
ItemFilter.tags("rename", "simple")  // By tags
ItemFilter.id("ITEM-001")           // Single item
```

## ResultStore

| Implementation                | Use case                      |
| ----------------------------- | ----------------------------- |
| `FileSystemResultStore(path)` | Production — persists to disk |
| `InMemoryResultStore()`       | Testing — HashMap-backed      |

Both implement:

```java theme={null}
void save(ExperimentResult result);
Optional<ExperimentResult> load(String id);
List<ExperimentResult> listByName(String experimentName);
Optional<ExperimentResult> mostRecent(String experimentName);
```

## ExperimentResult

| Method             | Type               | Description                  |
| ------------------ | ------------------ | ---------------------------- |
| `experimentId()`   | `String`           | Unique run ID                |
| `experimentName()` | `String`           | Experiment name from config  |
| `items()`          | `List<ItemResult>` | Per-item results             |
| `passCount()`      | `int`              | Items that passed all judges |
| `failCount()`      | `int`              | Items that failed            |
| `passRate()`       | `double`           | Pass count / total (0.0–1.0) |

***

## Re-Evaluation

Re-evaluate stored experiment results with a different jury without re-invoking the system under test.

### ReEvaluationContextFactory

Functional interface that reconstructs a `JudgmentContext` from a stored `ItemResult`:

```java theme={null}
@FunctionalInterface
public interface ReEvaluationContextFactory {
    Optional<JudgmentContext> create(ItemResult item);
}
```

Returns `Optional.empty()` when re-evaluation is not possible (failed item, missing execution detail, workspace not preserved).

### AgentReEvaluationContextFactory

Default implementation for agent experiment results. Pattern-matches on `InvocationResult` to reconstruct the context:

```java theme={null}
ReEvaluationContextFactory factory =
    AgentReEvaluationContextFactory.defaults();
```

Maps `TerminalStatus` to `ExecutionStatus` (`COMPLETED` → `SUCCESS`, `TIMEOUT` → `TIMEOUT`, `ERROR` → `FAILED`). Preserves original `costUsd` and `totalTokens`.

### ReEvaluator

Orchestrates post-hoc re-scoring of stored experiment results:

```java theme={null}
ReEvaluator reEvaluator = ReEvaluator.builder()
    .contextFactory(AgentReEvaluationContextFactory.defaults())
    .resultStore(store)
    .build();

ExperimentResult reScored = reEvaluator.reEvaluate(originalResult, newJury);
```

| Method                                  | Description                                                |
| --------------------------------------- | ---------------------------------------------------------- |
| `reEvaluate(ExperimentResult, Jury)`    | Re-score a loaded result with a new jury                   |
| `reEvaluate(String experimentId, Jury)` | Load by ID, then re-score                                  |
| `agentDefaults(ResultStore)`            | Convenience factory with `AgentReEvaluationContextFactory` |

Re-evaluated results carry metadata: `reEvaluated=true`, `systemReinvoked=false`, `originalCostUsd`, `reEvaluationJury`, `originalTimestamp`. Skipped items carry `reEvaluationSkipped=true` with a reason.

***

## Judge Experiment

Run a judge as the system under test against a labeled dataset to measure agreement rate.

### JudgeScorer

Functional interface that scores a candidate judge's `Judgment` against the expected label:

```java theme={null}
@FunctionalInterface
public interface JudgeScorer {
    JudgeScorerResult score(JudgeScoringInput input);
}
```

### JudgeScoringInput

```java theme={null}
public record JudgeScoringInput(
    DatasetItem item,        // dataset item (for item-level context)
    Judgment actual,         // candidate judge's judgment
    String expectedLabel     // expected label from dataset
)
```

### JudgeScorerResult

```java theme={null}
public record JudgeScorerResult(
    boolean match,           // judge agreed with expected label
    double score,            // normalized agreement score [0, 1]
    String reasoning         // explanation of match/mismatch
)
```

### JudgeScorers

Built-in scoring implementations:

| Factory Method               | Scoring Rule                                                          |
| ---------------------------- | --------------------------------------------------------------------- |
| `exactVerdictMatch()`        | PASS/FAIL must exactly match expected `"PASS"`/`"FAIL"` label         |
| `exactCategoryMatch()`       | `CategoricalScore` value must match expected label (case-insensitive) |
| `numericalTolerance(double)` | `NumericalScore` within tolerance of expected numeric value           |

### JudgeExecutionDetail

Domain evidence preserved for each item:

```java theme={null}
public record JudgeExecutionDetail(
    Judgment candidateJudgment,
    String expectedLabel,
    JudgeScorerResult scorerResult
) implements ExecutionDetail
```

### JudgeExperiment

Builder-based experiment runner where the system under test is a `Judge`:

```java theme={null}
JudgeExperimentResult result = JudgeExperiment.builder()
    .name("correctness-judge-calibration")
    .candidate(myCorrectnessJudge)
    .items(labeledItems)
    .input(item -> buildContextFromItem(item))
    .expected(item -> item.metadata().get("expectedVerdict"))
    .scorer(JudgeScorers.exactVerdictMatch())
    .resultStore(store)
    .build()
    .run();
```

| Builder Method                                  | Required | Description                       |
| ----------------------------------------------- | -------- | --------------------------------- |
| `name(String)`                                  | Yes      | Experiment name                   |
| `candidate(Judge)`                              | Yes      | Judge to evaluate                 |
| `items(List<DatasetItem>)`                      | Yes      | Labeled dataset items             |
| `input(Function<DatasetItem, JudgmentContext>)` | Yes      | Builds context from item          |
| `expected(Function<DatasetItem, String>)`       | Yes      | Extracts expected label from item |
| `scorer(JudgeScorer)`                           | Yes      | Scoring strategy                  |
| `resultStore(ResultStore)`                      | Yes      | Persistence                       |
| `datasetVersion(String)`                        | No       | Defaults to `"1.0.0"`             |

Takes `List<DatasetItem>` directly — judge datasets do not require filesystem loading.

### JudgeExperimentResult

```java theme={null}
public record JudgeExperimentResult(
    ExperimentResult experimentResult,
    double agreementRate,
    List<JudgeDisagreement> disagreements
)
```

| Method                   | Description                                                               |
| ------------------------ | ------------------------------------------------------------------------- |
| `agreementRate()`        | Fraction of items where judge agreed with expected label                  |
| `disagreements()`        | Items where judge disagreed                                               |
| `from(ExperimentResult)` | Create from an `ExperimentResult` containing `JudgeExecutionDetail` items |
| `asExperimentResult()`   | Unwrap for `ComparisonEngine` and `ResultStore` compatibility             |

### JudgeDisagreement

```java theme={null}
public record JudgeDisagreement(
    String itemId,
    JudgeExecutionDetail detail
)
```

***

## Modules

```xml theme={null}
<!-- Core: runner, dataset, jury, stores, diagnostics -->
<dependency>
    <groupId>io.github.markpollack</groupId>
    <artifactId>experiment-core</artifactId>
</dependency>

<!-- Claude SDK integration (ClaudeSdkInvoker, SemanticDiffJudge) -->
<dependency>
    <groupId>io.github.markpollack</groupId>
    <artifactId>experiment-claude</artifactId>
</dependency>
```