> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Creating Experiments

> Design datasets, define variant ladders, filter items, and analyze results

## Design Philosophy

Every experiment tests a hypothesis about what makes agents better. The experiment driver makes the independent variables explicit:

| Variable                | How you control it                                                    |
| ----------------------- | --------------------------------------------------------------------- |
| **Knowledge**           | `knowledgeRefs` in dataset items, `knowledgeBaseDir` in config        |
| **Prompt structure**    | `promptTemplate` with `{{task}}` and `{{knowledgeRefs}}` placeholders |
| **Model**               | `model` field in config                                               |
| **Execution strategy**  | Your `AgentInvoker` implementation                                    |
| **Evaluation criteria** | Your `Jury` wiring                                                    |

## Variant Ladders

The most informative experiments use a **progressive variant ladder** — each variant adds one thing to the previous:

| Variant                  | Change from previous               | Tests                          |
| ------------------------ | ---------------------------------- | ------------------------------ |
| 1. Simple prompt         | — (baseline)                       | Model's raw capability         |
| 2. + System prompt       | Add domain instructions            | Does framing help?             |
| 3. + Knowledge base      | Add `knowledgeRefs`                | Does knowledge help?           |
| 4. + Skills (SkillsJars) | Same content, structured packaging | Does structure help?           |
| 5. + SAE                 | Add Structured Agent Execution     | Does execution structure help? |

Each step isolates one variable. If variant 4 outperforms variant 3 with identical knowledge content, the structure is what matters — not just the knowledge.

## Improvement Flywheel

Variant ladders can be pre-planned, but the most effective experiments use **empirically motivated variants** — each exists because the previous variant's analysis revealed a specific gap. This follows the [Improvement Flywheel](https://github.com/markpollack/agento-forge/blob/main/concepts/improvement-flywheel.md) methodology:

```
1. RUN        — Execute a variant and capture journals
2. MEASURE    — Compute scores, traces, behavioral metrics
3. DIAGNOSE   — Convert signals into hypotheses about causes
4. INTERVENE  — Change prompt, KB, tool, workflow, or rubric
5. VERIFY     — Re-run and compare deltas / check for regressions
```

### Iteration metadata

Each variant records what motivated it using `IterationMetadata`:

```java theme={null}
public record IterationMetadata(@Nullable String finding, String hypothesis) {}
```

In the [agent-experiment-template](https://github.com/markpollack/agent-experiment-template), this is configured in `experiment-config.yaml`:

```yaml theme={null}
variants:
  - name: control
    promptFile: v0-naive.txt
    iteration:
      finding: null
      hypothesis: "Establish baseline agent behavior"

  - name: variant-a
    promptFile: v1-hardened.txt
    iteration:
      finding: "v0 BUILD→FIX loop amplification 3.2"
      hypothesis: "Structured execution steps reduce fix loops"
```

This creates an audit trail: for every variant you can trace back to the observation that motivated it and verify whether the hypothesis held.

### Intervention levers

The type of loss determines which lever to pull:

| Lever                   | When to use                                         |
| ----------------------- | --------------------------------------------------- |
| **Prompt**              | Diffuse waste, no dominant failure pattern          |
| **Knowledge / skills**  | Friction loops around a specific knowledge gap      |
| **Execution structure** | Loops around states that could be deterministic     |
| **Model**               | Agent fundamentally cannot perform the task         |
| **Rubric / evaluation** | Judge variance, scores don't correlate with quality |

### Comparison reporting

`GrowthStoryReporter` (in the template) generates a markdown comparison report across variants. It:

* Shows per-judge score deltas, improvements, and regressions for each variant pair
* Flags regressions with explicit warnings when any `ScoreComparison.regressions() > 0`
* Includes iteration motivation (finding + hypothesis) before each variant's scores when `IterationMetadata` is present

The report is written to `analysis/comparison-report.md` and provides the MEASURE output that feeds the next DIAGNOSE step.

## Dataset Design

### Item structure

Each item needs:

* **`developerTask`** — what you're asking the agent to do (natural language)
* **`before/`** — the starting state (real source code)
* **`reference/`** — the correct result (for judge comparison)
* **`bucket`** — difficulty classification (A = easy, B = medium, C = hard)
* **`knowledgeRefs`** — paths to relevant KB entries (relative to `knowledgeBaseDir`)

### Buckets

Use buckets to stratify difficulty:

| Bucket | Typical characteristics                                                    |
| ------ | -------------------------------------------------------------------------- |
| **A**  | Single file, mechanical change, clear instructions                         |
| **B**  | Multi-file, requires understanding, some ambiguity                         |
| **C**  | Cross-cutting concern, requires domain knowledge, creative problem-solving |

### Filtering

Run subsets of the dataset:

```java theme={null}
// Run only bucket A items
ExperimentConfig.builder()
    .itemFilter(ItemFilter.bucket("A"))
    // ...

// Run items with specific tags
ExperimentConfig.builder()
    .itemFilter(ItemFilter.tags("rename", "simple"))
    // ...

// Run a single item by ID
ExperimentConfig.builder()
    .itemFilter(ItemFilter.id("RENAME-001"))
    // ...
```

## ExperimentConfig Reference

| Field               | Required | Default   | Description                                       |
| ------------------- | -------- | --------- | ------------------------------------------------- |
| `experimentName`    | Yes      | —         | Experiment identifier                             |
| `datasetDir`        | Yes      | —         | Path to dataset directory                         |
| `model`             | Yes      | —         | LLM model (`sonnet`, `opus`, `haiku`, or full ID) |
| `promptTemplate`    | Yes      | —         | Template with `{{task}}` and `{{knowledgeRefs}}`  |
| `perItemTimeout`    | Yes      | —         | Timeout per item invocation                       |
| `itemFilter`        | No       | all items | Filter by bucket, tags, ID, status                |
| `knowledgeBaseDir`  | No       | —         | KB root (for ablation tracking)                   |
| `outputDir`         | No       | —         | Directory for workspaces and logs                 |
| `experimentTimeout` | No       | —         | Timeout for entire experiment                     |
| `metadata`          | No       | —         | Arbitrary key-value pairs                         |
| `baselineId`        | No       | —         | Reference experiment for comparison               |

## Result Structure

Results are persisted by `FileSystemResultStore`:

```
results/
└── rename-field-v1/
    ├── index.json                    # Experiment history
    └── a1b2c3d4.json               # Individual experiment result
```

Each result contains:

* Experiment metadata (name, config, git version, timestamps)
* Per-item results (agent output, jury verdict, tokens, cost, duration)
* Aggregate statistics (pass rate, total cost, total duration)

## Cross-Run Comparison

```java theme={null}
ComparisonEngine comparison = new ComparisonEngine();
ComparisonResult diff = comparison.compare(resultA, resultB);
```

The comparison engine aligns items by ID across two experiments and reports per-item and aggregate deltas.

## Related

<CardGroup cols={2}>
  <Card title="Building a Jury" icon="scale-balanced" href="/docs/experiment-driver/jury-system">
    Three-tier evaluation: deterministic, structural, semantic
  </Card>

  <Card title="API Reference" icon="code" href="/docs/experiment-driver/api-reference">
    Full config, dataset format, invoker contract
  </Card>
</CardGroup>