> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Creating Experiments

> Design datasets, define variant ladders, filter items, and analyze results

## Design Philosophy

Every experiment tests a hypothesis about what makes agents better. The experiment driver makes the independent variables explicit:

| Variable | How you control it |
| - | - |
| **Knowledge** | `knowledgeRefs` in dataset items, `knowledgeBaseDir` in config |
| **Prompt structure** | `promptTemplate` with `{{task}}` and `{{knowledgeRefs}}` placeholders |
| **Model** | `model` field in config |
| **Execution strategy** | Your `AgentInvoker` implementation |
| **Evaluation criteria** | Your `Jury` wiring |

## Variant Ladders

The most informative experiments use a **progressive variant ladder** — each variant adds one thing to the previous:

| Variant | Change from previous | Tests |
| - | - | - |
| 1. Simple prompt | — (baseline) | Model's raw capability |
| 2. + System prompt | Add domain instructions | Does framing help? |
| 3. + Knowledge base | Add `knowledgeRefs` | Does knowledge help? |
| 4. + Skills (SkillsJars) | Same content, structured packaging | Does structure help? |
| 5. + SAE | Add Structured Agent Execution | Does execution structure help? |

Each step isolates one variable. If variant 4 outperforms variant 3 with identical knowledge content, the structure is what matters — not just the knowledge.

## Improvement Flywheel

Variant ladders can be pre-planned, but the most effective experiments use **empirically motivated variants** — each exists because the previous variant's analysis revealed a specific gap. This follows the [Improvement Flywheel](https://github.com/markpollack/agento-forge/blob/main/concepts/improvement-flywheel.md) methodology:

```
1. RUN        — Execute a variant and capture journals
2. MEASURE    — Compute scores, traces, behavioral metrics
3. DIAGNOSE   — Convert signals into hypotheses about causes
4. INTERVENE  — Change prompt, KB, tool, workflow, or rubric
5. VERIFY     — Re-run and compare deltas / check for regressions
```

### Iteration metadata

Each variant records what motivated it using `IterationMetadata`:

```java theme={null}
public record IterationMetadata(@Nullable String finding, String hypothesis) {}
```

In the [agent-experiment-template](https://github.com/markpollack/agent-experiment-template), this is configured in `experiment-config.yaml`:

```yaml theme={null}
variants:
  - name: control
    promptFile: v0-naive.txt
    iteration:
      finding: null
      hypothesis: "Establish baseline agent behavior"

  - name: variant-a
    promptFile: v1-hardened.txt
    iteration:
      finding: "v0 BUILD→FIX loop amplification 3.2"
      hypothesis: "Structured execution steps reduce fix loops"
```

This creates an audit trail: for every variant you can trace back to the observation that motivated it and verify whether the hypothesis held.

### Intervention levers

The type of loss determines which lever to pull:

| Lever | When to use |
| - | - |
| **Prompt** | Diffuse waste, no dominant failure pattern |
| **Knowledge / skills** | Friction loops around a specific knowledge gap |
| **Execution structure** | Loops around states that could be deterministic |
| **Model** | Agent fundamentally cannot perform the task |
| **Rubric / evaluation** | Judge variance, scores don't correlate with quality |

### Comparison reporting

`GrowthStoryReporter` (in the template) generates a markdown comparison report across variants. It:

* Shows per-judge score deltas, improvements, and regressions for each variant pair
* Flags regressions with explicit warnings when any `ScoreComparison.regressions() > 0`
* Includes iteration motivation (finding + hypothesis) before each variant's scores when `IterationMetadata` is present

The report is written to `analysis/comparison-report.md` and provides the MEASURE output that feeds the next DIAGNOSE step.

## Dataset Design

### Item structure

Each item needs:

* **`developerTask`** — what you're asking the agent to do (natural language)
* **`before/`** — the starting state (real source code)
* **`reference/`** — the correct result (for judge comparison)
* **`bucket`** — difficulty classification (A = easy, B = medium, C = hard)
* **`knowledgeRefs`** — paths to relevant KB entries (relative to `knowledgeBaseDir`)

### Buckets

Use buckets to stratify difficulty:

| Bucket | Typical characteristics |
| - | - |
| **A** | Single file, mechanical change, clear instructions |
| **B** | Multi-file, requires understanding, some ambiguity |
| **C** | Cross-cutting concern, requires domain knowledge, creative problem-solving |

### Filtering

Run subsets of the dataset:

```java theme={null}
// Run only bucket A items
ExperimentConfig.builder()
    .itemFilter(ItemFilter.bucket("A"))
    // ...

// Run items with specific tags
ExperimentConfig.builder()
    .itemFilter(ItemFilter.tags("rename", "simple"))
    // ...

// Run a single item by ID
ExperimentConfig.builder()
    .itemFilter(ItemFilter.id("RENAME-001"))
    // ...
```

## ExperimentConfig Reference

| Field | Required | Default | Description |
| - | - | - | - |
| `experimentName` | Yes | — | Experiment identifier |
| `datasetDir` | Yes | — | Path to dataset directory |
| `model` | Yes | — | LLM model (`sonnet`, `opus`, `haiku`, or full ID) |
| `promptTemplate` | Yes | — | Template with `{{task}}` and `{{knowledgeRefs}}` |
| `perItemTimeout` | Yes | — | Timeout per item invocation |
| `itemFilter` | No | all items | Filter by bucket, tags, ID, status |
| `knowledgeBaseDir` | No | — | KB root (for ablation tracking) |
| `outputDir` | No | — | Directory for workspaces and logs |
| `experimentTimeout` | No | — | Timeout for entire experiment |
| `metadata` | No | — | Arbitrary key-value pairs |
| `baselineId` | No | — | Reference experiment for comparison |

## Result Structure

Results are persisted by `FileSystemResultStore`:

```
results/
└── rename-field-v1/
    ├── index.json                    # Experiment history
    └── a1b2c3d4.json               # Individual experiment result
```

Each result contains:

* Experiment metadata (name, config, git version, timestamps)
* Per-item results (agent output, jury verdict, tokens, cost, duration)
* Aggregate statistics (pass rate, total cost, total duration)

## Cross-Run Comparison

```java theme={null}
ComparisonEngine comparison = new ComparisonEngine();
ComparisonResult diff = comparison.compare(resultA, resultB);
```

The comparison engine aligns items by ID across two experiments and reports per-item and aggregate deltas.

## Related

<CardGroup cols={2}>
  <Card title="Building a Jury" icon="scale-balanced" href="/docs/agent-experiment/jury-system">
    Three-tier evaluation: deterministic, structural, semantic
  </Card>

  <Card title="API Reference" icon="code" href="/docs/agent-experiment/api-reference">
    Full config, dataset format, invoker contract
  </Card>
</CardGroup>