> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Controlled studies measuring what moves the needle for AI agent reliability

Every experiment in this lab follows the same pattern: **define variants, control for one variable, measure with the four-tier jury, analyze behavioral traces.**

We don't just measure whether agents succeed — we measure *how* they behave on the way to success or failure, using [Markov chain analysis](/methodology/markov-fingerprinting) of tool-call traces.

## Active Experiments

<CardGroup cols={2}>
  <Card title="Issue Classification" icon="tags" href="/experiments/issue-classification">
    Infrastructure vs prompts on SWE-bench Lite — does tooling beat prompt engineering at 300-task scale?
  </Card>
</CardGroup>

## Completed Experiments

<CardGroup cols={2}>
  <Card title="Code Coverage v3" icon="check-circle" href="/experiments/code-coverage-v3">
    The exemplar effect — when existing tests use older patterns, skills can't override them. T2 = 0.667 across all 6 runs.
  </Card>

  <Card title="Code Coverage v2" icon="check-circle" href="/experiments/code-coverage-v2">
    Skills vs flat knowledge bases — 7 variants on Spring PetClinic. Skills+preanalysis achieves -31% steps with no quality loss.
  </Card>

  <Card title="Code Coverage v1" icon="check-circle" href="/experiments/code-coverage-v1">
    Knowledge injection baseline — 9 variants, two independent axes discovered.
  </Card>
</CardGroup>

## Upcoming

| Experiment        | Question                                          | Status  |
| ----------------- | ------------------------------------------------- | ------- |
| Code Coverage v4  | Fix the exemplar with a separate upgrade step     | Planned |
| SWE-bench Results | Cross-experiment comparison on standardized tasks | Planned |

***

## Experiment Design Principles

1. **One variable per experiment** — Isolate the thing being tested
2. **Deterministic preprocessing** — Parse inputs before the LLM sees them (zero LLM cost)
3. **Cascaded evaluation** — T0→T1→T2→T3, cheap filters first
4. **Behavioral analysis** — Markov chains on tool-call traces, not just pass/fail
5. **Reproducibility** — All experiment repos are public with full trace data