Every experiment in this lab follows the same pattern: define variants, control for one variable, measure with the four-tier jury, analyze behavioral traces. We don’t just measure whether agents succeed — we measure how they behave on the way to success or failure, using Markov chain analysis of tool-call traces.Documentation Index
Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
Use this file to discover all available pages before exploring further.
Active Experiments
Issue Classification
Infrastructure vs prompts on SWE-bench Lite — does tooling beat prompt engineering at 300-task scale?
Completed Experiments
Code Coverage v3
The exemplar effect — when existing tests use older patterns, skills can’t override them. T2 = 0.667 across all 6 runs.
Code Coverage v2
Skills vs flat knowledge bases — 7 variants on Spring PetClinic. Skills+preanalysis achieves -31% steps with no quality loss.
Code Coverage v1
Knowledge injection baseline — 9 variants, two independent axes discovered.
Upcoming
| Experiment | Question | Status |
|---|---|---|
| Code Coverage v4 | Fix the exemplar with a separate upgrade step | Planned |
| SWE-bench Results | Cross-experiment comparison on standardized tasks | Planned |
Experiment Design Principles
- One variable per experiment — Isolate the thing being tested
- Deterministic preprocessing — Parse inputs before the LLM sees them (zero LLM cost)
- Cascaded evaluation — T0→T1→T2→T3, cheap filters first
- Behavioral analysis — Markov chains on tool-call traces, not just pass/fail
- Reproducibility — All experiment repos are public with full trace data