> ## Documentation Index
> Fetch the complete documentation index at: https://lab.pollack.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Diagnostic Reasoning

> Classify failure gaps, generate remediation actions, and feed the improvement flywheel

## What Diagnostic Reasoning Does

After an experiment runs and the jury produces verdicts, the diagnostic system answers: **why did items fail, and what should change?**

It works in three stages:

1. **Gap classification** — map each failing verdict to a category (agent error, plan gap, missing knowledge, etc.)
2. **Deterministic reasoning** — apply rule-based logic to produce actionable fixes
3. **LLM fallback** — for checks the rules can't resolve, an LLM analyzes execution traces and proposes new artifacts

```
ExperimentResult (jury verdicts)
        │
        ▼
  DiagnosticAnalyzer
   └─ GapClassifier: verdict → DiagnosticCheck (with GapCategory)
        │
        ▼
  DiagnosticReport (per-item checks, gap distribution)
        │
        ▼
  DiagnosticReasoner
   ├─ DeterministicReasoner → RemediationAction[]
   └─ LlmDiagnosticReasoner → RemediationAction[] + RemediationProposal[]
        │
        ▼
  RemediationReport (actions, proposals, unresolved checks)
```

## Gap Categories

Every failing verdict check is classified into a gap category that identifies **where in the system** the problem lives:

| Category              | Meaning                                             | Fix target                |
| --------------------- | --------------------------------------------------- | ------------------------- |
| `AGENT_EXECUTION_GAP` | Plan was correct, agent didn't follow it            | Agent prompting           |
| `PLAN_GENERATION_GAP` | Plan didn't cover this pattern                      | Planner or planning model |
| `KB_GAP`              | Knowledge base doesn't cover this pattern           | Add KB entry              |
| `TOOL_GAP`            | No deterministic tool handles this                  | Build new tool            |
| `ANALYSIS_GAP`        | Static analysis missed a signal                     | Improve analysis tools    |
| `CRITERIA_GAP`        | VERIFY criteria were redundant/ambiguous/missing    | Criteria generation       |
| `EVALUATION_GAP`      | Jury itself is wrong (false positive/negative)      | Judge calibration         |
| `STOCHASTICITY_GAP`   | Same config produces different outcomes across runs | Requires N≥3 runs         |

## DiagnosticAnalyzer

Entry point for analysis. Takes an `ExperimentResult` and produces a `DiagnosticReport`:

```java theme={null}
DiagnosticAnalyzer analyzer = new DiagnosticAnalyzer(gapClassifier);
DiagnosticReport report = analyzer.analyze(experimentResult);

report.distribution().dominant();  // e.g., AGENT_EXECUTION_GAP
report.items();                    // per-item ItemDiagnostic list
report.recommendations();         // human-readable suggestions
```

### GapClassifier

Assigns gap categories to verdict checks. The default `HeuristicGapClassifier` uses 22 judge-specific classification rules to map failures to categories based on the judge name, check content, and available analysis data.

```java theme={null}
GapClassifier classifier = new HeuristicGapClassifier();
List<DiagnosticCheck> checks = classifier.classify(verdict, analysisEnvelope, executionPlan);
```

### DiagnosticReport

| Field             | Type                   | Description                                  |
| ----------------- | ---------------------- | -------------------------------------------- |
| `experimentId`    | `String`               | Experiment run ID                            |
| `items`           | `List<ItemDiagnostic>` | Per-item classified checks with dominant gap |
| `distribution`    | `GapDistribution`      | Aggregate counts and fractions by category   |
| `recommendations` | `List<String>`         | Human-readable improvement suggestions       |

`GapDistribution.dominant()` returns the most frequent gap category — the highest-leverage fix target.

## DiagnosticReasoner

Transforms a `DiagnosticReport` into actionable remediation:

```java theme={null}
public interface DiagnosticReasoner {
    RemediationReport reason(DiagnosticReport report, ReasoningContext context);
}
```

### ReasoningContext

Provides the full data menu for reasoning — analysis output, execution plan, trajectory exhaust, and file pointers:

| Field            | Type                 | Description                             |
| ---------------- | -------------------- | --------------------------------------- |
| `analysis`       | `AnalysisEnvelope`   | Static analysis data (from pipeline)    |
| `plan`           | `ExecutionPlan`      | Execution roadmap (from pipeline)       |
| `availableTools` | `Set<String>`        | Tools available to the agent            |
| `phases`         | `List<PhaseCapture>` | Agent thinking, tool calls, and results |

Helper methods: `unusedTools()`, `errorToolResults()`, `toolUsesByName(String)`.

### DeterministicReasoner

Rule-based reasoning with two rule categories:

**Verdict rules** (fire on failing judge checks):

* Pattern-match on gap category and structured analysis data
* Target specific components: planner-prompt, pom-upgrader, agent-prompt, dependency-analysis

**Trajectory rules** (fire on execution context regardless of judge outcomes):

* Detect efficiency gaps where the agent recovered but deterministic tools could have prevented the problem
* Examples: unused tools, implicit JDK dependencies, repeated build errors, format violations

```java theme={null}
DeterministicReasoner reasoner = new DeterministicReasoner();
RemediationReport report = reasoner.reason(diagnosticReport, context);

report.remediations();      // actionable fixes
report.unresolvedChecks();  // checks the rules couldn't resolve
```

### LlmDiagnosticReasoner

Handles checks that deterministic rules can't resolve. Analyzes execution traces (thinking, tool calls, results) and produces:

* **RemediationActions** — fixes with `LLM_INFERRED` confidence
* **RemediationProposals** — new deterministic artifacts (rules, KB entries, tool specs) for the flywheel

```java theme={null}
public interface LlmDiagnosticReasoner {
    LlmReasoningResult reasonUnresolved(
        List<DiagnosticCheck> unresolvedChecks, ReasoningContext context);
}
```

### CompositeDiagnosticReasoner

Chains deterministic and LLM reasoning:

```java theme={null}
CompositeDiagnosticReasoner reasoner = new CompositeDiagnosticReasoner(
    new DeterministicReasoner(), llmReasoner);

RemediationReport report = reasoner.reason(diagnosticReport, context);
```

1. Deterministic layer runs first (fast, proof-based)
2. If unresolved checks remain and an LLM fallback is available, forward them to the LLM
3. Merge results into a single `RemediationReport`

If deterministic reasoning resolves all checks, the LLM is never called.

## RemediationReport

| Field              | Type                        | Description                            |
| ------------------ | --------------------------- | -------------------------------------- |
| `experimentId`     | `String`                    | Experiment run ID                      |
| `remediations`     | `List<RemediationAction>`   | Actionable fixes, highest-impact first |
| `proposals`        | `List<RemediationProposal>` | New artifacts proposed by LLM          |
| `unresolvedChecks` | `List<DiagnosticCheck>`     | Checks neither layer could resolve     |

### RemediationAction

Each action targets a specific component and carries a confidence level:

| Field        | Type         | Description                                                                                                                 |
| ------------ | ------------ | --------------------------------------------------------------------------------------------------------------------------- |
| `target`     | `String`     | Component to fix (e.g., "pom-upgrader", "agent-prompt")                                                                     |
| `actionType` | `ActionType` | `ADD_RULE`, `ENHANCE_TOOL`, `IMPROVE_PROMPT`, `ADD_KB_ENTRY`, `ENHANCE_ANALYSIS`, `CALIBRATE_JUDGE`, `MANUAL_INVESTIGATION` |
| `summary`    | `String`     | One-line description                                                                                                        |
| `detail`     | `String`     | Full explanation                                                                                                            |
| `confidence` | `Confidence` | `DETERMINISTIC`, `HEURISTIC`, or `LLM_INFERRED`                                                                             |

### RemediationProposal

LLM-discovered patterns that can become new deterministic infrastructure:

| Field              | Type           | Description                                                                                                     |
| ------------------ | -------------- | --------------------------------------------------------------------------------------------------------------- |
| `proposalType`     | `ProposalType` | `NEW_REASONER_RULE`, `KB_ENTRY_DRAFT`, `TOOL_ENHANCEMENT`, `PROMPT_PATCH`, `NEW_TOOL_SPEC`, `JUDGE_CALIBRATION` |
| `target`           | `String`       | Component the proposal targets                                                                                  |
| `proposalMarkdown` | `String`       | Full specification for review                                                                                   |
| `confidence`       | `Confidence`   | Always `LLM_INFERRED`                                                                                           |

## The Flywheel

RemediationProposals are the flywheel mechanism. When the LLM discovers a novel failure pattern:

1. It creates a `RemediationProposal` (e.g., a new deterministic reasoner rule)
2. A human reviews and applies the proposal
3. The new rule is added to `DeterministicReasoner`
4. On the next run, that pattern is resolved deterministically — faster, cheaper, and with higher confidence

Over time, the LLM fallback is invoked less as more patterns move into deterministic rules.

## Cross-Run Aggregation

`DiagnosticAggregator` analyzes multiple `DiagnosticReport` instances from repeated runs to detect stochasticity:

```java theme={null}
DiagnosticAggregator aggregator = new DiagnosticAggregator();
AggregatedDiagnostic agg = aggregator.aggregate(List.of(report1, report2, report3));

agg.stochasticItems();    // items with different dominant gaps across runs
agg.stableItems();        // items that fail consistently for the same reason
agg.stabilityFraction();  // fraction of items that are stable
```

An item classified as `AGENT_EXECUTION_GAP` in one run and `PLAN_GENERATION_GAP` in another is flagged as stochastic. Stochastic items need N≥3 runs before you can draw reliable conclusions. Stable items are immediately actionable.

## Efficiency Evaluation

`EfficiencyEvaluator` scores execution efficiency across four metrics:

| Metric            | Weight | What it measures                               |
| ----------------- | ------ | ---------------------------------------------- |
| `buildErrors`     | 0.35   | How many build errors occurred before success  |
| `toolUtilization` | 0.25   | Fraction of available tools actually used      |
| `cost`            | 0.20   | LLM cost relative to a configurable ceiling    |
| `recoveryCycles`  | 0.20   | How many error-recovery loops the agent needed |

```java theme={null}
EfficiencyConfig config = new EfficiencyConfig(5.0, defaultWeights, 8);
EfficiencyReport report = evaluator.evaluate(result, context, config);

report.compositeScore();  // weighted average [0, 1] where 1.0 = perfect
report.checks();          // per-metric breakdown
```

Metrics gracefully degrade — if data for a metric is missing, the metric is omitted rather than failing.

## Behavioral Diagnostics (Markov Analysis)

Gap classification and remediation operate on **judge verdicts** — the outcome layer. A complementary diagnostic lens operates on **tool-call sequences** — the behavioral layer. The [agent-experiment-template](https://github.com/markpollack/agent-experiment-template) includes Markov chain analysis scripts that reveal *how* the agent behaves, not just *what* it produces.

### Loop amplification

The Markov analysis computes **loop amplification** — how many times the agent revisits a state before moving forward. High amplification indicates friction or failure loops:

| Signal                               | Diagnosis                                                   | Intervention                                                       |
| ------------------------------------ | ----------------------------------------------------------- | ------------------------------------------------------------------ |
| Amplification > 2.0 on BUILD→FIX     | Agent in a fix loop — build fails, fix fails, rebuild fails | Knowledge (add fix patterns) or execution structure (pre-validate) |
| Amplification > 2.0 on SEARCH states | Agent searching for something it can't find                 | Knowledge (add the target information)                             |
| Amplification > 2.0 on EXPLORE       | Agent reading many files without progress                   | Prompt (clarify task decomposition) or pre-analysis script         |

### Loop types

Not all loops are problems. Classify before intervening:

| Loop type      | Pattern                                | Action                           |
| -------------- | -------------------------------------- | -------------------------------- |
| **Productive** | WRITE → VERIFY → FIX → VERIFY          | Leave it alone                   |
| **Friction**   | SEARCH → READ → SEARCH → READ          | Add knowledge or routing         |
| **Failure**    | BUILD → FIX → BUILD → FIX (same error) | Change strategy, not retry count |
| **Diagnostic** | BUILD → ERROR → READ\_LOG → FIX        | Leave it alone                   |
| **Degenerate** | EXPLORE → EXPLORE → EXPLORE            | Agent is stuck — intervene       |

### Interpretation output

The template's `make_markov_analysis.py` writes `analysis/markov-interpretation.md` with:

* Per-variant loop amplification summary with threshold-based classification
* Recommended intervention lever for each high-amplification state
* Suggested next variant with a hypothesis template

This connects the behavioral DIAGNOSE step to the flywheel's INTERVENE step — the interpretation tells you *which lever to pull* based on measured state-transition patterns. See the [Improvement Flywheel](https://github.com/markpollack/agento-forge/blob/main/concepts/improvement-flywheel.md) for the full methodology.

## Related

<CardGroup cols={2}>
  <Card title="Pipeline" icon="diagram-project" href="/docs/experiment-driver/pipeline">
    Analyze, plan, and execute — the three pipeline phases
  </Card>

  <Card title="Jury System" icon="scale-balanced" href="/docs/experiment-driver/jury-system">
    Three-tier evaluation: deterministic, structural, semantic
  </Card>
</CardGroup>