The Insight
Agent tool-call sequences are approximately first-order Markov chains. The probability of the next tool call depends primarily on the current tool call, not the full history. This means we can apply well-studied stochastic process analysis to understand agent behavior.The 9-State Taxonomy
Every tool call maps to one of nine behavioral states:What Markov Analysis Reveals
Transition Probability Engineering (TPE)
The transition matrix P(next_state | current_state) is the agent’s “behavioral fingerprint.” Different variants produce measurably different fingerprints.Key Metrics
- Expected steps to completion — From the fundamental matrix N = (I - Q)^
- P(success) — Absorbing chain probability of reaching success vs. failure
- Thrash score — Loop amplification in BUILD→TEST→EDIT cycles
- JAR cluster % — Time spent in dependency inspection (correlates with knowledge availability)
What We’ve Found
From Code Coverage v1:- Knowledge reduces JAR inspection — Variants with domain knowledge spend less time inspecting dependencies
- Thrashing predicts failure — High BUILD→TEST→EDIT loop counts correlate with lower T3 scores
- SAE changes the fingerprint — Structured Agent Execution produces measurably different transition matrices
- Two independent axes — Knowledge injection and prompt hardening affect different parts of the transition matrix
Tools
The analysis pipeline is implemented in the markov-agent-analysis Python library.build_absorbing_chain_from_traces() — transforms raw tool-call logs into absorbing Markov chains.
Role in the Growth Cycle
Markov analysis is the primary diagnostic lens in the DIAGNOSE step of the Improvement Flywheel. It converts raw tool-call traces into actionable signals about where the agent gets stuck and why.Loop Amplification Signals
Transition Gap Signals
Failure Pattern Signals
Loop Type Classification
Not all loops are problems. Classify before intervening:Related
Improvement Flywheel
The feedback loop that this analysis drives
Forge Methodology
Define → Forge → Run → Grow pipeline
Blog: I Read My Agent's Diary
Narrative walkthrough of the Markov analysis
Agent Journal
The trace capture layer that feeds this analysis