Research Foundations

Agent Judge evaluates whether work satisfies an explicit definition of done, using executable evidence and independent checks—not whether it resembles one reference answer.

​

The motivating failure

The investigation began when exact reference-file comparison rated two independently successful Spring Boot migrations as failures. Both implementations built, but each made valid choices that differed from the reference, including the framework version it targeted. The comparison answered “Does this look like the reference?” when the useful question was “Does this satisfy the migration goal?” Exact comparison is still appropriate when exact output is part of the specification. For open-ended generation, refactoring, and migration, it should be one possible check rather than the complete definition of correctness.

​

Five principles

These are design principles, not claims that Agent Judge implements every technique proposed in the literature. Calibrated guarantees of human agreement, dynamic judge-team selection, multi-round debate, and formal program-equivalence proofs remain research or application-level concerns.

​

Ground semantic judgment in tools

Research on tool-augmented LLM evaluation found that web search and code execution can improve judge performance in many, though not all, settings. Code-evaluation systems likewise use execution and other tools to supplement model judgment. The practical implication is bounded: a semantic judge can add value, but it should not replace a build, test, or runtime check when one is available. Agent Judge therefore treats deterministic and execution-based checks as first-class judges. Model-backed judges are intended for criteria-based questions that the available tools cannot settle directly.

​

Allow more than one correct implementation

Program-equivalence research shows that structural difference is not evidence of behavioral difference, while current models still struggle with difficult equivalence cases. Migration benchmarks and production migration systems evaluate combinations of compilation, tests, dependency state, structural invariants, and runtime behavior rather than relying on textual similarity alone. This leads to a requirements-based evaluation model: specify what must remain true, what must change, and the evidence that demonstrates both.

​

Use cascades and juries deliberately

Selective-evaluation research motivates starting with lower-cost evaluators and escalating uncertain cases. Ensemble-judge research highlights the value—and limitations—of combining perspectives. Agent Judge exposes those choices as application policy: which judges participate, how their findings aggregate, how errors are handled, and when another tier runs. A jury is therefore a composition of visible judgments, not merely several models producing one opaque score.

​

Protect the definition of done

Migration research also documents why weak success criteria are dangerous. An agent can appear to improve a metric by removing tests, reducing exercised behavior, or satisfying only a superficial similarity check. Coverage preservation, compilation, test execution, and project-specific structural invariants make such shortcuts observable. Agent Judge supplies mechanisms for these safeguards. The application still owns its definition of done; no generic judge suite can infer all of a project’s requirements automatically.

​

Source map

​

Evaluation architectures

• Findeis et al., Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
• Jung et al., Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
• Li et al., LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• Wang et al., CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
• Zhou et al., An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
• Yu, When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
• He et al., LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead

​

Equivalence and verification

• Wei et al., EquiBench: Benchmarking Code Reasoning Capabilities of Large Language Models via Equivalence Checking
• Dilhara et al., Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example

​

Migration evaluation

• Ziftci et al., Migrating Code At Scale With LLMs At Google
• Liu et al., MIGRATION-BENCH: Repository-Level Code Migration Benchmark from Java 8
• May et al., FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration
• Amin et al., JMigBench: A Benchmark for Evaluating LLMs on Source Code Migration
• Cheng et al., CODEMENV: Benchmarking Large Language Models on Code Migration

Results and numerical claims in these sources belong to their respective studies. They are research inputs to Agent Judge’s design, not Agent Judge benchmark results.

​

Related

• Design philosophy
• Built-in judges
• Jury system
• Executable tutorial