Requirements judges

When we can write the oracle, we already have excellent machinery: javac, JUnit, and ArchUnit. The gap is a requirement that is written down but needs judgment to establish whether the implementation satisfies it. EarsJudge reads EARS acceptance criteria; Rfc2119Judge reads RFC 2119 architectural constraints. Both run a written requirements document back against the implementation it describes. The input is something somebody wrote before the code existed. The model assesses each requirement; the code decides the verdict.

Type	Purpose
`EarsCriterion`	One acceptance criterion, parsed from the specification
`EarsJudge`	Answers every acceptance criterion
`Rfc2119Constraint`	One architectural constraint, with its RFC 2119 keyword
`Rfc2119Judge`	Answers every architectural constraint
`Observation`	Non-binding evidence noticed while establishing a judgment

The document supplies the roster

This is what separates these from the other judges, and it changes how they aggregate. General jury strategies can drop an ABSTAIN from the vote; since 0.17, that status means undecided, and NOT_APPLICABLE is a separate status for a question that does not apply. Here the roster is fixed: the document says the requirement applies, so “could not be established” is not “does not apply.” The rollup is therefore strict, and abstentions are not dropped:

any ERROR        -> ERROR
else any FAIL    -> FAIL
else any ABSTAIN -> ABSTAIN
else             -> PASS

Fifty-one criteria established and one unsettled is ABSTAIN, not PASS. The specification has not been shown to hold — which is a different statement from it having been shown to fail, and both are different from success. Understand this before composing these judges with anything else, because a strategy that drops abstentions will silently convert “unverified” into “fine.”

AllMustPassStrategy drops abstentions when aggregating; it does not implement this strict roster rollup. Passing 51 PASS judgments and one ABSTAIN through it yields PASS. Since 0.17, a criterion or constraint can declare an optional applicability clause before assessment. Only those conditional items may receive NOT_APPLICABLE, with a mandatory reason; authorized exclusions are left out and counted separately. Excluding an unconditional item, or omitting the reason, is a protocol error. The examples below use unconditional requirements. Both judges were introduced in 0.16.0, which allowed an empty roster to return PASS. In 0.17, construction and rollup each refuse an empty roster: nothing was checked, so nothing can be claimed.

Run the examples

You need Java 21 or newer and Maven 3.9 or newer. These examples run with no credentials, no API key, and no network during execution. Maven needs network access to download dependencies on the first build. JudgeModel is a functional interface, so each example supplies the model call as a fixture lambda. The answers and file-and-line citations are fixtures: these examples verify wiring, parsing, and rollup rather than independently assessing an implementation. Running either judge against a real model needs a backend, its credentials, and access to the implementation being assessed. Create a directory for the examples, with src/main/java beneath it. Save this complete pom.xml in that directory. The BOM supplies both Agent Judge module versions; the five requirements types above live in agent-judge-ai-core.

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>example</groupId>
  <artifactId>requirements-judges-demo</artifactId>
  <version>1.0-SNAPSHOT</version>
  <properties>
    <maven.compiler.release>21</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>io.github.markpollack</groupId>
        <artifactId>agentworks-bom</artifactId>
        <version>1.21.0</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>io.github.markpollack</groupId>
      <artifactId>agent-judge-core</artifactId>
    </dependency>
    <dependency>
      <groupId>io.github.markpollack</groupId>
      <artifactId>agent-judge-ai-core</artifactId>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.14.0</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>3.8.1</version>
      </plugin>
    </plugins>
  </build>
</project>

EARS: an unsettled requirement keeps the judgment open

Save this input document as criteria.md beside pom.xml. The parser reads a numbered heading and the requirement sentence beneath it.

### UC1-AC1: List owners
When the user opens the owner list, the system shall display the registered owners.

### UC1-AC2: Find an owner
When the user searches by last name, the system shall display matching owners.

### UC1-AC3: Handle an empty search
If no owner matches the search, then the system shall display an empty result message.

Save this as src/main/java/EarsExample.java. The fixture establishes two criteria and leaves UC1-AC3 unsettled.

import java.nio.file.Path;
import java.util.Map;

import io.github.markpollack.judge.ai.model.JudgeModel;
import io.github.markpollack.judge.ai.model.JudgeModelResponse;
import io.github.markpollack.judge.ai.requirements.EarsCriterion;
import io.github.markpollack.judge.ai.requirements.EarsJudge;
import io.github.markpollack.judge.context.ExecutionStatus;
import io.github.markpollack.judge.context.JudgmentContext;
import io.github.markpollack.judge.result.Judgment;

public class EarsExample {
    public static void main(String[] args) {
        var criteria = EarsCriterion.from(Path.of("criteria.md"));
        // Fixture answers, not findings from an inspection of real code.
        String answers = """
            UC1-AC1: PASS - OwnerController.java:30 lists owners
            UC1-AC2: PASS - OwnerController.java:45 searches by last name
            UC1-AC3: CANNOT_DETERMINE - the empty result path has not been established
            """;
        JudgeModel model = request ->
            new JudgeModelResponse(answers, "fixture", null, Map.of());
        JudgmentContext context = JudgmentContext.builder()
            .goal("audit the owner search requirements")
            .status(ExecutionStatus.SUCCESS)
            .build();

        Judgment judgment = EarsJudge.create("owner-search", criteria, model).judge(context);
        System.out.println("status     = " + judgment.status());
        System.out.println("reasoning  = " + judgment.reasoning());
        System.out.println("unestablished = " + judgment.metadata().get("unestablished"));
        System.out.println("checks     = " + judgment.checks().size());
    }
}

From the directory containing pom.xml, run these commands in a POSIX shell:

mvn -q compile dependency:build-classpath -Dmdep.outputFile=target/classpath.txt
java -cp "target/classes:$(cat target/classpath.txt)" EarsExample

The judgment is ABSTAIN, and it names the open criterion in both the reasoning and metadata:

status     = ABSTAIN
reasoning  = 2 of 3 established, 1 could not be established: UC1-AC3
unestablished = UC1-AC3
checks     = 3

Each example also prints three SLF4J warning lines on stderr because no logging provider is declared:

SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.

RFC 2119: one violated constraint fails the judgment

Save this input document as rules.md beside pom.xml. Each numbered rule carries its keyword, requirement, and reason.

### RULE-1
**MUST** keep transactions on the service boundary.
**Reason:** Consistency across operations.

### RULE-2
**MUST NOT** expose entities from controllers.
**Reason:** Keep persistence details out of the public API.

### RULE-3
**MUST** acquire the owner lock before the appointment lock on both paths.
**Reason:** Keep lock ordering consistent.

Save this as src/main/java/Rfc2119Example.java. The fixture reports a violation of RULE-2; its citations are invented example evidence.

import java.nio.file.Path;
import java.util.Map;

import io.github.markpollack.judge.ai.model.JudgeModel;
import io.github.markpollack.judge.ai.model.JudgeModelResponse;
import io.github.markpollack.judge.ai.requirements.Rfc2119Constraint;
import io.github.markpollack.judge.ai.requirements.Rfc2119Judge;
import io.github.markpollack.judge.context.ExecutionStatus;
import io.github.markpollack.judge.context.JudgmentContext;
import io.github.markpollack.judge.result.Judgment;

public class Rfc2119Example {
    public static void main(String[] args) {
        var constraints = Rfc2119Constraint.from(Path.of("rules.md"));
        // Fixture answers, not findings from an inspection of real code.
        String answers = """
            RULE-1: PASS - ClinicService.java:42 is annotated
            RULE-2: FAIL - OwnerController.java:60 returns the entity directly
            RULE-3: PASS - both paths take owner before appointment
            """;
        JudgeModel model = request ->
            new JudgeModelResponse(answers, "fixture", null, Map.of());
        JudgmentContext context = JudgmentContext.builder()
            .goal("audit the architecture constraints")
            .status(ExecutionStatus.SUCCESS)
            .build();

        Judgment judgment = Rfc2119Judge.create("architecture", constraints, model).judge(context);
        System.out.println("status    = " + judgment.status());
        System.out.println("reasoning = " + judgment.reasoning());
        for (var check : judgment.checks()) {
            System.out.println("  " + (check.passed() ? "PASS" : "not passed")
                + "  " + check.message());
        }
    }
}

Run it from the same directory:

mvn -q compile dependency:build-classpath -Dmdep.outputFile=target/classpath.txt
java -cp "target/classes:$(cat target/classpath.txt)" Rfc2119Example

The judgment is FAIL, with the evidence for all three constraints retained:

status    = FAIL
reasoning = 2 of 3 hold, 1 violated
  PASS  ClinicService.java:42 is annotated
  not passed  OwnerController.java:60 returns the entity directly
  PASS  both paths take owner before appointment

What happened on a real implementation

A spec-driven PetClinic branch began with 438 requirements across fifteen documents, written before the code by the same pipeline that then wrote the code, wrote 290 passing tests, and reviewed its own work without finding anything wrong. The case study read two of those fifteen documents back against the implementation. Behaviour: 51 of 52 EARS criteria established, none failed, and UC6-AC41 unsettled — ABSTAIN, not a pass. Architecture, on the same commit: 13 RFC 2119 constraints, five passed and eight failed. Those eight failures are addresses for investigation, not eight bugs. One was followed to its consequence: a lock-ordering violation between two paths that a scheduled sweeper causes to meet. The lock ordering and its reachability were verified by hand in the source. Continue with the PetClinic learning path, the conference talk, or the runnable case study.

Projects

AgentWorks

Supporting Projects

Migration

Explorations

Older / Inactive Projects

The document supplies the roster

Run the examples

EARS: an unsettled requirement keeps the judgment open

RFC 2119: one violated constraint fails the judgment

What happened on a real implementation

​The document supplies the roster

​Run the examples

​EARS: an unsettled requirement keeps the judgment open

​RFC 2119: one violated constraint fails the judgment

​What happened on a real implementation

The document supplies the roster

Run the examples

EARS: an unsettled requirement keeps the judgment open

RFC 2119: one violated constraint fails the judgment

What happened on a real implementation