Chapter 3 – Validators: What Checks Can Establish

{
  "validator": "json_schema",
  "status": "fail",
  "artifact": "generated/pr_summary_bad.json",
  "errors": [
    {
      "path": "reviewer_suggestions",
      "message": "'charlie_qa' is not of type 'array'"
    }
  ]
}

The model produced plausible-looking output. A cheap deterministic check caught a structural bug before review and emitted a precise finding: the property checked, the affected path, and the reason for failure.

The book calls a declared check a Validator. A Validator measures one selected property and turns “looks fine” into structured evidence. It does not decide the entire outcome or make its own result authoritative. In the book’s role vocabulary, it is a Sensor rather than a Judge. In the narrow Chapter 1 loop, each result has one fixed destination. When the same evidence could support several allowed routes or requires interpretation, Chapter 5 introduces a separate Judge.

Validators are not omniscient. They are authored measurements of what matters. A passing check establishes only the property encoded by its rule and exercised by its evidence.

Checks as Workflow Evidence

Validators may be scripts, library calls, or explicit workflow steps. What matters is a stable input, a structured result, and a traceable place in the workflow:

The same artifact checks can grade output from one model call, an agent harness, or a child workflow. The enclosing workflow still governs how the candidate was produced: tool use, attempted effects, declared outputs, budgets, and provenance.

The separation is not between candidate production and the Workflow’s checks. One Workflow may propose code, add or modify tests, run existing unit and integration suites, and retain every result. Once recorded with provenance, those test artifacts and results are evidence. Their origin changes what they can establish.

An unchanged required test can corroborate a behavior claim when the candidate could not alter its acceptance meaning, including its harness and required configuration. A new or modified candidate test is evidence about the joint code-and-test proposal. It may reproduce a defect, state an example, or improve regression coverage, but it cannot by itself redefine success. A Mission may authorize a protected protocol for judging such tests. For a bug fix, that protocol might require the new test to fail against the pinned baseline, pass against the candidate, and survive a declared mutation check. If the proposed test should become a protected Validator for future Missions, it follows the separate Governance Mission and Adoption path.

Run cheap checks first. Parsers, schemas, formatting, and types can reject malformed candidates before slower behavioral or semantic evaluation. Failing early is useful only if every required check is still represented in the final evidence; a fast pass is not a substitute for the complete set of checks required for advancement.

What a Check Relies On

Classify a Validator by what grounds its rule, not merely by whether it returns the same result twice.

The opening schema result establishes that reviewer_suggestions is an array. It does not establish that the suggested reviewers exist, are available, or are authorized. A separate system-specific check could compare the names with an approved directory. Likewise:

• a compatibility check can reject an undeclared breaking OpenAPI change;
• a coupled-surface check can require documentation when a public interface changes;
• a safety rule can require additional evidence when authentication or encryption surfaces change.

These rules should emit stable findings rather than reassuring prose. Another workflow component must be able to route the result without guessing what the check meant.

All of these checks, together with the budgets, effect limits, and gates that surround them, form an enforcement boundary the book calls Physics. The name is a control-model metaphor; it does not claim that the checks are complete truth.

Authored Rules Remain Partial

Because checks define advancement, ordinary candidate work must not be able to weaken the required rules that judge it. Core invariants stay outside candidate authority. AI may produce a candidate test, schema, or policy and the Workflow may run and record it under the declared protocol. It becomes a protected rule for future runs only through the separate governance path.

Validators themselves need cases on both sides of their boundary. A check that has never demonstrated a meaningful pass and failure is an assertion of confidence, not evidence of discrimination.

Four related jobs are easy to blur:

• Verification checks that an artifact is formed or aligned as declared.
• Validation asks whether the candidate satisfies the adopted intent and acceptance criteria.
• Evaluation supplies heuristic or model-based evidence when a property cannot be reduced to a deterministic rule.
• Runtime assurance observes whether the admitted system continues to operate within selected bounds.

All four produce partial evidence. None allows a candidate or evaluator to widen authority or admit or adopt its own result.

False Positives and False Negatives

Authored checks fail in two directions:

The answer is not to abandon validation or weaken unrelated rules. Adjust the boundary, add the missing evidence, and retain the case so the same mistake remains visible.

When Evaluation Must Remain Probabilistic

Prefer deterministic checks when they can settle the property. The stricter case arises when a consequential judgment, such as whether a migration plan addresses rollback risk, requires model-based evaluation. A second model call is not automatically independent: if the generator and evaluator share a model family, provider, rubric, Context Packet, retrieval source, or omitted fact, they can agree for the same wrong reason.

When probabilistic evaluation participates in a Validator or Judge, the Mission proposal must bind a small evaluation contract under adopted policy before final review and Activation:

• Protected protocol: the contract binds the evaluator, rubric, tools, relevant data, decoding controls, and retry limits. The bound protocol remains outside candidate authority, with enough retained provenance to reconstruct each result.
• Known shared dependencies: the contract binds selectable shared models, providers, context sources, retrieval sources, fixtures, and derived-input rules. It records their actual identities so shared failure modes remain visible.
• Calibration: the contract includes pass, fail, boundary, and adversarial fixtures kept outside candidate authority and appropriate to the property. Any materially changed protocol requires revalidation before it can support Activation.
• Declared decision rule: the contract selects sampling or an ensemble in proportion to consequence and aggregates through a fixed rule. Sampling until the desired answer appears is not a valid decision rule, and repeated correlated judgments do not provide independent corroboration.
• Inconclusive outcome: stale calibration, malformed output, missing inputs, out-of-policy configuration, or disagreement beyond the declared threshold must not advance a required property.

The source or decision procedure allowed to settle a property is called an oracle. For a high-consequence property checked under a protected protocol, a probabilistic evaluator must not be the only oracle unless adopted policy has explicitly accepted the measured error of that evaluation protocol for that class of work. Otherwise it needs corroborating evidence from deterministic checks, behavioral tests, runtime observations, review independent of candidate production, or another declared oracle.

The runtime still enforces evidence requirements, scope, budgets, transition eligibility, stopping, and admission outside the model. Calibration makes a bounded protocol measurable; it does not turn a model into ground truth.

Suites and Coverage

One Validator checks one property. A reusable suite contains the checks available for a surface. For one Mission, the work contract names which checks are required. The completed evidence records which required checks actually ran and their results.

A practical coverage strategy begins with cheap structural checks and the highest-consequence known failure, then adds checks as incidents reveal missing properties. Greater strictness is worth adding only when the rule is stable enough to own; expensive behavioral or probabilistic checks remain for the work that justifies them.

By declaring which checks count, what they establish, and how their findings affect advancement, the organization makes another part of the change process explicit and enforceable. More candidates need not mean proportionally more unvalidated claims reaching review or production.

That closes Part I: one workflow with declared effects, one probabilistic change step between deterministic Prep and Validation, and partial evidence evaluated under acceptance rules the candidate cannot rewrite. Part II explains why those mechanics are necessary and how retries and context remain bounded.