A client names an eval run's modes and its regression baseline

Summary

An eval run on the v2 wire cannot be told which RFC 0081 modes to run or which prior run is its regression baseline: the createRun eval body carries only mode, evalSuiteRef and agentId. So RFC 0244 §F's paired verdict, which both production hosts advertise, cannot be caused by any client, and its rows cannot be witnessed. This RFC adds two optional createRun properties, evalModes and baselineRunId, and an optional facet member, agents.evalSuite.baselineSelection, that says a host accepts them. A baseline the caller cannot use is refused with one uniform 400 validation_error, so the refusal never tells a caller whether another tenant's run exists. Nothing changes for a client that sends neither property.

Motivation

RFC 0081 §D defines regression mode as "re-run … and diff scores vs a baselineRunId", and the suite schema already says "a run MUST request only modes the suite declares AND the host advertises". Neither rule has a request carrier. The baseline appears only in what the host emits: eval.started.baselineRunId and EvalSummary.regression.baselineRunId.

Both production hosts advertise agents.evalSuite.modes containing regression and regressionVerdict: true (MyndHyve's certified 2.46.13 cut; openwop-app's 2.46.23 cut). openwop-app fills the gap with a host-local extension: it reads configurable.evalModes and configurable.baselineRunId, validates the baseline (same tenant, an eval run, the same suiteId, completed with a summary) and computes the paired verdict. Its eval branch returns before the RFC 0171 §D.1 closed-configurable check, so it accepts keys the v2 wire does not sanction, and no other host or client can rely on them. A portable carrier turns RFC 0244 §F from "not probed" into witnessable-gated, and lets openwop-app drop the extension.

Proposal

§A — The facet member

spec/v2/facets/agents.schema.json evalSuite gains an optional boolean:

"baselineSelection": true

true means the host accepts evalModes and baselineRunId on an eval createRun as this RFC defines. A client MUST NOT send either property to a host that does not advertise it; such a host MAY refuse the request with 400 validation_error (the v2 body is closed).

§B — The request

The createRun body gains two optional properties, meaningful only with mode: "eval":

• evalModes: a non-empty array of distinct RFC 0081 §D modes. It MUST be a subset of the suite's modes and of the host's advertised agents.evalSuite.modes. Absent ⇒ the host's default, as today.
• baselineRunId: a runId, the prior eval run this run is compared against.

A host advertising baselineSelection MUST refuse with 400 validation_error, before the run starts:

• either property without mode: "eval";
• evalModes outside the suite's or the host's modes;
• baselineRunId without regression in evalModes, or regression in evalModes without baselineRunId;
• a baseline it cannot use, per §C.

§C — The baseline

A usable baseline is an eval run that the caller's owner triple can read, that ran a suite with the same suiteId as this run's suite, and that completed with a served EvalSummary. Every other value is refused with the same status and the same error code, whether the run does not exist, belongs to another tenant, is not an eval run, ran another suite or has not completed. The refusal MUST NOT echo the supplied id. A caller learns no more about a foreign run than about an id that never existed: accepting a foreign baseline would expose its aggregateScore through scoreDelta, which the evalSuite rule "a suite or summary MUST NOT be readable across tenants" forbids.

The baseline may run another version of the suite, so the two runs may hold different tasks. That is what RFC 0244 §F pairing by taskId is for.

§D — What the run reports

For an accepted baseline, the host MUST put the same id on eval.started.baselineRunId and on EvalSummary.regression.baselineRunId, and regression.scoreDelta MUST be this run's aggregateScore minus the baseline's. With regressionVerdict advertised, RFC 0244 §F applies unchanged. The rules live in the baselineSelection and createRun property descriptions; no core prose changes.

Examples

Compatibility

Additive (COMPATIBILITY.md §2.4): two optional properties on the closed createRun body and one optional boolean on a closed facet. A request without them is handled exactly as today. A host that does not advertise baselineSelection is unaffected, and a client that reads the facet never sends the properties to it. No event, summary field, error code or MUST changes. openwop-app's configurable keys are a host-local extension this RFC replaces; retiring them is the host's own change.

Conformance

New scenario v2-eval-regression-baseline, gated on agents.evalSuite.baselineSelection, regression in modes, and both fixtures in discovery fixtures. The fixtures conformance-eval-regression-base (version 1.0.0, tasks base-only, shared-a, shared-b) and conformance-eval-regression-candidate (version 1.1.0, tasks shared-a, shared-b, candidate-only) share one suiteId. base-only fails, the rest pass, and the two shared tasks are identical, so the aggregates differ by 1/3 while every paired score is equal. A host that compares aggregates instead of pairing tasks would call the candidate improved.

Falsifiability — one row per normative requirement

Alternatives considered

1. Do nothing. §F stays unwitnessable and openwop-app keeps an unsanctioned configurable extension.
2. An RFC 0171 configurable section for eval. Rejected: configurable is workflow input, persisted and carried by a fork; the baseline is a run-selection parameter, like evalSuiteRef, so it sits beside it at the root.
3. The host picks the latest prior run of the suite. Rejected: neither host does it, it races concurrent runs, and the result would not be reproducible.
4. Distinct error codes per refusal reason. Rejected: a reason that separates "another tenant's run" from "no such run" is an oracle. One code for every case.
5. Fold baselineSelection into regressionVerdict. Rejected: MyndHyve advertises regressionVerdict without the request properties; reusing the flag would make its advertisement false.

Unresolved questions

All were resolved on 2026-10-26 by /architect:

1. Placement: root properties beside evalSuiteRef, not configurable (alternative 2).
2. Signal: a new boolean baselineSelection, so no current advertisement becomes false (alternative 5).
3. Refusal: 400 validation_error, one answer for every unusable baseline, no details (the registry row has none).
4. Suite identity across versions: the baseline matches on suiteId; versions may differ (§C).

Implementation notes (non-normative)

openwop-app already validates and pairs as §C and §D require (createEvalRun.ts, buildSummary); it maps the two root properties onto that path, advertises baselineSelection, and lists the two fixtures. Its suite store must hold two documents with one suiteId and different versions, resolved by fixture id. MyndHyve is unaffected until it opts in. The SDK follow-up is G3.

Acceptance criteria

• [x] The two createRun properties (derived api/v2/openapi.yaml) and the facet member (derived capabilities schema), with the rules in their descriptions.
• [x] v2-eval-regression-baseline, its two fixtures in fixtures.md, judges with self-tests.
• [x] A host advertises baselineSelection, and a certified major-2 bundle records the 0251. host legs and the two 0244.regression- legs executed-pass. (2026-10-26: openwop-app's cut 230c, build 24b6a05ff; evidence/v2-host-bundles/openwop-workflow-engine-2.46.28-24b6a05ff.json.)
• [x] The two properties in the v2 SDKs (G3): openwop-sdks #62, 2026-10-26, implemented instead of filed as an issue.

References

• RFC 0081 §D, RFC 0244 §F, RFC 0171 §D.1.
• Runs, Compatibility.
• Gap register, risk register.