# RFC 0158: Durable Execution and Disaster-Recovery Qualification > Source: https://openwop.dev/rfcs/0158-durable-execution-and-disaster-recovery-qualification.html · OpenWOP, the open-source protocol for multi-agent workflow orchestration. CC BY 4.0. | Field | Value | | ----------------- | ------------------------------------------------------------------------ | | **RFC** | 0158 | | **Title** | Durable Execution and Disaster-Recovery Qualification | | **Status** | `Accepted` | | **Author(s)** | openwop-app-f4 (host maintainer, reference host) | | **Created** | 2026-08-18 | | **Updated** | 2026-08-20 — §Conformance witness discipline (revised on reference-host review): recovery rows worded on the observable property not the trigger, `kill-after-accept` is a hold-dispatch row, seam gated on an unnamed deployment-time flag (fail-closed) rather than a second env name, declared operator preconditions with `blocked`-not-`inapplicable` disposition, `peer-resume` bundle-witnessed via an opaque per-boot token (no discovery field, §E.10), and acceptance scoped per-claimed-rung so the `durable-single-instance` witness graduates the RFC. · **`Draft → Active`** 2026-08-20 (window-waived, additive per §Compatibility): witness discipline settled and reviewed by the openwop-app reference host, which is witnessing the `durable-single-instance` rung (`kill-during-execution` observed non-vacuously across a real `SIGKILL`); `Accepted` gates on the non-vacuous single-instance bundle. · §Conformance note added: the recovery interval is measured kill → **resumption** (first re-execution observation, e.g. a second `run.started`), never kill → terminal — from a second tier-1 measured failure where time-to-terminal read as a false §B.5 violation. · 2026-09-20 — §"Witnessing the recovery rows" note: the kill rows observe until the host's declared recovery bound elapses (they read once, at the instant of return, and so mandated a fast bound); `kill-during-execution` asserts both clauses of item 11; the registry's recovery events count as resumption; `duplicate-delivery`'s seam chooses the effectful work **and the effect is counted at a receiver the suite owns**, because a ledger keyed on effect identity cannot show a double-fire. Scenario corrected in suite 2.32.0. No normative requirement changed. · 2026-09-21 — **the `durable-single-instance` rung is witnessed and the status does NOT move.** Tier-1: the v2 reference host, suite 2.33.1, guard closed, nothing relaxed, five rows `executed-pass` under a real `SIGKILL` (criterion 2 ticked). The flip was prepared and then stopped on reading criterion 1: the evidence-bundle fields §E rests on do not exist, and two `Affects` documents carry no text. `Accepted` waits on those, then a re-cut that carries them. · 2026-09-21 (later) — both landed, corpus 2.34.0: the bundle fields (§E note) and a v2 normative home; `Affects` amended to drop `replay.md` and `capabilities.md`, which this RFC never gave anything to carry. Status still `Active` — no host bundle carries the fields until a host is re-cut on the published suite. No normative requirement changed. · 2026-09-21 — **`Active → Accepted`. Tier-1, steward reference host (single-witness).** `openwop-host-v2-reference` re-cut on the published suite 2.34.0 (build `commit:2e6e42d`, guard closed, nothing relaxed, 242 / 0 / 0): the bundle itself now states `durability.rung: durable-single-instance`, and the verifier re-derives it from the signed rows — class `single-instance-restart`, bound 6 000 ms = 1 000 + 5 000, resumption observed at 1 787 ms and 1 883 ms after a real `SIGKILL`. **Not witnessed, by anyone:** peer resume, `durable-multi-instance`, `multi-region-qualified`, and the lowest rung on any production host. `bound-is-derived` is a paper check. | | **Affects** | `spec/v1/idempotency.md`, `spec/v1/storage-adapters.md`, `spec/v2/core/persistence.md`, `spec/v2/core/conformance.md`, `schemas/v2/certification-bundle.schema.json`, conformance `durability/*`. *Amended 2026-09-21: `spec/v1/replay.md` and `capabilities.md` were named at drafting and this RFC never gave either anything to carry — no clause of §A–§D touches replay or fork, and §E.10 mints no capability, so there is nothing to advertise. Dropped rather than filled with text written to satisfy a header. The v2 homes and the bundle schema are what it actually changed.* | | **Compatibility** | `additive` | | **Supersedes** | — | | **Superseded by** | — | > **Renumbered `0162` → `0158` on 2026-08-19.** This RFC was minted at 0162 against a planned block > 0158–0161 that was never authored and, on the dossier's own analysis, will not be: 0159 is folded into > RFC 0150 §D, 0160 restates RFC 0154 §E/UQ3, and 0161 is v2 work. Leaving a four-number hole to reserve > RFCs that were argued *against* would have made the gap permanent and unexplained. External references > to "RFC 0162" — reference-host ADRs, branch names, and an open host PR — predate the change and mean > this document. Nothing about its content moved; it was `Draft` and unreleased when renumbered, so no > tagged corpus release ever carried the old number. ## Summary Defines what it means for a host to *durably execute* work it has accepted, and a **qualification ladder** — `durable-single-instance` → `durable-multi-instance` → `multi-region-qualified` — that says which durability claims a host has actually demonstrated rather than which it believes. The central normative move is small and cheap: **a host that accepts work MUST declare the bound within which another instance may resume that work after the accepting instance dies, and MUST NOT leave the bound unstated.** A slow recovery is a legitimate posture. An unknown one is not. ## Motivation Three gaps, each observed on a real host rather than reasoned about. **1. "Recovery" is asserted, never bounded.** RFC 0151's acceptance criterion asked for *"one reference host demonstrates non-vacuous recovery from a mid-unwind crash"*, which reads as a fast test and thereby implies an architecture — a compensation runner with its own short lease — that the profile never required. It was amended (#1085) to *"recovery within a host-declared bound, and the bound is advertised."* **That amended criterion belongs here**, applied to all accepted work rather than to compensation alone. **2. A lease is not a liveness signal, and hosts conflate the two.** On the reference host, the run dispatch lease is set exactly once at dispatch and never renewed: ``` grep -rn 'setRunDispatchLease(' src/ -> ONE call site ``` So it answers *"could this run still legitimately be running?"*, not *"is the worker alive?"*. It must therefore be at least as long as the longest legal run, and crash detection takes as long as the maximum run duration — twelve minutes on that host. This is not a bug in that host; it is the predictable result of a spec that requires durability without distinguishing **liveness** from **duration**. Temporal separates them for exactly this reason: a long execution timeout *"would increase the delay before a stuck or crashed worker would be identified."* **3. Shortening the lease is unsafe in a way the spec does not warn about.** Without a liveness signal, a shorter lease declares *live* long-running work dead and re-dispatches it. A host with no effect fencing (`idempotency.crossRegion: single-region`) then converts a stall into **duplicate external effects** — and where the work is a compensation inverse, into duplicate refunds. **On such a host the long lease is standing in for fencing.** A spec that asks for fast recovery without saying this invites the dangerous fix. ## Proposal ### §A Durable acceptance 1. A host that returns a success status for work it has accepted **MUST** have made the intent to perform that work durable in the same transaction as the work record. A wakeup, hint, or in-process dispatch **MUST NOT** be the only record that the work was accepted. 2. A host **MUST** be able to resume accepted-but-unstarted work after the accepting process dies, without client action. ### §B Liveness and duration are separate bounds 3. A host **MUST** distinguish, in its own implementation, the maximum duration a unit of work may legitimately run from the interval within which a live worker demonstrates liveness. It **MUST NOT** use the duration bound as the sole liveness signal. 4. A host **MUST** declare a **recovery bound**: the maximum interval between an instance ceasing to make progress and another instance becoming eligible to resume its work. 5. The recovery bound **MUST** be derived from the mechanism that enforces it, not stated independently of it. A declared bound that no mechanism produces is a claim, not a bound. 6. A host **MAY** declare a recovery bound of any length. **A long bound is conformant; an undeclared or unenforced one is not.** > **Non-normative, and the reason §B exists.** Requirement 3 is satisfiable by a heartbeat, by a supervisor, or > by any other liveness mechanism. It is *not* satisfiable by lengthening a lease. A host that shortens its > lease to satisfy requirement 4 without satisfying requirement 3 has made itself less safe, not more — see > Motivation 3. ### §C Duplicate delivery and poison work 7. Duplicate delivery of the same accepted work **MUST NOT** produce duplicate external effects. Hosts **MUST** dedupe on an identity that survives redelivery, per RFC 0150 effect identity. 8. Work that fails deterministically **MUST** reach a terminal, operator-visible state within a bounded number of attempts, and **MUST NOT** be redelivered indefinitely. ### §D The qualification ladder > **These rungs are a THIRD axis, orthogonal to two vocabularies they will be read beside.** Naming them here > because the next reader will otherwise ask whether this re-mints something the corpus already has — the > question that folded proposed RFC 0159 into RFC 0150 §D. > > - **Scale profiles** (`minimal` / `production` / `high-throughput`) describe *how much load* a host is built > for. `production-profile.md` §"Durability" is the very MUST this RFC qualifies, so that document is the most > likely to be co-read — and it uses "production" for capacity, not for durability evidence. **A `minimal` > host may be `durable-multi-instance`; a `high-throughput` host may be unqualified.** > - **`capabilities.idempotency.crossRegion`** (`single-region` / `reconciled-records` / `fenced-effects`) > describes *what happens to effects across a partition*. `multi-region-qualified` sits one word from > `single-region` and means something else: the crossRegion enum is a **safety posture**, this rung is > **demonstrated recovery**. A host may be `fenced-effects` and hold no rung, having never run the exercises. > > Scale is capacity, `crossRegion` is effect safety, a rung is evidence. A host's position on each says nothing > about the other two. A host **MAY** claim a rung only with the evidence named for it. Rungs are cumulative. | rung | claim | evidence required | |---|---|---| | `durable-single-instance` | accepted work survives process death and resumes | kill-after-accept-before-dispatch, and kill-during-execution, both resuming within the declared recovery bound | | `durable-multi-instance` | a *peer* instance resumes it, not only a restart of the same one | the above, executed with the resuming instance being a different process from the accepting one | | `multi-region-qualified` | the above across a region boundary, with a stated RPO/RTO | the above plus a restore-from-backup and a region-evacuation exercise | 9. A host **MUST NOT** claim a rung on the basis of tests **in which no process was actually terminated**. Claim semantics asserted without a process death are evidence about **the claim**, not about recovery. > Reworded from "unit tests over its storage layer alone", which named the *artifact* rather than the > property and was therefore satisfiable by unit-testing a lease helper instead. It also generalises: a host > whose durability comes from an external orchestrator has no storage adapter to unit-test, and the earlier > wording would have been trivially satisfied rather than meaningfully met. ### §E Advertisement — deferred by design 10. This RFC defines the ladder and its evidence. It **does not** mint a new advertised capability field. > **Why.** A qualification ladder is **evidence about behaviour that already exists**, so the useful place for > a rung and a recovery bound is the host's **conformance evidence bundle** (RFC 0148) — where claims already > live, and where a claim without evidence is already a defect. A discovery field would let a host assert a rung > with nothing behind it, which is the failure this RFC exists to prevent. > > A capability field remains available as a later one-field additive revision if bundle-only publication proves > insufficient in practice. It is deliberately the smaller, later change, and it would need the falsifiability > table `RFCS/0000-template.md` requires — which this surface needs more than most: **four of the six rows in > §Conformance require a process termination the black-box suite cannot cause.** > **The bundle fields *(added 2026-09-21, suite 2.34.0)*.** Until this date the paragraph above had nothing > behind it: `certification-bundle.schema.json` had no seat for a rung, a bound, or its terms. It now has: > > - the `bound-is-derived` row carries `evidence.recoveryBounds[]` — `{ class, bound, terms[{ name, ms }] }` **per > recovery class**, no aggregate (Unresolved Question 1); > - each kill row carries `evidence.recovery` — `{ class, boundMs, observedMs }`, the kill → resumption interval > the scenario always measured and, on a pass, never recorded; > - the bundle MAY claim `durability.rung`, and a verifier **re-derives** it from those rows and rejects a claim > it cannot derive (`rung-not-derivable`). > > Evidence rides on **rows** because the attestation covers the witness digest and the witness digest covers > rows; a root-level block would be editable after signing. It enters the digest only when present, so every > earlier bundle still verifies. `class` and term names are opaque host-chosen identifiers — "leased" and > "boot re-entry" are two hosts' mechanisms, and an enum would select for an architecture. Only > `durable-single-instance` is derivable; `peer-resume` evidence is not carried yet, so a higher claim is refused. > > **What class-binding does not catch.** It refuses an undeclared class, a bound label that disagrees with its > class, and a resumption outside the bound. It does **not** refuse a seam that kills *before* the execution > claim is held and labels the exercise with the slower class: the faster lane rescues the run well inside the > longer bound, and no arithmetic separates that from a fast recovery. A tier-1 host shipped exactly this > (11.6 s recorded against a 750 s leased class). The evidence makes it **visible** where before nothing was > recorded; killing only once the claim is held remains the host's obligation. ## Compatibility **Additive.** No existing field changes meaning, no required field is added, and no host becomes non-conformant by doing nothing. A host that declares no rung is simply unqualified, which is its status today. The one behavioural requirement that could bite an existing host is §B.4 — declaring a recovery bound. A host with no liveness mechanism can satisfy it immediately by declaring the bound it *already has* (its duration ceiling), which is honest and is the point: it makes a twelve-minute recovery visible rather than unknown. ## Conformance | scenario | asserts | |---|---| | `durability/kill-after-accept` | work accepted, process killed before dispatch, resumed within the declared bound | | `durability/kill-during-execution` | killed mid-execution, resumed within the declared bound, **no duplicate external effect** | | `durability/peer-resume` | the resuming instance is a *different process* — the `durable-multi-instance` discriminator | | `durability/duplicate-delivery` | the same accepted work delivered twice fires each effect exactly once, asserted **per effect identity**, not by end state | | `durability/poison-exhaustion` | deterministic failure reaches a terminal operator-visible state within the attempt bound | | `durability/bound-is-derived` | the declared recovery bound matches the mechanism that enforces it — a host that states a bound it cannot produce **fails** | > **What `bound-is-derived` cannot catch** *(added 2026-08-19, from a measured failure on a tier-1 host)*. This > row checks that the declared number follows from the mechanism. It cannot check that **the mechanism runs.** A > host whose sweeper wedges — one lane's await never settling, leaving a shared re-entry flag set, with nothing > logged because nothing threw — has a derivation that stays perfectly correct while the bound is not produced at > all. A run sat unclaimed for 16 minutes against a derived bound of 12.5, and every isolated check of the > mechanism passed: the storage query returned the run as claimable, the sweep function reclaimed it on a > reproduced state, and a fresh daemon reclaimed a seeded orphan in 24 s. > > **Only an exercise that actually kills a process and waits out the bound distinguishes the two cases.** That is > the strongest argument yet for §D.9's insistence on real process termination — and, for the conformance suite, > for the kill seam the recovery rows need: `bound-is-derived` is a paper check by construction, and no amount of > strengthening makes it a liveness check. `storage-adapters.md` §"Expiry is authority; the sweeper is the > exercise of it" carries the normative consequence. `duplicate-delivery` asserts **invocation counts per identity**, not final state: a legal end state is exactly what a double-fire produces, so an end-state assertion passes on the defect it exists to catch. ### Witnessing the recovery rows *(added 2026-08-20)* The three recovery rows — `kill-after-accept`, `kill-during-execution`, `peer-resume` — need a process termination the black-box suite cannot itself cause (§E), so a host under test exposes a **conformance fixture seam** the suite drives. The requirements below are written on the **observable property**, not on any particular trigger: hosts differ in whether a run is served synchronously or accepted asynchronously, so a requirement phrased on "the request that is killed" would describe one host's mechanism and force another to fake it. What the suite observes is what is normative. 11. **A real process termination MUST have occurred, and no in-flight work is ever reported complete.** The seam performs a genuine `exit` / signal, not a simulated one. **Work that was accepted or executing when the process died MUST NOT be observable as completed or successful**, and resumption MUST be observed on a **subsequent** observation within the declared recovery bound. How the kill is triggered — a killed request, an out-of-band signal to an async worker, a supervisor — is the host's choice; asserting recovery without a real death is the "claim semantics asserted without a process death" §D.9 rejects. - `kill-during-execution` witnesses a **termination** while work executes. - `kill-after-accept` is a **hold-dispatch** row, not a termination-timing row: on hosts where acceptance and dispatch are microseconds apart, the seam **MUST hold dispatch**, take the kill during the hold, and show the accepted-but-undispatched work dispatching on resume. A seam that races a kill into that window cannot reliably hit it and must not pretend to. > **The measured interval is kill → *resumption*, never kill → *terminal*** *(added 2026-08-20, from a > second measured failure on a tier-1 host)*. §B.4 bounds the interval until another instance becomes > eligible to **resume** the work — not until the work **finishes**. Timing `kill → terminal status` adds > the work's own execution time to the recovery interval, and a host whose runs do real work will then > report an observed figure that exceeds its own derived bound and reads as a §B.5 violation that did not > happen. The witness **MUST** record the interval to the **first observation that the work is being > re-executed** — a subsequent run-lifecycle re-start on the same run id (a second `run.started`), or any > equivalent progress-past-the-pre-kill-point signal — and **MUST NOT** substitute time-to-completion. A > resumption observation of this kind needs no seam and no new field; it is exactly the "resumption observed > on a **subsequent** observation" item 11 requires. It does **not** discriminate a *peer*: a second > `run.started` cannot tell "a different process resumed" from "the same process restarted", so it does not > substitute for `peer-resume`'s opaque per-boot token. > **The observation runs to the declared bound, and the seam chooses the effectful work** *(added > 2026-09-20, from a reference-host review of the scenario before any host had built the seam)*. Three > defects in `v2-durability-recovery`, all of the same kind — the scenario asserting something narrower or > wider than this section says: > > - **Both kill rows read the log once, the instant the service answered again.** That demands resumption > within ~0 ms of the listener returning — a fast bound, which §"Alternatives considered" rejects and §B.6 > makes non-normative. A host whose sweeper first ticks 5 s after boot, against a derived 65 s bound, read > zero `run.started` and failed, then resumed correctly ten seconds later. The witness **MUST** keep > observing until resumption is seen **or the host's own declared recovery bound elapses**. The bound is a > ceiling for waiting only. A host whose bound is per class (Unresolved Question 1) names the governing > figure on the seam's own response. The suite's observation ceiling is **operator-raisable**: a fixed one > is the same defect moved from 0 s to the ceiling — a conformant 12.5-minute leased-class bound would record > `blocked`, which denies certification, and the only way out would be shortening a lease, the outcome > §"Alternatives considered" rejects. A bound over the ceiling is `blocked` with that precondition named. > - **`kill-during-execution` asserted only this item's first clause.** "Not completed, or re-started" holds > forever for a run that is never resumed, so the row passed on a host that lost the work. Both clauses are > now asserted: never observable as completed un-re-executed (latched across every observation), **and** > resumed within the bound. > - **A resumption signal is a further `run.started` *or* the registry's own recovery event** > (`workflow.restored`, `run.restored-from-snapshot`) — the "equivalent progress-past-the-pre-kill-point > signal" this item already allows, now named so a host is not failed for using the event minted for it. > - **`duplicate-delivery` named `conformance-noop`,** which records no effect on a host whose noop is a true > no-op, so the row could only ever record `blocked` — and `blocked` denies certification. No canonical > fixture is guaranteed effectful, so the scenario sends no `workflowId` for this mode: the seam chooses > the work. > - **`duplicate-delivery` counted rows per identity on the host's own effect ledger, which cannot see a > double-fire.** A ledger keyed on effect identity admits at most one row per identity *by construction* — > a second fire at the same identity writes the same key. Measured on a tier-1 host before it built to the > row: with its dedup claim forced to always win, and then with the effect emitted **twice per fire**, the > per-identity count stayed 1 and the row stayed green. §C asks for **invocation** counts; the projection > exposes **identities**. The only black-box oracle for "fired once" is a destination the suite owns, so > **the effect is counted where it lands**: the scenario passes `effectUrl` (its own receiver), the seam > **MUST** stage work that performs exactly one outbound effect addressed to it and deliver that work twice, > and the row asserts exactly one arrival. The ledger read is kept as a secondary check and labelled for > what it is — that the projection is consistent with the destination, **not** that no double-fire occurred. > Operator precondition, declared: the receiver must be reachable through the host's egress guard, exactly as > for the webhook rows. > - **The scenario read the run log from the wrong one of the corpus's own two operations.** At major 2 > `GET /runs/{runId}/events` is `streamRunEvents`, `text/event-stream` only; the JSON read is > `GET /runs/{runId}/events/poll`. A host serving exactly what the OpenAPI states answered SSE, the JSON > parse came back null, and `poison-exhaustion` — having already asserted the terminal status — > soft-skipped its load-bearing clause and resolved **`executed-pass`** with the detail > `partial-witness: blocked`. Found on the v2 reference host's first run of these rows: a vacuous pass that > had been sitting in that host's bundle. An unreadable log is now `blocked` on the kill rows too, never > "nothing resumed". > - **"During execution" means after the host holds its execution claim on the run.** A seam that kills at > `run.started` on a host that appends `run.started` *before* taking its dispatch lease kills work that is > still unclaimed — and then the easy recovery class (accepted, never claimed) is witnessed under the hard > one's name. Measured on a tier-1 host: the run was rescued by its outbox lane in 11.6 s while the seam > reported the 750 s leased class. The black-box suite cannot see which class recovered the work, so the > kill point is the **host's** obligation: the seam **MUST** kill only once the execution claim is held, > and a host with more than one recovery class **MUST** pin that host-side. 12. **The seam MUST be gated on a deployment-time flag that is unset in production, and MUST be fail-closed.** This RFC names no specific environment variable — a host that already gates a test seam (e.g. a boot-read flag) rides that gate; minting a second flag for one boundary is itself a hazard, since one gets set in a context the other does not and the fail-closed property silently stops holding (the shared-flag failure RFC 0144 and `SECURITY/threat-model-secret-leakage.md` name). The gate MUST be a deployment-time property (read at boot / per-revision), **not** a per-request toggle: a self-terminating endpoint reachable in production is a denial-of-service surface. The seam is a **non-normative host-extension route** (`host-extensions.md`): it advertises nothing, this RFC mints no capability field for it (§E.10), and a host that never runs the durability exercises exposes no such route. It is test infrastructure, not protocol surface — which is why a host may implement it before this RFC reaches `Accepted` without advertising a claim it cannot yet defend. **Operator preconditions are declared, not hidden.** These rows do not run against an unattended black-box host unchanged: `kill-after-accept` / `kill-during-execution` need a **restart supervisor** (a black-box suite cannot itself restart a killed single instance — something must, e.g. the harness as parent process), and `peer-resume` needs **two live instances at the moment the kill lands**. A row whose operator precondition is unmet is **`blocked`** with the precondition **named** (the disposition shape `production-profile.md` uses for `OPENWOP_WEBHOOK_ALLOW_PRIVATE`), never silently skipped or reported as a pass. **`peer-resume` disposition and witness.** `peer-resume` is the `durable-multi-instance` discriminator (§D). It is **not** `inapplicable` to a host with the mechanism — it is **`blocked` on the ≥2-live-instances precondition**; a host that never claims `durable-multi-instance` simply does not run it. Witnessing that a *different process* resumed the work needs a per-boot observable, and there is none on the wire today. Per §E.10's deferral of new discovery fields to a later, smaller revision, **this RFC does NOT add a discovery field for it**: the resuming instance's opaque per-boot incarnation token (opaque, never a raw pid — a pid leaks infrastructure) is recorded in the host's **RFC 0148 evidence bundle** alongside `recoveryBoundTerms()`, together with the continuous-reachability trace that shows the service never went down across the kill. So `peer-resume` is **bundle-witnessed**, and the table reads it as such; a discovery `processIncarnation` field, with its own falsifiability table, remains the deferred later revision §E.10 anticipates should black-box witnessing ever be wanted. **`bound-is-derived` evidence.** The derivation — the per-class arithmetic, not a single total (Unresolved Question 1) — is emitted into the host's RFC 0148 evidence bundle, where a reader can recompute it. It is **not** advertised as a discovery field (§E.10). Per the note above, this row is a paper check and **MUST NOT** be cited as evidence that the recovery mechanism runs; only the kill rows witness that. ## Alternatives considered - **Require a fast recovery bound (e.g. ≤ 60s).** Rejected: it selects for an architecture rather than a property, and a host that cannot meet it is pushed toward shortening a lease without a liveness signal — the exact unsafe fix in Motivation 3. - **Mint `capabilities.durability.rung` now.** Rejected in favour of bundle-first publication; see §E. - **Fold this into RFC 0150.** Rejected: 0150 is about effect identity and split-brain *safety*; this is about *qualification* — what a host has demonstrated. Merging them would let a host inherit a durability claim from an idempotency implementation. ## Unresolved questions 1. ~~Should the recovery bound be a single scalar, or per work class?~~ **Resolved 2026-08-18: PER WORK CLASS, and §B.5 already derives it.** "The recovery bound MUST be derived from the mechanism that enforces it" — and the corpus has at least two independent enforcing mechanisms (the run/dispatch claim lease, `storage-adapters.md` §"Claim acquisition"; the Layer-1 idempotency record's own lease, `idempotency.md`). Several mechanisms, therefore several bounds. **A single scalar would have to be the maximum, which overstates recovery for every faster class — a lie by aggregation, which is the failure this RFC exists to stop.** A host with one mechanism declares one bound; that is the degenerate case, not the general one. 2. ~~Does `multi-region-qualified` require a *tested* evacuation, or a *rehearsed* one?~~ **Resolved 2026-08-19: TESTED — traffic MUST actually be served from the surviving region, with RPO/RTO MEASURED rather than declared.** §D.9 already derives this: it refuses a rung claimed on "tests in which no process was actually terminated", so a region that was never left cannot qualify the rung above. A rehearsal that moves no traffic is a runbook review, and accepting it would make the ladder inconsistent with its own lowest rung. **PLANNED is fine; SIMULATED is not.** A scheduled failover drill in which the secondary genuinely serves traffic satisfies this; an unplanned outage is not required. The bar is *did the traffic move*, not *did the host suffer* — which is also how disaster recovery is qualified in practice. 3. ~~Should §C.8's attempt bound be normative, or host-declared?~~ **Resolved 2026-08-19: HOST-DECLARED, with two normative constraints — it MUST be finite, and it MUST be derived from the mechanism that enforces it.** §B.6 already settled this pattern for the recovery bound ("a host MAY declare a bound of any length; a long bound is conformant, an undeclared or unenforced one is not"), and a fixed number would select for an architecture: three attempts is right for expensive external effects, fifty for cheap idempotent work. Selecting for an architecture is precisely what this RFC declined to do when it refused to mandate a fast recovery bound. **The two constraints exist because this bound is NOT symmetric with the recovery bound**, and the asymmetry is the whole reason to write them down. A slow recovery bound is harmless — it delays resumption and says so. An attempt bound declared as effectively infinite would satisfy a bare declaration requirement while reproducing the unbounded-redelivery failure §C.8 exists to stop. Finiteness is the property that matters; the §B.5 derivation rule stops a host declaring a number nothing enforces. ## Implementation notes (non-normative) The reference host satisfies §A today and does not satisfy §B.3: its lease is its only liveness proxy. Its planned remedy is a heartbeat that renews the lease while work is active, keeping the advertised duration ceiling unchanged (host ADR 0585). That ordering matters — the heartbeat first, the shorter reclaim threshold second — and this RFC's §B is written so the reverse ordering is visibly non-conformant. ## Acceptance criteria - [x] Spec text, conformance scenarios, and the evidence-bundle fields land. **Met 2026-09-21, corpus 2.34.0 — and ticked only once a host bundle CARRIED the fields.** *Conformance scenarios:* five `durable-single-instance` rows, suite 2.27.0–2.33.1. *Spec text:* `spec/v1/idempotency.md`, `spec/v1/storage-adapters.md` §"Expiry is authority…", and — the home that outlives v1 — `spec/v2/core/persistence.md` §"Durable acceptance and recovery"; `Affects` amended to drop two documents this RFC never gave anything to carry. *Evidence-bundle fields:* row `evidence.recoveryBounds[]` / `evidence.recovery` and the re-derived `durability.rung` claim (§E note; `spec/v2/core/conformance.md` §"Recovery evidence"). **Carried by:** `evidence/v2-host-bundles/openwop-host-v2-reference.json` — suite 2.34.0 from npm, verified in a clean directory, rung re-derived, a tampered interval rejected (`witness-digest`) and a higher rung refused (`rung-not-derivable`). *Owed to the next release, not to this criterion:* `persistence.md`'s Stable banner does not yet list this RFC — a Stable banner may cite only `Accepted` RFCs, so it could not before today, and the file ships inside the already-published `spec-artifacts@2.34.0`, so it is not edited under that version. - [x] At least one host executes, in strict mode, **every scenario applicable to the rung(s) it claims** (RFC 0147 requirement 5 — shape-only evidence does not suffice). Witnessing the **`durable-single-instance`** rung — `kill-after-accept`, `kill-during-execution`, `duplicate-delivery`, `poison-exhaustion`, and `bound-is-derived` — is sufficient to accept this RFC. The higher rungs are **defined-but-unclaimed** until a host at that rung witnesses them; because §E.10 mints no capability field, an unwitnessed rung advertises nothing and so cannot be a vacuous claim. **Met 2026-09-21 — tier-1, steward-verified.** `openwop-host-v2-reference` build `commit:a084218`, suite 2.33.1, strict, **egress guard closed and `host.relaxations` empty**: all five rows `executed-pass` with empty detail, under a real `SIGKILL` — the restart supervisor itself counted 2 deaths for the 2 kill rows, and `duplicate-delivery` is one arrival at a receiver the suite owns. Declared bound 6 000 ms = 1 000 (supervisor restart, the OPERATOR's term) + 5 000 (boot re-entry budget). A single-process SQLite reference example: no production host has witnessed the rung. **Second witness, 2026-09-21 — tier-2, production (not an acceptance input; recorded as evidence).** MyndHyve `workflow-runtime`, suite 2.34.1, build `commit:0259dc9e` (verified against its live `/readiness`), nothing relaxed: five rows `executed-pass`, bundle states the rung and the steward's verifier re-derives it. One class, `dispatch-lease`, 100 000 ms; observed 71 157 ms (`kill-during-execution`) and 16 376 ms (`kill-after-accept`). Re-cut the same day (build `commit:44c8d4e0`) with the after-accept path in its own class, `dispatch-hold` (45 000 ms): observed 15 371 ms against it, and 70 460 ms for `kill-during-execution` against `dispatch-lease`. **Re-witnessed the same day on suite 2.34.0** (build `commit:2e6e42d`, same posture), this time with the evidence in the bundle: resumption at 1 787 ms (`kill-after-accept`) and 1 883 ms (`kill-during-execution`) against the 6 000 ms bound of class `single-instance-restart`. - [ ] deferred: **Gating the `durable-multi-instance` rung (not this RFC's acceptance):** no host claims that rung until `peer-resume` is witnessed by a host running **two processes**, not two workers in one. A host that cannot guarantee a second live instance at kill time marks the row `inapplicable` and holds only the lower rung. - [x] `docs/KNOWN-LIMITS.md` and `INTEROP-MATRIX.md` updated with each host's declared rung and bound. *(2026-09-21.)*