# Task26 Recovery Runbook, candidate 2e0894ea (v6)

Version: v6 (2026-08-15, supersedes v5, v4, v3, v2, and v1 recovery runbooks). v6 makes exactly one runtime-baseline correction to G12, replacing the v1 invite harness with the sealed invite harness v2 and preserving every v5 gate, order, and command otherwise.
Status: READY_FOR_ONE_INVITE_EXECUTION. All v2 open blockers (B3, G12, B6) are closed with sealed evidence. This document supports the v6 Golden Path `task26-golden-path-2e0894ea-v6.md` and binds the same candidate `2e0894eac92bc396cc4723bf1f18ebc653b95018dd41574df435941c235da925`, wheel `af4a9d0a1ffffb6eb7551c1d6dc2b32853ca6d024332a4f8f5702bbf992f141b`, plan `7ace03c6dad33d2fc3ef223621cbca68a150fde8429932138e252fb8498ac582`, actor `8527916639`.

Supersession (append-only; all prior documents remain byte-identical):

- Supersedes `task26-recovery-runbook-2e0894ea-v5.md` sha256 `e2fbff811e5b0caf4551010c1a47d640bba17febffb951f0525b6ff4dccd71d3` and `task26-golden-path-2e0894ea-v5.md` sha256 `3c6a0f24124142fc1b03c350a357d5530aa1a561c4b332e2e6c29a0de74c87e2` plus receipt `runbook-binding-receipt-v5.json` sha256 `2fb0019a8d9e46cc197f72685a89fc54ca5fb98a52840f41ade5d51f76d86736` (v5; only the G13 observer contract changed there).
- Supersedes `task26-recovery-runbook-2e0894ea-v4.md` sha256 `f539460f91096cfe32140dd6d5b5a940ff38a13ad67bf6d83302f6732aa719a1` and `task26-golden-path-2e0894ea-v4.md` sha256 `e0979931fb863d957403f71305d33470697813c5e740622b6b05a23a65c76359` plus receipt `runbook-binding-receipt-v4.json` sha256 `2a00bb6d8103086e729ac5c36041d0e7d602a1c660c21eda20463ea8de1b78e7` (v4; only G13 is corrected here).
- Supersedes `task26-recovery-runbook-2e0894ea-v3.md` sha256 `e62691493d2a75c312dd248a07bbc52f7f1d98d4acd701b824bbaf5edf8bdd3a` and `task26-golden-path-2e0894ea-v3.md` sha256 `ff230719eb5a69ae821bb0a4d58356d59b2803c490d692659a197e22083793c7` plus receipt `runbook-binding-receipt-v3.json` sha256 `fd77a4d387b9bab7978c84cb51b1ab3200d52a126dea411db98492f6be67fb88`.
- Supersedes `task26-recovery-runbook-2e0894ea-v2.md` sha256 `6024c26ccae3cad7fcf46a77bca8607d73293a92ddb3dcf8fa5a278361869f1d` and `task26-golden-path-2e0894ea-v2.md` sha256 `633d5c25ce70cb9ab5d19c434122b14b46fb7108962ab6e453ce6ab3ede1848f`.
- Supersedes `task26-recovery-runbook-2e0894ea.md` sha256 `f4f2d2347b4b68a080b9c437a813accbef8c4bcc471cbf21d67ffff31e56f89b` and `task26-golden-path-2e0894ea.md` sha256 `1f55b8f967c6564113574d41bf1c15ce977d35c38b5687f1623ba37f235a7bdb` (v1; must remain byte-identical for the sealed invite harness).
- Still supersedes `.omo/evidence/dualcoach-recovery-runbook.md` sha256 `ae2f5f9046c06f8f0b42693be5aa0d9c31cada8024ae1e5ab8ba19a3cf10f6fc` and `.omo/evidence/dualcoach-golden-path-contract.md` sha256 `32a379d855c6e5af978bd9886e3bf49c616c7c20f1c1d5f100c19d8adfc5eb4a` (stale candidate `19ed0e6047f0a4d7650c47a7413ee0298a243f768a63bcb8fb9cffe44d140a1a`; void).

Hard safety rules: stop all mutating action before any diagnosis; no restore, prepopulation, repair-in-place, profile-state hand-edits, raw Telegram listeners, cursor edits, service replay, forced publication, candidate rebuild, manual `gateway.lock` deletion, or recovery shortcut, ever. All evidence in this run's evidence directory, mode 600 (directory 700). No change to `physique-coach` or `quarantine` at any point.

## 1. Supported command set (the only supported CLIs; run from the repository root unless stated; PY = `/home/cube/projects/richard/hermes-agent/.venv/bin/python`)

| Interface | Supported command shape | Notes |
|---|---|---|
| Candidate verifier | `python3 <successor dir>/verify_candidate.py <successor dir>/verifier-input.json` | Successor dir = `.omo/evidence/task26/task26-repaired-archive-successor-2e0894eac92bc396cc4723bf1f18ebc653b95018dd41574df435941c235da925`; exit 0 = PASS; exit 1/2 = FAIL; no flags |
| Pre-reset controller | `python3 '.omo/evidence/task26/reset-controller-st_01a0054d/reset_controller.py' {dry-run,execute,verify} --profile ... --other-profile ... --archive-root ... --contract ... --permission ... --candidate ... --wheel-sha256 ... --plan-sha256 ... --approval ... --run-id ...` | Exact execute command in Golden Path Section 3; verify is post-execute only; run ids one-use |
| Cleanup controller | `python3 '.omo/evidence/task26/reset-controller-st_01a0054d/post-lifecycle-cleanup-v2/cleanup_controller.py' {dry-run,execute,verify} ... --actor-id '8527916639' --run-id ...` | Same flags as pre-reset plus `--actor-id`; verify takes `--archive`; optional `--systemctl`/`--service` defaults `/usr/bin/systemctl` / `hermes-gateway-dualcoachtest.service`; exact commands in Golden Path Section 14 |
| Invite harness v2 | `PY -B .omo/evidence/task26/task26-invite-harness-st_01a0056a-v2/invite_harness_v2.py {dry-run,verify,prepare,observe,expire} --profile <profile> --permission .omo/evidence/task26/task26-invite-harness-st_01a0056a-v2/permission-seal-v2.json --receipt <path>` | Exact commands from `OPERATIONS-v2.md`; canonical absent-ledger baseline; stable `data` parent watch before prepare; one candidate creation and one generation `1` PREPARED session; no manual empty-ledger initialization; v2 root only |
| Archive verifier | `python3 .omo/evidence/task26/verify_retained_archives.py .omo/evidence/task26/retained-archive-schema-inventory.json --permission-seal .omo/evidence/task26/retained-archive-verifier-permission-seal.json [--receipt <out>] (actual seal file, sha256 28ff72fd...; strict argument order per the sealed verifier: positional inventory, then --permission-seal, then optional --receipt; exit 0 = PASS, exit 1 = FAIL)` | Verifier sha256 `12b97aa7...`; inventory `29db87f2...`; seal `28ff72fd...`; sealed receipt `597a37e4...` (PASS, five archives, four schemas) |
| Provider auth | `HERMES_HOME=/home/cube/.hermes/profiles/dualcoachtest /home/cube/projects/richard/hermes-agent/.venv/bin/dualcoach_admin provider-auth check [--probe chat.completions.create] [--allow-billable-active-probe]` | The `HERMES_HOME` profile binding is mandatory; see R10 for the READY receipt evidence |
| Customer registry | `PY -m checkin_cli.customer_admin --registry /home/cube/.hermes/profiles/dualcoachtest/customers/registry.json <subcommand>` | Real subcommands only: `add`, `disable`, `activate`, `record-payment`, `record-satisfaction`, `record-operator-time`, `judge-kpi`, `list`, `validate`, `consent`, `revoke-consent`. `activate` takes `--profile-root --data-root --checklist-evidence <customer_id>`; `disable` takes `<customer_key>` only (no reason flag) |
| Readiness audit | `PY -m checkin_cli.readiness_cli --profile-root /home/cube/.hermes/profiles --customer-key <key>` | Read-only; exit 0 with all items satisfied |
| Nutrition onboarding CLI | `PY -m checkin_cli.nutrition_onboarding_cli {status,purge-incomplete,operator-review,seed-kb,migration-preflight,migration-commit}` | Diagnostic use only in this rehearsal |
| Lifecycle observer G13 | `python3 -B .omo/evidence/task26/task26-lifecycle-observer-st_01a005b4/lifecycle_observer.py {subscribe-outbox,audit-tail,watch-deliveries,arm-only,multi-observer} --profile ... --manifest ... --after-seq ... --timeout ... --output ...` | Sealed root only; exact commands in Golden Path Section 5.2 and `EXACT_COMMANDS.md`; event-driven bounded inotify, multi-observer READY barrier before any trigger, 0600 receipts, re-arm from `transition.revision_after`; no sleep, poll, raw Telegram, cursor, drop, or replay |
| Service control | `systemctl --user {start,stop,restart,status} hermes-gateway-dualcoachtest.service`; `journalctl --user -u hermes-gateway-dualcoachtest.service` | User-scope unit; unit file `/home/cube/.config/systemd/user/hermes-gateway-dualcoachtest.service`; `Restart=always`, `RestartForceExitStatus=75` |
| Gateway admin | `PY -m hermes_cli.main --profile dualcoachtest gateway ...` | Runtime interpreter is the shared venv above (no profile-local venv exists) |

Void interfaces (do not invoke; some were wrongly cited in v1/v2): `python -m hermes_cli.main gateway run --port`, `curl http://127.0.0.1:8080/...` (no documented HTTP surface), any `/reload`, `/validate`, `/redeliver`, or replay control, any bootstrap/customer DB or ledger edit command, `hermes verify`, any task-scoped controller's internal reset modes beyond the sealed `dry-run/execute/verify`, `hermes_cli.nutrition_readiness` and `hermes_cli.nutrition_review_service` (neither module exists anywhere; there is no generation CLI at all — generation is enqueued atomically at check-in completion), `customer_admin create/show/reset/set-next-checkin/delete` (do not exist), and any manual `gateway.lock` deletion. The plan's "planned interface" commands (`process-state`, `generation reconcile`, `gate telegram-registration`, `notify-unbound`, rollback to a per-profile venv, candidate wheel rebuild) remain VOID (not implemented).

## 2. Global abort rules

Abort immediately to the relevant procedure on: any gate FAIL or UNKNOWN; a mutating command failing; a `RuntimeError`, `auth_key_duplicated`, flood, `ApiIdInvalid`, `EOFError`, `TimedOut`, privacy, or consistency journal error; a state transition outside the golden order; any pending `conflict_review` or unresolved conflict; any second invite preparation attempt, second Start claim, second DM or delivery, or implicit send; missing journal evidence for a claimed transition; any touched-file digest in the other profiles changing; any session/cron/process still running after disable; any reset/cleanup archive verification FAIL; any human-reported crash. On abort: capture the journal tail, subscription transcripts, and CLI output first; then diagnose; the only permitted mutation is the documented procedure step. Bounded deadlines (watch 120s, connect 90s, claim 120s, consent-to-review 300s, generation card 600s, send-to-DM 300s, CLI exit 60s): an exceeded bound is a STOP, not a retry.

## 3. Failure procedures

R1 (service failed to start): `status`, `is-enabled`, `is-active`; `journalctl --user -u hermes-gateway-dualcoachtest.service -n 200 --no-pager` (capture first). If the port is in use, identify the holder with `ss -ltnp`; do not kill by guess. Never loop restarts: after two failed starts, stop and capture. If config YAML fails to parse, restore only from the documented config file, never by hand-editing state.

R2 (stale `gateway.lock`): pre-reset, the lock's presence is allowed only as sealed reset input. Prove no active holder with `flock -n ... true`; then the sealed pre-reset (Golden Path Section 3) archives and disposes of it through `approved_clear_scopes`. The sealed invite harness v2 refuses to run while the lock exists, which enforces the ordering. Post-reset, the post-reset gate requires the lock absent. Any other deletion of the lock file is forbidden. No manual removal at any point.

R3 (invite expired or preparation rejected): preparation rejects with exit 2 when the lock is still present (ordering violation: run the pre-reset first) or when the sealed one-prepare maximum is already spent. If the invite expires unclaimed, mark it terminal through the sealed harness exactly once: `PY -B .omo/evidence/task26/task26-invite-harness-st_01a0056a-v2/invite_harness_v2.py expire --profile ... --permission ... --session-id <session-id> --sid-hash <sid-hash> --receipt <run-id>-invite-expiry.redacted.json`, then run one new sealed preparation (Golden Path Section 6). Never reuse an expired token, never re-arm the historical manifest, never edit `ledger.json` by hand. Evidence: preparation/expiry receipts.

R4 (claim failed or wrong actor claimed): read `data/customer_bootstrap/ledger.json`; the real session must be `expired` or `cancelled` with the conflicting `role_claims` recorded. No ledger hand-edit. Return to R3.

R5 (consent timeout): wait to the 120-second bound once; capture the audit transcript (`canonical_customer_consent_*`). If no consent arrives, leave the session `claimed`, stop, and return to R3. Do not press buttons for the user.

R6 (onboarding abandoned mid-flow): bounded wait once; do not re-DM the user. If abandoned, the session stays non-terminal; capture evidence and escalate. Do not synthesize a terminal state.

R7 (safety_hold): use the owner card's `safety_hold` action; never bypass via config or state edits. If the safety_hold cannot be cleared, escalate with the audit transcript.

R8 (owner card missing/stale): confirm the onboarding ledger shows `owner_review`/`safety_hold` and the audit shows `canonical_customer_onboarding_review_card`; a stale card is superseded by the newest card message, which is the only authority. Do not double-send cards from the CLI.

R9 (customer_admin CLI error): use only the real subcommands in Section 1. `activate` requires `--profile-root`, `--data-root`, `--checklist-evidence`, and the customer id; it is one-use and fails closed on a second attempt (treat a second attempt as an incident). `disable` takes only the customer key (no reason flag exists). If activation fails, capture stderr; the only retry is after proving the first attempt did not commit (registry `service_state` unchanged).

R10 (provider auth/check failures): the authorized active probe already passed with a sealed READY receipt (raw file sha256 `8278dde4efa8bcd366fce873ccc07f651818275238b67452b9e6cb93c67afd6d`, payload `receipt_sha256` `6aeaaab42e0030b0f626037d7b9ef6661ec93fb57fd991c843967c1fc695905c`, index `16acf1b355715379ad526433803353e1f34c418e9560fc1466f3f77f951df919`, one request attempt, `billable: true`, `store: false`, 26 total tokens, pre/post profile snapshot identical at `7d163cb0...`). Re-diagnosis is read-only: run `provider-auth check` (no probe) with the mandatory `HERMES_HOME=/home/cube/.hermes/profiles/dualcoachtest` binding. The receipt field `candidate_digest` `f9a46172386333a0067f43695fb1429a043a606f04baceb06e8c59b90c36235c` is the provider command digest emitted by module `gateway/platforms/dualcoach_admin.py` (`0e9b4b1f...`); it is a different domain from the candidate envelope digest `2e0894ea...` and is not candidate drift. If a new billable probe is ever needed, it requires its own human authorization first. If auth is missing or expired: stop; no token/secret mutation inside the rehearsal window.

R11 (subscription or audit gap): re-arm the subscription with `--after-seq` past the last observed `seq`. A transition with no durable event is unproven; treat as a defect, not as "probably fine".

R12 (check-in interrupted): the check-in state machine is durable; the customer may resume by DM. Do not restart the flow for the user. On repeated interruption, capture the transcript and escalate.

R13 (generation or review stall): check the audit/outbox transcripts for the generation job, the draft, and the staff-review card within the 600-second bound. No forced generation command exists (the v1/v2 `request-generation` CLI was invented and is void); never call the provider directly. If the job is missing while the check-in is complete, that violates the atomic-enqueue invariant: stop, capture, escalate.

R14 (duplicate delivery, implicit send, or reconciliation mismatch): stop immediately. Capture the outbox/audit/delivery transcripts. Do not send corrective DMs. Classify per the exactly-once contract; escalate with evidence.

R15 (cleanup archive verification FAIL): do not delete anything; capture the cleanup controller verify stdout and the archive manifest/receipt digests; rerun the controller's `verify` once with the same run id; if still FAIL, escalate. Never rerun the cleanup `execute` with a new run id without a superseding permission seal. Do not open or extract archives (they are hash-pinned evidence).

R16 (post-lifecycle cleanup blocked or rejected): the sealed cleanup controller is the ONLY cleanup path (B6 resolved; it archives the full bootstrap ledger byte-exact with the ACTIVE row preserved and forbids any ACTIVE-to-fake-terminal mutation). Its fail-closed rejections include: zero or more than one authorized ACTIVE lifecycle derived from the live ledger (the current zero-ACTIVE state rejects by design, receipt `769db2bd...`); a non-terminal prior session; a customer not disabled; a delivery not sent-and-audited with a terminal provider receipt; unreconciled generation/draft/card/outbox state; lingering jobs, claims, leases, attempt locks, or pending/unknown outcomes; the service still active; archive-commit or immutability failures. On any rejection: stop, capture the receipt, fix the underlying state only through canonical surfaces (for example, run Golden Path Section 13 disable first if the customer is still active), then rerun `dry-run`; never bypass with hand-edits. Approval phrase: the human presents `TASK26_POST_LIFECYCLE_ARCHIVE_CLEANUP_APPROVED`; the run id `task26-post-lifecycle-cleanup-2e0894ea` is one-use; the mode order is `dry-run` -> `execute` -> `verify` with the exact commands in Golden Path Section 14.

## 4. Abort decision matrix (all lanes decision-complete; no row is blocked)

| Gate / phase | Pass condition | Abort action |
|---|---|---|
| Candidate verifier G1 | exit 0, PASS | Stop; rebind required |
| Wheel G2 | digest equal | Stop; wrong wheel |
| Authorization G3 | ledger row 128 present | Stop; escalate to plan owner |
| Config/auth G4 | digests/modes match | Stop; restore config/auth |
| Module pins G5 | all seven equal | Stop; wrong wheel build |
| Authority snapshot G6 | digests match manifest | Stop; drift detected, reconcile evidence |
| Provider auth G7 (B3 closed) | READY receipt pins match | Stop; read-only recheck only (R10) |
| Stale lock G8 | no holder, in contract scope | Stop; escalate; never hand-remove |
| Unit file G9 | sha256 `0b46e887...` | Stop; restore unit |
| Reset controller G10 (B2 closed) | pins match | Stop; escalate |
| Archive verifier G11 (B5 closed) | pins match, receipt PASS | Stop; escalate |
| Invite harness G12 (closed) | pins match; post-reset dry-run/verify exit 0 | Stop; ordering: pre-reset must complete first |
| Sealed lifecycle observer G13 | harness, permission, schema, manifest, verification, inventory, tests9, Ruff, LSP, compile, live-arm-only, exact CLI, modes, order, negative paths, and before/after authority equality all PASS | Stop before any trigger if the three observer READY barrier is absent or any pin drifts |
| Bot API authority G14 | all Golden Path Section 1.2 predicates hold | Stop; no mutation; escalate |
| Post-reset lock G15 | lock absent after reset | Stop; reset incomplete; rerun controller verify (R15) |
| Cleanup controller G16 (B6 closed) | pins match | Stop; escalate |
| Reset dry-run/execute/verify | PASS each | Stop; R15 |
| Harness dry-run/verify (post-reset) | exit 0, ready true | Stop; check lock-absence and pins |
| Deployment | direct_url + module bytes match | Stop; rollback is redeploying the same sealed wheel only |
| Invite preparation | one PASS receipt | Stop; R3 (expiry path if applicable) |
| Claim observation | READY then one accepted claim, one role-swap-block audit | Stop; R4 |
| Consent | accept committed | R5 |
| Onboarding | reaches owner_review/safety_hold | R6/R7 |
| Owner review | card authoritative | R8 |
| Readiness/activation | readiness exit 0; one activation PASS | R9 |
| Check-in | 12 answers, durable job | R12/R13 |
| Generation/review | one draft, one card, one approval | R13 |
| Delivery | exactly one DM, one audit, one receipt | R14 |
| Disable | exit 0, registry disabled | R9 |
| Cleanup dry-run/execute/verify | PASS each; lock absent after | R15/R16 |

## 5. Other profiles (untouched)

Read-only proof only: for `physique-coach` and `quarantine`, capture before and after the rehearsal the sha256 of `customers/registry.json`, `data/customer_bootstrap/ledger.json`, `config.yaml`, the unit file, and the venv `direct_url.json` (`physique-coach` only), plus `systemctl --user is-active hermes-gateway-physique-coach.service hermes-gateway-quarantine.service`. Any change: abort everything. The pinned `physique-coach` tree digest from the sealed reset manifest is `435d8b83e12ede00ca1494e254e146bcdcda06210779cb08fb38b3ecb4a897d8`; both controllers fail closed if it changes.

## 6. Redaction and evidence rules

Raw invite tokens, nonce values, and the 0600 handoff file never appear in shared receipts (the harness emits redacted receipts by design); no customer health data in staff rooms, logs, callback payloads, or idempotency keys; evidence files are mode 600 in a 700 directory; journal captures redact tokens by truncation to the first 8 characters. No raw bot token appears in any receipt (hash-and-length proofs only, Golden Path Section 1.2); never create, copy, restore, or require the obsolete session files (`auth_39664143.session`, `.session-journal`, `.json` under `~/.local/share/hermes/telegram/`); their absence is correct by design.

G12 invite baseline failure: stop before preparation if the post-reset bootstrap ledger or `data/onboarding` is present, if the stable 0700 `data` parent cannot be watched before the candidate API call or its race check fails, if a second creation event occurs, if the prepared session is not exactly generation `1` and state `PREPARED`, or if any operator or harness path manually initializes an empty ledger. The candidate initializer is the sole creator of the absent baseline state. Exact v2 dry-run and verify must pass with `RESET_BOUND_LEDGER_ABSENT`, `mutations: 0`, and `ready_for_one_invite: true`; the exact-wheel deployment and loaded-byte proof in Golden Path Section 4 must pass before preparation.

G13 observer failure: stop before any trigger on missing READY, a nonzero or unknown observer outcome, timeout, duplicate, skipped revision, wrong candidate or lifecycle ID, schema drift, symlink or hardlink, mode or ownership drift, inotify overflow, or any attempted sleep, poll, raw Telegram call, cursor change, drop, or replay. Re-arm only with the prior receipt's `transition.revision_after` in both the manifest and `--after-seq`; never reuse a stale manifest or output. The arm-only proof must show identical before and after live authority hashes.

## 7. Recovery receipt inventory (per incident)

Incident id, timestamp, phase, triggering symptom, commands run (verbatim), outputs captured (digests of files), procedure followed (R number), outcome (recovered or escalated), and the digests of all new evidence files. Append to the run's evidence directory; never overwrite earlier receipts.