# Task26 Recovery Runbook, candidate 2e0894ea (v4)

Version: v4 (2026-08-15, supersedes the v3, v2, and v1 recovery runbooks listed below). v4 corrects exactly two path contracts (the archive-verifier permission seal path in Section 1, and the G14 Bot API authority predicates) and preserves all v3 logic and gates otherwise.
Status: READY. All v2 open blockers (B3, G12, B6) are closed with sealed evidence. This document supports the v4 Golden Path `task26-golden-path-2e0894ea-v4.md` and binds the same candidate `2e0894eac92bc396cc4723bf1f18ebc653b95018dd41574df435941c235da925`, wheel `af4a9d0a1ffffb6eb7551c1d6dc2b32853ca6d024332a4f8f5702bbf992f141b`, plan `7ace03c6dad33d2fc3ef223621cbca68a150fde8429932138e252fb8498ac582`, actor `8527916639`.

Supersession (append-only; all prior documents remain byte-identical):

- Supersedes `task26-recovery-runbook-2e0894ea-v3.md` sha256 `e62691493d2a75c312dd248a07bbc52f7f1d98d4acd701b824bbaf5edf8bdd3a` and `task26-golden-path-2e0894ea-v3.md` sha256 `ff230719eb5a69ae821bb0a4d58356d59b2803c490d692659a197e22083793c7` plus receipt `runbook-binding-receipt-v3.json` sha256 `fd77a4d387b9bab7978c84cb51b1ab3200d52a126dea411db98492f6be67fb88`.
- Supersedes `task26-recovery-runbook-2e0894ea-v2.md` sha256 `6024c26ccae3cad7fcf46a77bca8607d73293a92ddb3dcf8fa5a278361869f1d` and `task26-golden-path-2e0894ea-v2.md` sha256 `633d5c25ce70cb9ab5d19c434122b14b46fb7108962ab6e453ce6ab3ede1848f`.
- Supersedes `task26-recovery-runbook-2e0894ea.md` sha256 `f4f2d2347b4b68a080b9c437a813accbef8c4bcc471cbf21d67ffff31e56f89b` and `task26-golden-path-2e0894ea.md` sha256 `1f55b8f967c6564113574d41bf1c15ce977d35c38b5687f1623ba37f235a7bdb` (v1; must remain byte-identical for the sealed invite harness).
- Still supersedes `.omo/evidence/dualcoach-recovery-runbook.md` sha256 `ae2f5f9046c06f8f0b42693be5aa0d9c31cada8024ae1e5ab8ba19a3cf10f6fc` and `.omo/evidence/dualcoach-golden-path-contract.md` sha256 `32a379d855c6e5af978bd9886e3bf49c616c7c20f1c1d5f100c19d8adfc5eb4a` (stale candidate `19ed0e6047f0a4d7650c47a7413ee0298a243f768a63bcb8fb9cffe44d140a1a`; void).

Hard safety rules: stop all mutating action before any diagnosis; no restore, prepopulation, repair-in-place, profile-state hand-edits, raw Telegram listeners, cursor edits, service replay, forced publication, candidate rebuild, manual `gateway.lock` deletion, or recovery shortcut, ever. All evidence in this run's evidence directory, mode 600 (directory 700). No change to `physique-coach` or `quarantine` at any point.

## 1. Supported command set (the only supported CLIs; run from the repository root unless stated; PY = `/home/cube/projects/richard/hermes-agent/.venv/bin/python`)

| Interface | Supported command shape | Notes |
|---|---|---|
| Candidate verifier | `python3 <successor dir>/verify_candidate.py <successor dir>/verifier-input.json` | Successor dir = `.omo/evidence/task26/task26-repaired-archive-successor-2e0894eac92bc396cc4723bf1f18ebc653b95018dd41574df435941c235da925`; exit 0 = PASS; exit 1/2 = FAIL; no flags |
| Pre-reset controller | `python3 '.omo/evidence/task26/reset-controller-st_01a0054d/reset_controller.py' {dry-run,execute,verify} --profile ... --other-profile ... --archive-root ... --contract ... --permission ... --candidate ... --wheel-sha256 ... --plan-sha256 ... --approval ... --run-id ...` | Exact execute command in Golden Path Section 3; verify is post-execute only; run ids one-use |
| Cleanup controller | `python3 '.omo/evidence/task26/reset-controller-st_01a0054d/post-lifecycle-cleanup-v2/cleanup_controller.py' {dry-run,execute,verify} ... --actor-id '8527916639' --run-id ...` | Same flags as pre-reset plus `--actor-id`; verify takes `--archive`; optional `--systemctl`/`--service` defaults `/usr/bin/systemctl` / `hermes-gateway-dualcoachtest.service`; exact commands in Golden Path Section 14 |
| Invite harness | `PY -B invite_harness.py {dry-run,verify,prepare,observe,expire} --profile <profile> --permission <permission-seal.json> --receipt <path>` | `prepare` adds `--draft/--handoff [--evidence-root]`; `observe` adds `--session-id/--sid-hash/--ready/--timeout-seconds [--ready-fd]`; all run from repository root; full commands in Golden Path Sections 5-7 |
| Archive verifier | `python3 .omo/evidence/task26/verify_retained_archives.py .omo/evidence/task26/retained-archive-schema-inventory.json --permission-seal .omo/evidence/task26/retained-archive-verifier-permission-seal.json [--receipt <out>] (actual seal file, sha256 28ff72fd...; strict argument order per the sealed verifier: positional inventory, then --permission-seal, then optional --receipt; exit 0 = PASS, exit 1 = FAIL)` | Verifier sha256 `12b97aa7...`; inventory `29db87f2...`; seal `28ff72fd...`; sealed receipt `597a37e4...` (PASS, five archives, four schemas) |
| Provider auth | `HERMES_HOME=/home/cube/.hermes/profiles/dualcoachtest /home/cube/projects/richard/hermes-agent/.venv/bin/dualcoach_admin provider-auth check [--probe chat.completions.create] [--allow-billable-active-probe]` | The `HERMES_HOME` profile binding is mandatory; see R10 for the READY receipt evidence |
| Customer registry | `PY -m checkin_cli.customer_admin --registry /home/cube/.hermes/profiles/dualcoachtest/customers/registry.json <subcommand>` | Real subcommands only: `add`, `disable`, `activate`, `record-payment`, `record-satisfaction`, `record-operator-time`, `judge-kpi`, `list`, `validate`, `consent`, `revoke-consent`. `activate` takes `--profile-root --data-root --checklist-evidence <customer_id>`; `disable` takes `<customer_key>` only (no reason flag) |
| Readiness audit | `PY -m checkin_cli.readiness_cli --profile-root /home/cube/.hermes/profiles --customer-key <key>` | Read-only; exit 0 with all items satisfied |
| Nutrition onboarding CLI | `PY -m checkin_cli.nutrition_onboarding_cli {status,purge-incomplete,operator-review,seed-kb,migration-preflight,migration-commit}` | Diagnostic use only in this rehearsal |
| Event watchers | `python3 scripts/telegram_nutrition_onboarding_e2e.py {subscribe-outbox,audit-tail,watch-deliveries} --profile /home/cube/.hermes/profiles/dualcoachtest --after-seq <n> --timeout <s> --output <path>` | Trusted workspace checkout only; bounded; re-arm after each firing |
| Service control | `systemctl --user {start,stop,restart,status} hermes-gateway-dualcoachtest.service`; `journalctl --user -u hermes-gateway-dualcoachtest.service` | User-scope unit; unit file `/home/cube/.config/systemd/user/hermes-gateway-dualcoachtest.service`; `Restart=always`, `RestartForceExitStatus=75` |
| Gateway admin | `PY -m hermes_cli.main --profile dualcoachtest gateway ...` | Runtime interpreter is the shared venv above (no profile-local venv exists) |

Void interfaces (do not invoke; some were wrongly cited in v1/v2): `python -m hermes_cli.main gateway run --port`, `curl http://127.0.0.1:8080/...` (no documented HTTP surface), any `/reload`, `/validate`, `/redeliver`, or replay control, any bootstrap/customer DB or ledger edit command, `hermes verify`, any task-scoped controller's internal reset modes beyond the sealed `dry-run/execute/verify`, `hermes_cli.nutrition_readiness` and `hermes_cli.nutrition_review_service` (neither module exists anywhere; there is no generation CLI at all — generation is enqueued atomically at check-in completion), `customer_admin create/show/reset/set-next-checkin/delete` (do not exist), and any manual `gateway.lock` deletion. The plan's "planned interface" commands (`process-state`, `generation reconcile`, `gate telegram-registration`, `notify-unbound`, rollback to a per-profile venv, candidate wheel rebuild) remain VOID (not implemented).

## 2. Global abort rules

Abort immediately to the relevant procedure on: any gate FAIL or UNKNOWN; a mutating command failing; a `RuntimeError`, `auth_key_duplicated`, flood, `ApiIdInvalid`, `EOFError`, `TimedOut`, privacy, or consistency journal error; a state transition outside the golden order; any pending `conflict_review` or unresolved conflict; any second invite preparation attempt, second Start claim, second DM or delivery, or implicit send; missing journal evidence for a claimed transition; any touched-file digest in the other profiles changing; any session/cron/process still running after disable; any reset/cleanup archive verification FAIL; any human-reported crash. On abort: capture the journal tail, subscription transcripts, and CLI output first; then diagnose; the only permitted mutation is the documented procedure step. Bounded deadlines (watch 120s, connect 90s, claim 120s, consent-to-review 300s, generation card 600s, send-to-DM 300s, CLI exit 60s): an exceeded bound is a STOP, not a retry.

## 3. Failure procedures

R1 (service failed to start): `status`, `is-enabled`, `is-active`; `journalctl --user -u hermes-gateway-dualcoachtest.service -n 200 --no-pager` (capture first). If the port is in use, identify the holder with `ss -ltnp`; do not kill by guess. Never loop restarts: after two failed starts, stop and capture. If config YAML fails to parse, restore only from the documented config file, never by hand-editing state.

R2 (stale `gateway.lock`): pre-reset, the lock's presence is allowed only as sealed reset input. Prove no active holder with `flock -n ... true`; then the sealed pre-reset (Golden Path Section 3) archives and disposes of it through `approved_clear_scopes`. The sealed invite harness refuses to run while the lock exists, which enforces the ordering. Post-reset, the post-reset gate requires the lock absent. Any other deletion of the lock file is forbidden. No manual removal at any point.

R3 (invite expired or preparation rejected): preparation rejects with exit 2 when the lock is still present (ordering violation: run the pre-reset first) or when the sealed one-prepare maximum is already spent. If the invite expires unclaimed, mark it terminal through the sealed harness exactly once: `PY -B invite_harness.py expire --profile ... --permission ... --receipt <run-id>-invite-expiry.redacted.json`, then run one new sealed preparation (Golden Path Section 6). Never reuse an expired token, never re-arm the historical manifest, never edit `ledger.json` by hand. Evidence: preparation/expiry receipts.

R4 (claim failed or wrong actor claimed): read `data/customer_bootstrap/ledger.json`; the real session must be `expired` or `cancelled` with the conflicting `role_claims` recorded. No ledger hand-edit. Return to R3.

R5 (consent timeout): wait to the 120-second bound once; capture the audit transcript (`canonical_customer_consent_*`). If no consent arrives, leave the session `claimed`, stop, and return to R3. Do not press buttons for the user.

R6 (onboarding abandoned mid-flow): bounded wait once; do not re-DM the user. If abandoned, the session stays non-terminal; capture evidence and escalate. Do not synthesize a terminal state.

R7 (safety_hold): use the owner card's `safety_hold` action; never bypass via config or state edits. If the safety_hold cannot be cleared, escalate with the audit transcript.

R8 (owner card missing/stale): confirm the onboarding ledger shows `owner_review`/`safety_hold` and the audit shows `canonical_customer_onboarding_review_card`; a stale card is superseded by the newest card message, which is the only authority. Do not double-send cards from the CLI.

R9 (customer_admin CLI error): use only the real subcommands in Section 1. `activate` requires `--profile-root`, `--data-root`, `--checklist-evidence`, and the customer id; it is one-use and fails closed on a second attempt (treat a second attempt as an incident). `disable` takes only the customer key (no reason flag exists). If activation fails, capture stderr; the only retry is after proving the first attempt did not commit (registry `service_state` unchanged).

R10 (provider auth/check failures): the authorized active probe already passed with a sealed READY receipt (raw file sha256 `8278dde4efa8bcd366fce873ccc07f651818275238b67452b9e6cb93c67afd6d`, payload `receipt_sha256` `6aeaaab42e0030b0f626037d7b9ef6661ec93fb57fd991c843967c1fc695905c`, index `16acf1b355715379ad526433803353e1f34c418e9560fc1466f3f77f951df919`, one request attempt, `billable: true`, `store: false`, 26 total tokens, pre/post profile snapshot identical at `7d163cb0...`). Re-diagnosis is read-only: run `provider-auth check` (no probe) with the mandatory `HERMES_HOME=/home/cube/.hermes/profiles/dualcoachtest` binding. The receipt field `candidate_digest` `f9a46172386333a0067f43695fb1429a043a606f04baceb06e8c59b90c36235c` is the provider command digest emitted by module `gateway/platforms/dualcoach_admin.py` (`0e9b4b1f...`); it is a different domain from the candidate envelope digest `2e0894ea...` and is not candidate drift. If a new billable probe is ever needed, it requires its own human authorization first. If auth is missing or expired: stop; no token/secret mutation inside the rehearsal window.

R11 (subscription or audit gap): re-arm the subscription with `--after-seq` past the last observed `seq`. A transition with no durable event is unproven; treat as a defect, not as "probably fine".

R12 (check-in interrupted): the check-in state machine is durable; the customer may resume by DM. Do not restart the flow for the user. On repeated interruption, capture the transcript and escalate.

R13 (generation or review stall): check the audit/outbox transcripts for the generation job, the draft, and the staff-review card within the 600-second bound. No forced generation command exists (the v1/v2 `request-generation` CLI was invented and is void); never call the provider directly. If the job is missing while the check-in is complete, that violates the atomic-enqueue invariant: stop, capture, escalate.

R14 (duplicate delivery, implicit send, or reconciliation mismatch): stop immediately. Capture the outbox/audit/delivery transcripts. Do not send corrective DMs. Classify per the exactly-once contract; escalate with evidence.

R15 (cleanup archive verification FAIL): do not delete anything; capture the cleanup controller verify stdout and the archive manifest/receipt digests; rerun the controller's `verify` once with the same run id; if still FAIL, escalate. Never rerun the cleanup `execute` with a new run id without a superseding permission seal. Do not open or extract archives (they are hash-pinned evidence).

R16 (post-lifecycle cleanup blocked or rejected): the sealed cleanup controller is the ONLY cleanup path (B6 resolved; it archives the full bootstrap ledger byte-exact with the ACTIVE row preserved and forbids any ACTIVE-to-fake-terminal mutation). Its fail-closed rejections include: zero or more than one authorized ACTIVE lifecycle derived from the live ledger (the current zero-ACTIVE state rejects by design, receipt `769db2bd...`); a non-terminal prior session; a customer not disabled; a delivery not sent-and-audited with a terminal provider receipt; unreconciled generation/draft/card/outbox state; lingering jobs, claims, leases, attempt locks, or pending/unknown outcomes; the service still active; archive-commit or immutability failures. On any rejection: stop, capture the receipt, fix the underlying state only through canonical surfaces (for example, run Golden Path Section 13 disable first if the customer is still active), then rerun `dry-run`; never bypass with hand-edits. Approval phrase: the human presents `TASK26_POST_LIFECYCLE_ARCHIVE_CLEANUP_APPROVED`; the run id `task26-post-lifecycle-cleanup-2e0894ea` is one-use; the mode order is `dry-run` -> `execute` -> `verify` with the exact commands in Golden Path Section 14.

## 4. Abort decision matrix (all lanes decision-complete; no row is blocked)

| Gate / phase | Pass condition | Abort action |
|---|---|---|
| Candidate verifier G1 | exit 0, PASS | Stop; rebind required |
| Wheel G2 | digest equal | Stop; wrong wheel |
| Authorization G3 | ledger row 128 present | Stop; escalate to plan owner |
| Config/auth G4 | digests/modes match | Stop; restore config/auth |
| Module pins G5 | all seven equal | Stop; wrong wheel build |
| Authority snapshot G6 | digests match manifest | Stop; drift detected, reconcile evidence |
| Provider auth G7 (B3 closed) | READY receipt pins match | Stop; read-only recheck only (R10) |
| Stale lock G8 | no holder, in contract scope | Stop; escalate; never hand-remove |
| Unit file G9 | sha256 `0b46e887...` | Stop; restore unit |
| Reset controller G10 (B2 closed) | pins match | Stop; escalate |
| Archive verifier G11 (B5 closed) | pins match, receipt PASS | Stop; escalate |
| Invite harness G12 (closed) | pins match; post-reset dry-run/verify exit 0 | Stop; ordering: pre-reset must complete first |
| Subscription harnesses G13 | syntax clean | Stage from trusted source; never the venv |
| Bot API authority G14 | all Golden Path Section 1.2 predicates hold | Stop; no mutation; escalate |
| Post-reset lock G15 | lock absent after reset | Stop; reset incomplete; rerun controller verify (R15) |
| Cleanup controller G16 (B6 closed) | pins match | Stop; escalate |
| Reset dry-run/execute/verify | PASS each | Stop; R15 |
| Harness dry-run/verify (post-reset) | exit 0, ready true | Stop; check lock-absence and pins |
| Deployment | direct_url + module bytes match | Stop; rollback is redeploying the same sealed wheel only |
| Invite preparation | one PASS receipt | Stop; R3 (expiry path if applicable) |
| Claim observation | READY then one accepted claim, one role-swap-block audit | Stop; R4 |
| Consent | accept committed | R5 |
| Onboarding | reaches owner_review/safety_hold | R6/R7 |
| Owner review | card authoritative | R8 |
| Readiness/activation | readiness exit 0; one activation PASS | R9 |
| Check-in | 12 answers, durable job | R12/R13 |
| Generation/review | one draft, one card, one approval | R13 |
| Delivery | exactly one DM, one audit, one receipt | R14 |
| Disable | exit 0, registry disabled | R9 |
| Cleanup dry-run/execute/verify | PASS each; lock absent after | R15/R16 |

## 5. Other profiles (untouched)

Read-only proof only: for `physique-coach` and `quarantine`, capture before and after the rehearsal the sha256 of `customers/registry.json`, `data/customer_bootstrap/ledger.json`, `config.yaml`, the unit file, and the venv `direct_url.json` (`physique-coach` only), plus `systemctl --user is-active hermes-gateway-physique-coach.service hermes-gateway-quarantine.service`. Any change: abort everything. The pinned `physique-coach` tree digest from the sealed reset manifest is `435d8b83e12ede00ca1494e254e146bcdcda06210779cb08fb38b3ecb4a897d8`; both controllers fail closed if it changes.

## 6. Redaction and evidence rules

Raw invite tokens, nonce values, and the 0600 handoff file never appear in shared receipts (the harness emits redacted receipts by design); no customer health data in staff rooms, logs, callback payloads, or idempotency keys; evidence files are mode 600 in a 700 directory; journal captures redact tokens by truncation to the first 8 characters. No raw bot token appears in any receipt (hash-and-length proofs only, Golden Path Section 1.2); never create, copy, restore, or require the obsolete session files (`auth_39664143.session`, `.session-journal`, `.json` under `~/.local/share/hermes/telegram/`); their absence is correct by design.

## 7. Recovery receipt inventory (per incident)

Incident id, timestamp, phase, triggering symptom, commands run (verbatim), outputs captured (digests of files), procedure followed (R number), outcome (recovered or escalated), and the digests of all new evidence files. Append to the run's evidence directory; never overwrite earlier receipts.