# Task26 Recovery Runbook: candidate 2e0894ea same-actor live rehearsal

Companion to `task26-golden-path-2e0894ea.md`. Same bindings, same authority,
same supersession. Append-only: never edit in place; corrections ship as new
superseding files.

## 0. Binding and supersession

| Binding | Value |
|---|---|
| Candidate full digest | `2e0894eac92bc396cc4723bf1f18ebc653b95018dd41574df435941c235da925` |
| Wheel SHA-256 | `af4a9d0a1ffffb6eb7551c1d6dc2b32853ca6d024332a4f8f5702bbf992f141b` |
| Amended plan SHA-256 | `7ace03c6dad33d2fc3ef223621cbca68a150fde8429932138e252fb8498ac582` |
| Rehearsal customer actor | `8527916639` (reused only as a new logical lifecycle) |
| Owner / operator | `8693203710` |
| Service | `hermes-gateway-dualcoachtest.service` |
| Profile root | `/home/cube/.hermes/profiles/dualcoachtest` |

**Supersession.** This runbook supersedes `.omo/evidence/dualcoach-recovery-runbook.md`
(SHA-256 `ae2f5f9046c06f8f0b42693be5aa0d9c31cada8024ae1e5ab8ba19a3cf10f6fc`)
and, together with the Golden Path, supersedes
`.omo/evidence/dualcoach-golden-path-contract.md` (SHA-256
`32a379d855c6e5af978bd9886e3bf49c616c7c20f1c1d5f100c19d8adfc5eb4a`) for this
rehearsal. Both old files stay byte-identical as history. Every candidate-`19ed0e9232e240553b9a96a2e6a33d10f1096be49e5ebafc019287d75984e45c`
command reference in them is void for this run.

## 1. Recovery authority model in this candidate

The candidate recovers by itself through durable reconciliation on gateway
start: bootstrap waiting-state recovery, onboarding publication receipt
recovery, generation lease expiry and retry, delivery outbox reconciliation,
activation-notice drain. A controlled service restart is the primary recovery
lever, because it re-runs all of that through canonical code.

Operator-run commands, the full supported set:

| Purpose | Exact command | Mutates? |
|---|---|---|
| Service status | `systemctl --user status hermes-gateway-dualcoachtest.service` | no |
| Journal | `journalctl --user -u hermes-gateway-dualcoachtest.service -n 200 --no-pager` | no |
| Active probe | `systemctl --user is-active hermes-gateway-dualcoachtest.service` | no |
| Service start | `systemctl --user start hermes-gateway-dualcoachtest.service` | service |
| Service stop | `systemctl --user stop hermes-gateway-dualcoachtest.service` (SIGTERM, `TimeoutStopSec=210`, `KillMode=mixed`) | service |
| Service restart | `systemctl --user restart hermes-gateway-dualcoachtest.service` | service |
| Gateway status (CLI) | `/home/cube/projects/richard/hermes-agent/.venv/bin/hermes --profile dualcoachtest gateway status` | no |
| Lock probe | `flock -n /home/cube/.hermes/profiles/dualcoachtest/gateway.lock true` | no |
| Provider auth | `/home/cube/projects/richard/hermes-agent/.venv/bin/dualcoach_admin provider-auth check --json [--allow-billable-active-probe]` | receipts only, and only with the billable flag under human authorization |
| Readiness audit | `.../python -m checkin_cli.readiness_cli --profile-root <profile> --customer-key <key>` (exit `0` ready, `2` not ready) | no |
| Onboarding status | `.../python -m checkin_cli.nutrition_onboarding_cli --profile-root <profile> --data-root <data> --customer-key <key> status` | no |
| Registry read | `.../python -m checkin_cli.customer_admin --registry <registry> list` / `validate` | no |
| Customer disable | `.../python -m checkin_cli.customer_admin --registry <registry> disable <customer_key>` | registry |
| Archive verify | `python3 <sealed verifier> <archive>` (candidate verifier exits `2` on PASS) | no |
| Reset controller | sealed artifact, modes `dry-run` / `verify` / `execute` (Golden Path Section 2) | profile data |

(`.../python` = `/home/cube/projects/richard/hermes-agent/.venv/bin/python`.)

Commands that do not exist in candidate `2e0894ea`, verified against the
installed argparse source of `hermes_cli.dualcoach_admin`: `process-state`,
`generation reconcile`, `callback inspect`, `card rebuild`, `updates reconcile`,
`outbox reconcile`, `customer disable`. Running any of them fails with argparse
exit `2`. Their duties are covered as follows: process state by systemd +
`gateway_state.json`, generation reconciliation by restart-driven durable
recovery, callback and card inspection by read-only ledger reads plus the
observer receipts, update reconciliation by the runtime, outbox reconciliation
by the runtime, customer disable by `checkin_cli.customer_admin disable`.

## 2. Global abort criteria

Abort the rehearsal (go to Section 4 rollback) when any of these holds:

- A step tempts a prohibited action: raw Telegram listener, `getUpdates`
  harness, update-cursor edit, `drop_pending_updates=True`, direct JSON or
  ledger mutation, manual claim deletion, manual delivery drain, forced card
  publication, direct service replay, hand removal of `gateway.lock`.
- Any other profile (`physique-coach`, `quarantine`, or a new one) shows drift
  from its pinned snapshot.
- A second invite, second Start, second delivery, or any reused ID appears.
- Two consecutive recovery attempts for the same fault fail.
- Provider auth can't reach `ready` through the supported command.
- Any ledger or receipt contradicts another ledger or receipt.

## 3. Procedures

Each procedure lists: trigger, detection, exact commands, expected state,
rollback, evidence. Timeouts are bounded (120 seconds unless stated); event
subscriptions come from the Golden Path observer pattern, never sleeps.

### R1. Service crash mid-lifecycle

- Trigger: `systemctl --user is-active` reports `failed`, or the gateway-state
  observer shows `stopped` unexpectedly.
- Detection: `systemctl --user status`, journal tail.
- Action: none at first. The unit has `Restart=always`, `RestartSec=5`; systemd
  restarts the gateway, and the gateway reconciles from durable state. Wait on
  the gateway-state observer for `running` + telegram `connected`.
- If the unit stays `failed` (start-limit or exit `75` planned-stop marker):
  one `systemctl --user start hermes-gateway-dualcoachtest.service`. Exit
  status `75` means a planned stop marker was written; inspect
  `journalctl --user -u ... -n 200` for the marker reason before the start.
- Expected: service `active`, gateway state `running`, telegram `connected`,
  no duplicate sends (delivery ledger unchanged except in-flight row
  completing exactly once).
- Rollback: if a second crash follows, abort rehearsal, Section 4.
- Evidence: `r1-recovery-receipt.json` with status output digests, journal
  excerpt hashes, before/after `gateway_state.json` digests.

### R2. Stale `gateway.lock` at startup (blocker B1 pattern)

- Trigger: service fails to start, journal reports the lock held by a dead PID.
- Detection: `flock -n <profile>/gateway.lock true`; read the lock JSON; check
  the recorded PID and `start_time` against `/proc`.
- Action: if the PID is dead and `start_time` mismatches, the lock is stale.
  Disposal happens only inside the sealed reset scope (`gateway.lock` is a
  cleared path) or by the candidate's own canonical lock classification at
  startup. Never delete the file by hand.
- Expected: lock absent or owned by the live gateway PID.
- Rollback: abort rehearsal; the pre-reset reset clears it through canonical
  scope.
- Evidence: lock probe output, lock JSON digest, disposition receipt.

### R3. Invite expired before claim

- Trigger: claim watch observes `PREPARED` past `expires_at`.
- Detection: bootstrap ledger read; session terminal `EXPIRED`.
- Action: none to retry. The exactly-one-invite rule stands. Expiry cleanup
  uses the supported pre-claim API `RoomBootstrapStore.expire_unbound` (already
  exercised for session `cb_rmDnfrqA6gkmjoQEdwxu0g`, receipt
  `task26-fresh-canonical-invite-20260815T040558Z/expiry-cleanup-receipt.json`),
  and only inside a sealed harness call.
- Expected: session `EXPIRED`, generation bumped, no claim, no registration.
- Rollback: abort rehearsal. A new invite requires a fresh explicit human
  authorization; it isn't a recovery action.
- Evidence: expiry receipt, ledger digests.

### R4. Claim collision, wrong-actor Start, malformed Start payload

- Trigger: a second actor presses Start, the same actor presses it twice, or a
  Start arrives malformed (wrong token format, group chat, extra text).
- Detection: bootstrap ledger shows the candidate tombstone path; ingress
  receipts count exceeds one for the claim window.
- Action: none. The candidate rejects duplicate or foreign claims through the
  candidate tombstone and binding checks. Malformed Starts never match the
  claim regex `^/start(?:@bot)?\s+(rc1_[A-Za-z0-9_-]{22})$`; the adapter
  ignores them. Confirm the first legitimate claim is intact.
- Expected: session stays claimed by `8527916639` or, if the first claim was
  the foreign one, terminal `FAILED`.
- Rollback: if the legitimate claim is lost, abort rehearsal.
- Evidence: collision receipt with update ids and ledger digests.

### R5. Onboarding stalls in `collecting`

- Trigger: onboarding observer times out with state still `collecting`.
- Detection: `nutrition_onboarding_cli ... status`; onboarding session file
  digest history.
- Action: none mechanical. The customer may still be answering; re-arm the
  observer once with a fresh 120-second bound after confirming with the human
  that the handset is mid-flow. If the customer abandoned the flow, the
  rehearsal ends: cancel through the card's own Cancel action if the card
  offers one; otherwise stop.
- Expected: forward progress or terminal `cancelled`.
- Rollback: abort; cleanup reset clears the partial session.
- Evidence: status output, observer timeout receipt, decision record.

### R5b. Onboarding revision conflict

- Trigger: journal shows a revision-mismatch write failure on the onboarding
  session file, or two writers appear to race one revision.
- Detection: onboarding session file digest history jumps; journal excerpt
  names the conflict.
- Action: one `systemctl --user restart hermes-gateway-dualcoachtest.service`.
  The store enforces revision-checked mutations and the runtime reconciles from
  durable state on startup. Never hand-merge or hand-edit revisions.
- Expected: one writer, forward progress, no lost answers.
- Rollback: conflict persists after the restart, abort rehearsal.
- Evidence: revision digests before/after, journal excerpt hash, restart
  receipt.

### R6. Safety hold during onboarding

- Trigger: onboarding state `safety_hold`.
- Action: no approval. Owner reviews the hold on the review card and rejects
  through the card if rejection is warranted. No direct state edits.
- Expected: `rejected` or a documented owner decision to abort.
- Rollback: abort rehearsal (happy path is dead once a hold fires for the
  synthetic answers, which were authored to avoid holds).
- Evidence: hold receipt, review card state, decision record.

### R7. Generation stuck or failed

- Trigger: generation observer sees `generation_pending` or `generating` past
  timeout, or a `failed` terminal record.
- Detection: read `data/owner/draft-generation-jobs.json`,
  `draft-generations.json`, worker journal digests; journal tail for the worker
  error.
- Action: one `systemctl --user restart hermes-gateway-dualcoachtest.service`.
  Lease expiry plus worker retry is the candidate's recovery; the restart
  re-runs it. Never delete claim or journal files, never re-queue by hand.
- Expected: `draft_created` exactly once, one provider receipt. If the failure
  is provider auth, run R10 first, then this restart.
- Rollback: second failure, abort rehearsal.
- Evidence: job and journal digests before/after, restart receipt.

### R8. Card publish or edit conflict (invite, consent, review)

- Trigger: journal shows Telegram `message not found` or `message is not
  modified` on card operations for any lifecycle card (invite, consent, or
  owner review); owner reports a stale card.
- Detection: card ledger digests (`draft-generation-cards.json`), journal
  excerpt.
- Action: one controlled `systemctl --user restart`. Onboarding and review
  surfaces reconcile from publication receipts on startup.
- Expected: exactly one current card, prior cards terminal, no duplicate card.
- Rollback: if the conflict persists, abort; no forced publication.
- Evidence: card ledger digests, journal excerpt hash, restart receipt.

### R9. Delivery failure or partial send

- Trigger: delivery row present but not terminal `sent_audited` after timeout,
  or customer reports no message while the row claims sent.
- Detection: read `draft-deliveries.json`; journal tail; provider receipt
  presence.
- Action: none that touches Telegram. One controlled restart lets the outbox
  reconciliation finish an in-flight row exactly once. Never re-send by hand,
  never drain the outbox, never replay a callback.
- Expected: one row, terminal `sent_audited`, one provider message. If the row
  is terminally failed, the delivery requirement can't be met this run.
- Rollback: abort rehearsal; the exactly-once requirement makes a retry a new
  rehearsal, not a recovery.
- Evidence: delivery ledger digest, provider receipt digest, restart receipt.

### R10. Provider auth or config failure mid-run

- Trigger: generation fails with auth errors, or G7 re-check fails.
- Detection: `dualcoach_admin provider-auth check --json` output and exit code
  (enum: `ready`, `config_missing`, `config_unreadable`, `config_invalid`,
  `endpoint_missing`, `endpoint_invalid`, `probe_unknown`, `probe_failed`,
  `cli_unavailable`, `cli_login_missing`, `probe_rejected`,
  `profile_mutation`, `receipt_write_failed`, `internal_error`).
- Action: fix only through the profile's normal config path with a human
  approving the edit, then re-run the check. For `openai-codex`, the billable
  active probe (`--allow-billable-active-probe`) requires explicit human
  authorization each time.
- Expected: exit `0` per provider before continuing.
- Rollback: if auth can't be restored, abort rehearsal.
- Evidence: check receipts (they self-write to
  `data/dualcoach-provider-auth`), decision record.

### R11. Duplicate update ingestion suspected

- Trigger: observer sees a ledger append that looks like a replay (same update
  id, same callback id).
- Detection: read-only count of consumed-update journals, ingress receipts,
  per-chat guard records.
- Action: none. The runtime dedupes through consumed-update journals and the
  per-chat ingress guard. Confirm counts stay at one per logical update.
- Expected: duplicates rejected, counts stable.
- Rollback: if a duplicate mutates state (double registration, double
  delivery), abort rehearsal; that's a candidate defect, not an operator fix.
- Evidence: journal counts, receipt digests, defect note if aborted.

### R12. Stale, duplicate, or failed callback

- Trigger: journal shows a callback answer failure (network or Telegram error),
  a callback arrives for a superseded card revision, or the same callback id
  appears twice.
- Detection: callback ingress receipt counts, one-use journal digests, journal
  excerpt.
- Action: none for stale or duplicate callbacks; the runtime rejects them as
  already consumed or superseded. For a failed callback answer, one controlled
  restart re-runs the canonical callback handling from durable state.
- Expected: consumption count stays one per logical callback; no double state
  transition; the card reflects exactly the authenticated owner action.
- Rollback: a stale or duplicate callback mutates state, abort rehearsal; that
  is a candidate defect.
- Evidence: ingress and journal digests, restart receipt if used, defect note
  if aborted.

### R13. Restart between approval and send, or between send and confirmation

- Trigger: the service stops (crash or planned) after owner approval but before
  the explicit send, or after the send tap but before the delivery row turns
  terminal.
- Detection: card ledger shows approved-but-unsent, or the delivery row sits
  non-terminal.
- Action: `systemctl --user start hermes-gateway-dualcoachtest.service` (or let
  `Restart=always` do it). Before-send: the approved card survives; the owner
  may send it once, through the card, after the gateway-state observer confirms
  `running` + `connected`. After-send: the outbox reconciliation finishes the
  in-flight row exactly once on startup; nobody re-taps Send.
- Expected: exactly one delivery row terminal `sent_audited`; one provider
  message; no duplicate card, DM, or receipt.
- Rollback: any duplicate appears, abort rehearsal.
- Evidence: card and delivery ledger digests before/after, observer captures.

### R14. Owner rejection

- Trigger: the owner taps Reject on a review card.
- Detection: card terminal `rejected`; onboarding or draft ledger records the
  rejection.
- Action: none. Rejection is a valid terminal outcome and ends the happy path.
- Expected: terminal rejection state, no send possible, no delivery row.
- Rollback: abort rehearsal; cleanup reset.
- Evidence: rejection receipt, ledger digests.

### R15. Recovery attempt itself fails

- Trigger: any command above exits non-zero unexpectedly, or an expected state
  doesn't arrive within bounds after the allowed retries.
- Action: stop. Write what was observed. No third tries.
- Expected: the fault stays contained; no further mutation of any durable
  state beyond receipts.
- Rollback: Section 4.
- Evidence: failure receipt with command, exit code, digests.

## 4. Rollback and abort matrix

| Stage when aborting | Immediate actions | Rollback target | Evidence |
|---|---|---|---|
| Preflight (any gate OPEN/FAIL) | none; don't start | n/a, nothing changed | preflight receipt |
| Pre-reset failed | stop; keep partial archive sealed | profile untouched or partially archived; no restore | pre-reset receipt |
| Deployment proof failed | stop; service stays stopped | reinstall the previously installed wheel recorded in G14 with the same pip command shape, re-run loaded-byte proof against its digest | deployment receipt |
| Post-invite, pre-claim | stop service: `systemctl --user stop` | expire the invite through the sealed harness (`expire_unbound`); cleanup reset | invite + stop receipts |
| Post-claim, pre-activation | stop service | canonical `disable` not needed (customer not enabled); cleanup reset clears partial state | lifecycle receipts |
| Post-activation, pre-delivery | `customer_admin disable <customer_key>`, then stop service | cleanup reset | disable + stop receipts |
| Post-delivery | `customer_admin disable`, stop service | cleanup reset; delivery receipts stand as evidence | full receipt set |
| Cleanup reset failed | stop; escalate to human with both archive manifests | no restore; archives immutable | cleanup receipt |

Every abort ends in: service stopped, customer disabled (if ever enabled),
cleanup reset executed or explicitly waived by the human in writing, final
verdict receipt with FAIL plus the failing stage.

## 5. Other profiles, secrets, evidence

- `physique-coach` and `quarantine` stay pinned to their snapshot digests
  (authoring-time pins: `e04e3a0a13a73bd7cf2259898e3c9198e5860a4a468a5342c7b0b537a4a19e1e`,
  `9ec0b0f3251f944d9d89a4b6cc3e5a45e15025b52d3a5c0d6a60ea04fa1dc031`).
  Re-verify after any recovery action. No `--all` gateway flags, ever.
- Redaction: receipts carry digests, counts, states, update ids. No bot token,
  provider secret, invite plaintext (outside the one 0600 invite receipt), or
  customer answer text.
- Every recovery action writes a 0600 receipt in the run's evidence dir before
  the next action runs.