# DualCoach Recovery Runbook

## Metadata and safety boundary

- Candidate digest: `19ed0e9232e240553b9a96a2e6a33d10f1096be49e5ebafc019287d75984e45c`.
- Baseline/release verdict: **NO-GO**; recovery interfaces below marked `planned interface` are not yet proof of working behavior.
- Execution repository: `/home/cube/projects/richard/hermes-agent`; preserve all unrelated dirty files. Never run `git reset`, `git stash`, `git clean`, raw Telegram backlog deletion, manual outbox deletion, or blind delivery retry.
- Current safe CLI diagnostic: `cd '/home/cube/projects/richard/hermes-agent' && python -m gateway.run --help`.

## Recovery procedures

`dualcoach_admin` commands below are **planned interfaces** for tasks 4-17. Until implemented, use only the stated current diagnostic alternative and keep release NO-GO.

| Procedure | Trigger/customer impact | Exact safe command or planned module invocation | Exact log signature | Durable query/expected values | Bounded observable completion (no sleep/poll) | Operator action | Safe/unsafe branch | Cleanup | Escalation/NO-GO |
|---|---|---|---|---|---|---|---|---|---|
| Startup/PID classification | startup, stale PID, possible duplicate gateway | planned: `python -m gateway.platforms.dualcoach_admin process-state --config "$CFG" --json`; current: `python -m gateway.run --help` plus `ps -eo pid=,args=` | `dualcoach.process_state classification=<not_running\|running_owned\|running_foreign\|stale_state>` | process registry/PID record; executable+config fingerprint must match | command exit/event, max command timeout 10s | start only from `not_running`; stop only `running_owned` | safe: owned fingerprint; unsafe: PID-only kill/foreign process | reconcile PID record after owned exit | any foreign/ambiguous PID is NO-GO |
| Generation retry/lease | job queued/running past lease; draft delayed | planned: `python -m gateway.platforms.dualcoach_admin generation reconcile --customer "$CID" --json`; current diagnostic: `python -m pytest -q tests/gateway/test_nutrition_coaching.py` | `dualcoach.generation_reconcile job=<id> from=<state> to=<state>` | job state, lease owner/expiry, input revision; one active lease | await reconciliation event; provider timeout bounded by config | expire only stale owned lease; requeue retryable state | unsafe: two leases or changed input revision | release lease/token | duplicate job or terminal ambiguity is NO-GO |
| Provider auth preflight | missing/expired auth; no draft | planned: `python -m gateway.platforms.dualcoach_admin provider-auth check --json`; current diagnostic: `python -m gateway.run --help` confirms only the config surface, so auth remains NO-GO | `dualcoach.provider_auth status=<ready\|missing\|expired>` | auth readiness metadata only; no secret material | command exit within 10s | reauthenticate outside customer flow; rerun preflight | unsafe: generation call before `ready` | clear diagnostic output; retain no token | missing/expired auth keeps customer disabled |
| Callback ack timeout | button spins; business transition may have committed | planned: `python -m gateway.platforms.dualcoach_admin callback inspect --callback "$CBID" --json`; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_nutrition_onboarding_callback_observability.py` | `dualcoach.callback_ack callback=<id> result=<ok\|timeout>` | callback receipt + business transition revision | await callback/business events independently; ack bound <= configured timeout | query authoritative state before retry | safe: retry UI ack only; unsafe: replay business transition blindly | invalidate stale callback token | unknown transition is NO-GO |
| Callback committed/card edit failed | state changed but staff card stale | planned: `python -m gateway.platforms.dualcoach_admin card rebuild --customer "$CID" --json`; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_nutrition_operator_console.py` | `dualcoach.card_edit result=failed state_committed=true` | authoritative review/delivery state + current card binding | await rebuild send/edit receipt | rebuild from durable state; never roll back commit for UI | unsafe: infer state from stale card | invalidate old card controls | card/privacy mismatch is NO-GO |
| Pending update preservation/replay | restart with Telegram backlog; input delayed | planned: `python -m gateway.platforms.dualcoach_admin updates reconcile --json`; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_room_bootstrap_transport.py` | `dualcoach.update_reconcile pending=<n> committed=<n>` | update cursor + committed update IDs; no forward cursor without commit | await exact update commit event | replay only uncommitted update through normal ingress | unsafe: delete backlog or advance cursor manually | confirm cursor follows last commit | any lost/duplicate update is NO-GO |
| Restart mid-generation | process exits in queued/running/provider call | planned generation reconcile command above; current diagnostic: `python -m pytest -q tests/gateway/test_nutrition_coaching.py` | `dualcoach.generation_resume job=<id> classification=<queued\|lease_expired\|unknown_provider>` | immutable input revision, job state, lease, attempt | await resumed job terminal/state event | resume queued; expire stale lease; classify uncertain provider result | unsafe: create new job for same revision | release owned lease | changed input/duplicate draft is NO-GO |
| Restart with delivery state | restart at intent/attempt/no receipt/unknown_delivery | planned: `python -m gateway.platforms.dualcoach_admin outbox reconcile --customer "$CID" --json`; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_nutrition_operator_console.py` | `dualcoach.outbox_reconcile key=<id> state=<intent\|attempt\|receipt\|unknown_delivery>` | one idempotency key; intent/attempt/receipt records | await durable reconciliation event | resume intent; reconcile attempt/unknown before any send | unsafe: blind retry after attempt | close only with receipt/evidence | unresolved unknown delivery is NO-GO |
| Operator card rebuild | restart/stale message/revision conflict | planned card rebuild command above; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_nutrition_operator_console.py` | `dualcoach.card_rebuild customer=<id> revision=<n>` | latest draft/review/delivery revision and safe metadata | await Telegram send/edit receipt | create one current card; disable stale controls | unsafe: expose health text or accept stale revision | persist new binding, invalidate old | privacy/revision mismatch is NO-GO |
| Receipt/duplicate callback reconciliation | repeated deliver callback or ambiguous provider response | planned outbox reconcile + callback inspect commands; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_nutrition_onboarding_callback_observability.py` | `dualcoach.delivery_receipt key=<id> telegram_message=<id>` | unique key, max one receipt; duplicate count/audit | await receipt or explicit unknown classification | return authoritative state; never second send | unsafe: treat callback ack/provider 200 as receipt | close duplicate audit | duplicate DM or missing receipt is NO-GO |
| Rehearsal disable/gateway cleanup | live drill completed/failed | planned: `python -m gateway.platforms.dualcoach_admin customer disable --customer "$CID" --json`; current diagnostic: `python -m pytest -q tests/gateway/test_telegram_room_bootstrap_activation.py`; then process-state command | `dualcoach.customer_disabled customer=<id>` and `dualcoach.process_state classification=not_running` | disabled customer; zero active jobs/intents/attempts; truthful PID | await disable commit and owned-process exit event | disable first; reconcile outbox; stop only owned PID | unsafe: leave active customer/pending delivery | revoke invite, clear test-only runtime data | any live resource/pending delivery is NO-GO |

## Adversarial recovery matrix

| Class | Required recovery proof |
|---|---|
| dirty_worktree | candidate digest is checked; unrelated bytes remain untouched; destructive Git commands forbidden |
| stale_state | PID/card/lease/cursor is cross-checked against durable authority, never trusted alone |
| malformed_input | update/callback/provider payload is rejected before partial commit; redacted reason recorded |
| prompt_injection | customer/provider prose remains data and cannot change recipient, authority, command, or recovery branch |
| cancel_resume | restart at every durable state resumes without lost job/update or duplicate delivery |
| hung_or_long_commands | command/provider/API/card operations have bounded timeout and explicit retry/unknown state |
| flaky_tests | subscribe to exact event/state before trigger; fixed sleeps and polling delays are forbidden |
| misleading_success_output | CLI/API/UI success is corroborated by durable state and provider receipt |
| repeated_interruptions | repeated restarts preserve one lease, one intent, one attempt/receipt per idempotency key |

## Current non-destructive manual QA drill

Status: **independent verifier must execute**.

```text
cd '/home/cube/projects/richard/hermes-agent' && python -m gateway.run --help
```

Expected: exit code `0`; output begins `usage: run.py [-h] [--config CONFIG] [--verbose]` and lists `--config` and `--verbose`. Trigger: operator needs a safe bounded gateway surface check. Observed state must be help-only: no gateway start, customer activation, Telegram delivery, durable mutation, credential output, or new long-lived process. This proves the current diagnostic boundary only; it does not prove planned process reconciliation, so release remains NO-GO.

## Cleanup receipt and residual gaps

The runbook itself creates no temp files, processes, credentials, customers, runtime state, or product/test changes. After any drill, disable the isolated customer, revoke invite, reconcile jobs/outbox, stop only the owned gateway, and confirm no pending delivery or live process. Tasks 4-17 must implement and test the planned interfaces; tasks 18-27 and F1-F5 must prove automated matrices, fresh Telegram behavior, cleanup, provenance, and independent release review before a bounded pilot verdict.