# Safe synthetic 30-day adaptive-history plan

## Status and scope

Planning only. Execution is limited to the registered disposable `dualcoachtest` customer `virtual_customer`. It must preserve revision 7, every existing canonical event and sequence row, and the current feature epoch (including `delivery=false` and `activation=true`). No provider, Telegram, gateway/service, deployment, formatter, or live production-proposal path is in scope. Reports must contain only dates, counts, booleans, decision enums, scenario/version labels, and opaque digests; never customer text, raw event IDs, destinations, credentials, or tokens.

Known pre-mutation baseline: canonical event/sequence counts are 44/44; existing synthetic observations overlap recent dates. Treat a mismatch in the read-only preflight as a stop condition, not an opportunity to repair or rewrite state.

## RALPLAN-DR

### Principles

1. **Canonical append-only truth:** create history only through validated `Event` objects and paired `CanonicalEventTransaction` writes; never write or edit JSONL directly.
2. **Schema/projection parity:** an importer may emit only fields represented in both strict `Event` and the canonical JSON schema, and every field used by adaptive projection must be typed and tested end-to-end.
3. **Determinism with provenance:** one versioned, seeded 30-calendar-day scenario produces stable identities, payloads, and dedupe keys; the only execution-time value is the audit recording timestamp.
4. **Fail closed and isolated:** bind the importer to the resolved registered customer/runtime, reject future days, collisions, malformed ledgers, and stale preconditions; do not activate, deliver, or call a provider.
5. **Non-destructive recovery:** preserve immutable records; recover with typed, append-only retraction semantics rather than deletion or raw-file restoration.

### Top decision drivers

1. The discovered P1 is a source contract defect: `baseline_events()` emits `history_imported`, while `project_canonical_events()` requires `import_manifest.observation_kst_day`; strict `Event` and the JSON schema reject that field today.
2. Adaptive decisions need complete canonical signals (including adherence), but `_canonical_adherence_values()` already reads an unmodelled `adherence_signal`; raw JSON cannot be used to bridge either gap.
3. The fixture already has a 44/44 immutable prefix, revision 7, overlapping synthetic dates, and delivery disabled; population must be additive, idempotent, and scoped to one registered customer.

### Options considered

| Option | Pros | Cons |
|---|---|---|
| A. Add a typed manifest/adherence contract and a registered synthetic-history importer, then reconcile and run pure local analysis. **Chosen.** | Repairs P1 at source; validates paired canonical writes; supplies deterministic provenance/adherence; supports idempotence and compensation. | Touches model, JSON schema, importer, projection, admin surface, and tests. |
| B. Hand-author canonical/adaptive JSONL rows or patch existing rows. | Fast apparent data creation. | Violates append-only and supported-API constraints; breaks sequence integrity and cannot prove idempotence. Rejected. |
| C. Drive historical data through current-date wizard/Telegram-like flows. | Reuses UI paths. | Wizard events timestamp at finalization, require transport-style context, and cannot faithfully create prior-day records; would risk live delivery boundaries. Rejected. |
| D. Change projection to infer missing manifest dates from `occurred_at_kst` without an importer contract. | Small code diff. | Hides provenance ambiguity, leaves strict schema drift, and cannot distinguish synthetic or legacy history. Rejected. |

### Chosen decision / ADR

**Decision:** introduce a versioned, strictly typed historical import envelope and a customer-bound synthetic-history API. Make `history_imported` projection consume the root event's typed manifest (including through a terminal correction), accept only a valid archived history import, and use a typed adherence signal. Populate only after importer-to-projection tests pass. Add an append-only retraction event/API so post-append recovery never deletes records.

**Why chosen:** it fixes the actual producer/consumer mismatch instead of making projection guess, gives synthetic records explicit provenance, and leaves the immutable prefix/revision intact. The broader contract changes are warranted because both `import_manifest` and the already-read adherence signal are impossible to persist under the current strict contract.

**Consequences:** legacy imported rows lacking a manifest remain non-projectable and untouched. New baseline imports gain a typed baseline manifest. The synthetic importer is the sole supported writer for this scenario; it is not a generic event append API. Retraction removes only the selected synthetic generation from adaptive projection, not from the canonical ledger.

## Exact implementation plan

### 1. Repair strict canonical schema and history producer first

Modify:

- `/home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli/checkin_cli/models.py`
  - Add frozen, `extra="forbid"` models `ImportManifest`, `AdherenceSignal`, and `SyntheticHistoryRetraction` (names may be adjusted only to match local naming).
  - Add optional `Event.import_manifest`, `Event.adherence_signal`, and `Event.synthetic_history_retraction` fields, and `EventType.SYNTHETIC_HISTORY_RETRACTED`.
  - Extend `Event.validate_discriminator()` so `history_imported` with a manifest has the matching typed shape, synthetic manifests require `provenance.source_type == "derived"`, and retraction events require only the typed retraction payload. Do not make a manifest globally required for existing legacy history rows.
  - `ImportManifest` must include: a fixed manifest version, `kind` (`historical_baseline` or `synthetic_30_day`), `observation_kst_day`, a non-sensitive generation/scenario digest, and for synthetic rows a fixed scenario version and deterministic seed digest. `AdherenceSignal` must formalize the exact target/actual/status/tolerance fields currently consumed by `_canonical_adherence_values()`.
- `/home/cube/.hermes/profiles/physique-coach/contracts/checkin-event.schema.json`
  - Add matching top-level definitions/properties with `additionalProperties: false`, update the event-type enum and conditional requirements, and keep all legacy event shapes valid.
  - Do not add a permissive `object` escape hatch or raw text fields.
- `/home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli/checkin_cli/history_imports.py`
  - Change `baseline_events()` / `_baseline_event()` to attach a deterministic typed `ImportManifest(kind="historical_baseline", observation_kst_day=entry.recorded_date, ...)`; retain archived status and existing deterministic identity semantics.
  - Add `SyntheticHistorySpec`, `SyntheticHistoryReceipt`, deterministic `build_synthetic_history_events(spec, existing_events)`, and a typed retraction-event builder. These constructors must return `Event` values only.

The synthetic scenario is exactly the 30 completed KST calendar days ending at the explicit `--ending-on` day, with `ending-on <= current KST day - 1`; no event day may be future. Its fixed scenario version and seed generate correlated, bounded data: stable/adherent days; planned-training days with proportionally higher carbohydrate/calorie values; a short low-sleep/high-stress/digestion disruption block with lower adherence and a bounded transient weight response; then recovery. Include exactly one no-row calendar omission, two partial-field omissions (one weight/sleep, one nutrition/adherence), and one root-plus-terminal correction whose magnitude is bounded. Never use personal text: structured enum tokens only. For date collisions, classify canonical terminal rows using a read-only snapshot: preserve existing rows; skip a compatible prior row for that day; reject an incompatible same-generation replay; do not supersede, correct, or overwrite pre-existing rows.

### 2. Make the paired transaction/API safe and expose only the bounded importer

Modify:

- `/home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli/checkin_cli/store.py`
  - Harden `CanonicalEventTransaction.append_many()` / `_append_many_locked()` so a dedupe collision is idempotent only when the full validated canonical event is equal; otherwise raise before append. Preserve one canonical row paired with one sequence row.
  - Add `EventStore.import_synthetic_history(spec)` and `EventStore.retract_synthetic_history(generation_digest, reason)`. Both must require a registered `EventStore`, use `_record_lock()` and the paired transaction, validate all candidates before writing, and return sanitized receipts (counts, day range, scenario/generation digests, canonical/sequence count deltas only).
  - Preflight inside the importer must use `CanonicalEventTransaction.read_snapshot_readonly()` and refuse corrupt/torn/mismatched pairs, non-completed days, wrong window size, unsafe source provenance, or a changed scenario generation. It must not call `_append`, write raw JSONL, or use filesystem restoration.
- `/home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli/checkin_cli/customer_admin.py`
  - Add a narrowly named `import_synthetic_history(...)` admin function plus `import-synthetic-history` CLI subcommand. Resolve the runtime exclusively with `_registered_customer()` from the supplied canonical registry and profile root; require explicit customer key, fixed scenario version, fixed seed, and ending date. The execution invocation must name only `virtual_customer` under the `dualcoachtest` profile.
  - Add a `retract-synthetic-history` command that accepts only the sanitized generation digest returned by the import receipt. Neither command may accept event JSON, arbitrary event type, arbitrary source refs, or provider/destination parameters.
  - Emit only sanitized JSON receipts.

### 3. Correct the adaptive projection at the source boundary

Modify `/home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli/checkin_cli/adaptive_nutrition.py`, especially `_EVENT_FLOW_BY_TYPE`, `_canonical_adherence_values()`, and `project_canonical_events()`:

1. Build the correction graph first as today, then for `history_imported` resolve the manifest from the root history event (`day_row`), not only the terminal correction row.
2. Accept an archived `history_imported` event only when its typed manifest validates; retain existing accepted/unsafe handling for live flows. A legacy history row without the manifest remains ignored rather than guessed.
3. Use the typed adherence signal from the validated canonical event; do not read an untyped dict from a raw JSON source.
4. Collect typed synthetic-retraction generation digests before building observations and exclude only matching synthetic history roots. Retraction must not hide baseline history or any nonmatching/pre-existing event.
5. Reject malformed manifest dates, invalid correction graphs, or invalid typed retractions before returning a snapshot. Preserve `canonical_projection=True` and safety-union behavior.

This specifically resolves P1: newly imported baseline and synthetic history rows are both schema-valid and projectable; a terminal correction retains the root observation day/provenance.

### 4. Tests before target mutation

Modify/add tests in:

- `.../checkin_cli/tests/test_event_contract.py`: strict-model/schema parity for valid baseline/synthetic manifests, typed adherence, retraction, forbidden extra fields, and invalid discriminator combinations.
- `.../checkin_cli/tests/test_history_imports.py`: baseline importer emits a manifest; synthetic importer creates only paired canonical rows; exact replay is idempotent; conflicting same dedupe fails without count change; future day/window/foreign runtime rejection; collision preservation; no raw source/customer text in receipts.
- `.../checkin_cli/tests/test_adaptive_nutrition.py`: importer-to-`project_canonical_events()` test proving baseline history contributes samples; a terminal correction uses the root manifest/day; synthetic adherence is counted; missing fields/days are not fabricated; malformed/legacy manifests fail closed or are ignored as specified; retraction excludes only its generation.
- `.../checkin_cli/tests/test_customer_admin.py`: registered customer binding, CLI argument restrictions, sanitized receipt, and retraction idempotence; assert no feature-epoch or proposal-revision write.
- `/home/cube/projects/richard/hermes-agent/tests/gateway/test_adaptive_nutrition.py`: production-adapter regression with schema-valid imported history and reconciled source mappings; corrupted canonical/sequence, unapproved artifacts, and feature state branches continue to reject without delivery.

Expanded test matrix:

- **Unit:** fixed seed/day fixture produces byte-stable event model dumps; all numeric bounds; correlation invariants; exactly bounded omissions/correction; manifest/adherence validation; correction/retraction graph validation; dedupe equality rule.
- **Integration:** registered `EventStore` import writes equal event/sequence deltas under concurrent replay; baseline → canonical projection → snapshot includes expected dates and adherence; reconciliation sees the new prefix without creating a proposal revision; no direct JSONL writes are used.
- **E2E (local-only disposable fixture):** start from a copied test profile, import, reconcile, run pure projection/proposal calculation, emit redacted report, replay import, retract a test generation, and prove original fixture rows/digests survive. Stub/forbid Telegram transport and provider construction; do not start services.
- **Observability:** assert receipts/reports contain scenario version, seed/generation digest, range, before/after counts, sequence parity, revision number, feature booleans, projection counts, decision enum, and fail-closed reason codes only. Assert absence of raw IDs, destinations, tokens, and customer text.

### 5. Target-only execution sequence after code verification

1. Read-only preflight using the registered runtime and `read_snapshot_readonly()`: require 44 events/44 sequence rows, sequence parity, revision 7, and the recorded delivery/activation booleans. Capture only opaque digest/count evidence. Any mismatch stops execution.
2. Run the focused test commands below. Do not populate the real disposable customer until all pass.
3. Invoke the new registered `customer_admin import-synthetic-history` command with the canonical dualcoachtest registry/profile root, `virtual_customer`, an explicit completed KST ending date, fixed scenario version, and fixed seed. The command must return a sanitized receipt. It appends only new paired rows and skips preserved overlapping compatible days.
4. Verify read-only that `event_delta == sequence_delta == receipt.appended_count`, every new row validates against the strict schema, no historical day is future, no existing event/sequence digest changed, and revision remains 7.
5. Call the supported `reconcile_adaptive_nutrition_journals()` once with the registered canonical source and registry so derived source-day mappings are updated. This is the only permitted derived-state mutation; it must not create proposals, revisions, activation changes, delivery rows, Telegram traffic, or provider calls.
6. Run local pure analysis only: `validate_typed_safety`, `project_canonical_events`, `load_approved_adaptive_artifacts`, and `propose`. Never call `AdaptiveNutritionCoordinator.create_production_proposal`, approval, activation, delivery, schedule reservation, or any gateway entrypoint because those can persist lifecycle state or touch provider boundaries.
7. Exercise fail-closed dry-runs in memory/read-only copies: malformed manifest, orphan correction, paired-ledger mismatch, and the current feature configuration with activation true/delivery false. `audit_gate_d_preflight()` is allowed only as a read-only audit; its feature-flag blocker is expected evidence, not a signal to change flags.
8. Write a sanitized results report under the current planning workspace, containing pre/post opaque digests and counts, calendar coverage/omissions, correction/retraction status, revision/epoch flags, decision summary, dry-run branch outcomes, commands, and no sensitive source data.

### Verification commands (execution phase only; do not run during this planning pass)

```bash
cd /home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli
python -m pytest tests/test_event_contract.py tests/test_history_imports.py tests/test_adaptive_nutrition.py tests/test_customer_admin.py

cd /home/cube/projects/richard/hermes-agent
.venv/bin/python -m pytest tests/gateway/test_adaptive_nutrition.py

# The implemented command must use canonical registered paths and explicit completed KST date;
# emit only the sanitized receipt and never accept event JSON.
cd /home/cube/.hermes/profiles/dualcoachtest/workspace/checkin_cli
python -m checkin_cli.customer_admin --registry <canonical-dualcoachtest-registry> import-synthetic-history virtual_customer --profile-root <dualcoachtest-profile-root> --ending-on <completed-kst-date> --scenario-version synthetic_30_day_v1 --seed <fixed-seed>
```

Add an execution-only read-only verification helper or command if needed; it must call `CanonicalEventTransaction.read_snapshot_readonly()` and expose counts/digests, not ledger lines. Run the import command twice and assert the second receipt has zero append delta. The actual command arguments and final report must not print the registry's raw IDs or customer content.

## Acceptance criteria

- P1 is fixed at source: a schema-valid baseline `history_imported` event has a typed `import_manifest` and projects into adaptive samples; a terminal correction retains root day/manifest.
- Strict model and canonical JSON schema agree for manifests, adherence, and typed retraction; no raw/untyped bypass exists.
- The fixed synthetic scenario covers exactly 30 completed KST days with documented, bounded missing/partial/correction cases and deterministic correlations; no future date is emitted.
- Importing via the registered typed API leaves all pre-existing 44/44 rows and revision 7 untouched; event/sequence deltas are equal; overlapping existing records are preserved; exact replay is a zero-write success and differing replay fails closed.
- Adaptive reconciliation and pure local analysis produce useful snapshot/decision evidence while delivery stays false, activation remains unchanged, and no Telegram/provider/deployment/service action occurs.
- Fail-closed branches are demonstrated without writing target data beyond the approved synthetic import/reconciliation; reports are redacted.
- Cleanup/recovery appends at most a typed generation retraction if needed, restores no files, deletes no pre-existing or synthetic canonical rows, leaves delivery false/revision 7 unchanged, and documents the immutable audit trail.

## Deliberate pre-mortem

1. **Schema extension appears valid in Python but JSON-schema validation rejects appends.** Early sign: `validate_event()` fails for a manifest/adherence field. Mitigation: schema/model parity tests run before any target command; importer validates every candidate before acquiring append intent.
2. **An overlap or rerun silently duplicates/skews observations.** Early sign: event and sequence deltas differ from the receipt, or same dedupe has unequal payload. Mitigation: read-only collision classification, equality-enforced `append_many`, fixed scenario/generation digest, and second-run zero-write verification.
3. **Projection still discards history or misplaces a corrected day.** Early sign: imported count rises but snapshot sample/adherence counts do not, or correction day moves to its recording date. Mitigation: root-manifest projection logic and baseline/importer-to-projection tests, including archived status and correction chain.
4. **A local analysis accidentally enters lifecycle/delivery code.** Early sign: proposal revision, adaptive reservation, feature epoch, or provider-call count changes. Mitigation: use only pure `project_canonical_events`/`propose`; forbid coordinator production methods in the execution harness; compare revision/epoch/reservation counts before and after.
5. **Post-append validation fails and an operator tries raw-file rollback.** Early sign: reconciliation or projection fails after a successful paired append. Mitigation: stop, preserve evidence, use the typed generation retraction API only after recording the failure; no truncation, deletion, or manual JSONL edit.

## Non-goals

- No modification of records outside `dualcoachtest` / `virtual_customer`, including profiles, registry entries, credentials, destinations, or providers.
- No migration, deletion, rewrite, resequencing, or repair of the existing 44/44 prefix or revision 7.
- No activation/delivery toggle, schedule reservation, provider call, Telegram send, gateway/service start, deployment, or live adaptive proposal lifecycle.
- No attempt to turn legacy manifest-less imported rows into synthetic or to infer their observation day.

## Follow-ups

Retain the sanitized result report and receipts as execution evidence. Any future live-profile use requires a separate approved plan and must not reuse this synthetic generation or its customer-bound API invocation.