# Task22 successor gateway-wheel reproducibility debug journal

Started: 2026-08-11 (session `st_019ff054`)

## Constraints

Read-only investigation until two reproductions and toggle proof. No live profile/service/Telegram/plan/ledger/todo operations. No destructive Git or shell file mutations. Before cleanup, record exact verifier-created paths, bytes, and provenance; delete only those paths via patch.

## Initial observations

- Verifier `st_019ff04a` reports two deterministic gateway builds at SHA-256 `252f8669199b9701f9246e20045009b4be58df8f625368927a591a9f5b7b3db2`, differing from sealed claim `8cb54f7e7fa420e8ea55136e9ebd0999ff4e00963fd7483786ad4128071845d5`.
- The profile wheel reproduced at `104cfdda58eeb3487a075426598a3548fbb7a7d16f50eb81cd8a47e0fc090347`.
- The verifier reports that its standard gateway build wrote three untracked `build/lib/...` copies and changed the status snapshot.
- Gateway repository HEAD is `0dc0f228dcec8fa54430882daf72e901d6a89f79`; the worktree is intentionally dirty and must be preserved.

## Orthogonal hypotheses and distinguishing evidence

1. **Build-path/staging/epoch mismatch.** The seal used `scripts/reproducible-wheel-build` or an equivalent copied-source deterministic path, while the verifier invoked the standard backend in the live tree. Distinguish by comparing exact commands, `SOURCE_DATE_EPOCH`, staging roots, backend/config settings, wheel member timestamps/order/modes, and by building the same bytes through both paths with toggles.
2. **Live `build/lib` contamination.** Standard backend-created copies are later consumed or alter status/candidate inputs. Distinguish by proving the exact newly created paths against source bytes/task transcript, then building with those paths present versus an isolated source copy without them; inspect backend discovery and wheel membership.
3. **Stale or differently bound pin.** `8cb54f...` belongs to another source set/toolchain/backend version despite deterministic current output. Distinguish by authenticating prior seal evidence, candidate leaf/source pins, Python/build/setuptools/wheel versions, generated `METADATA`/`WHEEL`/`RECORD`, and by checking whether canonical current builds consistently yield `252f...` regardless of contamination.

## Evidence log

### Pre-change reproduction and toggle proof

- Prior sealing transcript `st_019fefc3` proves the seal did **not** build the live repository directly. It made two `rsync` copies and used unanchored exclusions including `--exclude=dist`, then set all copy mtimes to epochs `1700000000` and `1800000000`. Both final wheels were `8cb54f7e7fa420e8ea55136e9ebd0999ff4e00963fd7483786ad4128071845d5`.
- Verifier `st_019ff04a` called `scripts/reproducible-wheel-build` twice with the live gateway root as `SOURCE_DIR`. Both outputs were `252f8669199b9701f9246e20045009b4be58df8f625368927a591a9f5b7b3db2`.
- Independent rerun before cleanup reproduced the live-source route twice at `252f866...` and the old unanchored clean-copy route twice at `8cb54f...`.
- Wheel comparison: live/correctly staged wheel has 991 members; old sealed wheel has 987. `METADATA` (`c5c638...`, 22628 bytes), `WHEEL` (`69e622...`, 91 bytes), member timestamps, modes, and compression are identical. Only `RECORD` differs because four package-owned members are absent from the old wheel.
- The omitted files are tracked and explicitly selected by `[tool.setuptools.package-data] plugins = ["*/dashboard/dist/*", ...]`:
  - `plugins/hermes-achievements/dashboard/dist/index.js` (46871 bytes, `0ea5a804...`)
  - `plugins/hermes-achievements/dashboard/dist/style.css` (17984 bytes, `275d7343...`)
  - `plugins/kanban/dashboard/dist/index.js` (163617 bytes, `a3a8e529...`)
  - `plugins/kanban/dashboard/dist/style.css` (45164 bytes, `977b3d63...`)
- Distinguishing toggle: an otherwise clean copy with only top-level generated directories excluded (anchored `/build`, `/dist`, `/*.egg-info`) preserves nested dashboard `dist` directories and reproduced `252f866...` twice across the two source-mtime epochs. Thus the stale pin is bound to a lossy ad hoc staging source set, not a toolchain, source-mtime, ZIP-metadata, or backend nondeterminism.
- Contamination toggle: adding only the three verifier `build/lib` copies to an otherwise old-style clean staging still yielded `8cb54f...`; adding only live `hermes_agent.egg-info` also yielded `8cb54f...`. The three copies affect worktree status but not wheel membership/digest.

### Verifier artifact provenance receipt (before deletion)

Verifier transcript identifies exactly these three status entries as new. Each is untracked, byte-identical to its canonical source, and has the source mtime rounded by setuptools copying:

- `build/lib/gateway/platforms/task22_callback_storage_adoption.py`: 87199 bytes, SHA-256 `fe3bea15e33b2feddf6aaa3ca3b919dee65287b5c988d95aae4db81b605b5e7c`.
- `build/lib/gateway/platforms/task22_trainer_authority_removal.py`: 79591 bytes, SHA-256 `6b89eaaa981a760436e8c510856bc6a13807a38c369ab363feb8f4a45d3978b2`.
- `build/lib/gateway/platforms/telegram_nutrition_onboarding_supplemental_review.py`: 69096 bytes, SHA-256 `ee61b316ae7d67de5558319bd5cee978b56a3569a4a37722602c6b0d1f51b0b0`.

Current pre-cleanup status is 995 records / 46241 bytes / SHA-256 `704ac72d50a609ec5c177890604c01d9b9add8005aafe6f4c17cb9b628dd8019`; expected sealed status is 992 records / 46023 bytes / SHA-256 `f8a6731706568ea377f17d1511b6ce0372da3483e2b539a1b39dded8f7fed2dc`. No other generated copy is authorized for removal.

## Root-cause decision before changes

Hypothesis 3 is confirmed, with hypothesis 1 explaining how it happened: the stale wheel pin was produced from an accidentally lossy staging set due to an unanchored `dist` exclusion. Hypothesis 2 is rejected as a wheel-digest cause but confirmed as status contamination. The product builder and backend deterministically include the declared package assets when given the complete source set, so no product behavior regression test applies; the owning seal/evidence pin and wording must be corrected. The existing deterministic-builder test remains valid.

## Cleanup and corrected seal receipt

- Deleted by exact generated patches, and no direct unlink/`rm`/clean/reset/restore was used:
  - `build/lib/gateway/platforms/task22_callback_storage_adoption.py`
  - `build/lib/gateway/platforms/task22_trainer_authority_removal.py`
  - `build/lib/gateway/platforms/telegram_nutrition_onboarding_supplemental_review.py`
- Immediately after deletion, all three were absent and gateway status returned exactly to 992 records / 46023 bytes / `f8a6731706568ea377f17d1511b6ce0372da3483e2b539a1b39dded8f7fed2dc`.
- Corrected only the owning wheel claim, manifest summary pins, freeze cross-binding, and standalone verifier constants. No candidate-root or profile-package byte changed.

## Final artifact and gate evidence

- Canonical isolated staging now excludes generated surfaces only at the source root while preserving package-owned nested `dist` directories. Gateway wheels at source epochs 1700000000 and 1800000000 both hashed `252f8669199b9701f9246e20045009b4be58df8f625368927a591a9f5b7b3db2` (991 members).
- Profile wheels at both epochs both hashed `104cfdda58eeb3487a075426598a3548fbb7a7d16f50eb81cd8a47e0fc090347` (50 members).
- Both first wheels installed offline with `uv pip install --offline --no-deps --target`; isolated imports resolved `gateway` and `checkin_cli` from the installation targets. Every wheel member timestamp was `2000-01-01T00:00:00Z` (`SOURCE_DATE_EPOCH=946684800`, `TZ=UTC`).
- Standalone candidate verifier passed all 173 leaves and recomputed:
  - candidate `be8cca6c176c6986057b763386751ac8b831f81d8a98b1d1370f6c4564f56c63`
  - package `3090dc7ca9e93236394dd4da88647b100ea70b22f00080950a24e1d254f575e6`
  - manifest `6ae6af442c440d3b82ec5aa24d3ddd29f158542641e67810017d27e1df599190`
  - freeze canonical receipt `fb5bdd81e12ea9d32fe620eae160628f396196951d11a4c1da1822b45964dc1a`
  - status `f8a6731706568ea377f17d1511b6ce0372da3483e2b539a1b39dded8f7fed2dc`
  - diff `7d582012d93f73ba9133c89f72793537e4fb38e597683b51f3e06bbf56f419c7`
  - pre-adoption `c62fe20d005c64854bc8f9950c768e6b0595c519f4e1dca5824c7bbee2b3f04d`
- Ruff passed all 164 candidate Python leaves; strict Ty (`--error all --error-on-warning`) passed the established 14-file release scope; redirected compileall passed 164 leaves (212 bytecode outputs in auto-cleaned temp storage); tracked/cached diff hygiene passed.
- Because candidate, package, status, and diff identities are unchanged, the sealed gateway 1936 passed + 1 deselected, profile 720 passed, and focused 264 passed receipts remain exactly reuse-eligible. No affected canonical behavioral gate exists to rerun.

## Current-source successor provenance

The `252f866...` statement above is historical. Later launcher hardening changed the untracked current source `gateway/platforms/task22_trainer_authority_removal.py` from `6b89eaaa981a760436e8c510856bc6a13807a38c369ab363feb8f4a45d3978b2` (79,591 bytes) to `46a316f99d7bed10b9efd606ca519e0c151a3f373e433b24443c925b05d42602` (80,002 bytes). Because the path remained untracked, the exact status digest stayed unchanged even though its content changed.

- Wheel forensics `st_019ff207` replayed the same anchored complete-source command with source epochs 1700000000 and 1800000000.
- Historical source reproduced `252f8669199b9701f9246e20045009b4be58df8f625368927a591a9f5b7b3db2` twice.
- Current source reproduced `bc7136b8614fb766cfba6b6017b61ab532479790f4bf2c259899272ceafc1e0d` twice.
- Both wheels have the same 991-member order, fixed `2000-01-01T00:00:00Z` timestamps, `METADATA`, and `WHEEL`; only `gateway/platforms/task22_trainer_authority_removal.py` and `RECORD` differ.
- The current wheel installed offline and imported both `gateway` and `gateway.platforms.task22_trainer_authority_removal` from the isolated target.
- The profile wheel remains exactly `104cfdda58eeb3487a075426598a3548fbb7a7d16f50eb81cd8a47e0fc090347` with 50 members.

That prior supervised successor seal owns `bc7136...`; `252f866...` is retained only as explicit predecessor provenance.

## Recovery diagnostic reseal

- The guarded diagnostic established that recovery entered canonical-target validation while the removal lock set was already held; the first denied mutation was `mkdir(data, exist_ok=True)` from `profile_authority_lock`, and the 3,020-record profile snapshot remained unchanged.
- The causal fix pins recovery `4a2005c8f4faddc177cdd17940db5b1c0dd2780212ed03d01e248e51e2c25b6f` and removal `e29b0397f87e0f9274634a68294d1621b2ded09c189841647f199944990869eb`.
- Replaying the same anchored complete-source build in isolated fixed-epoch copies produced `fc4ca963176010769bb2bbd4aa2a22565a1c3234775aa97924b6f12916a7f59e` with 993 members and fixed ZIP timestamps; the no-dependency offline import check passed. `wheel_provenance.current_gateway.source_sha256` follows the active recovery entrypoint source, as the earlier provenance records followed the active removal source.