# F3 — adapter-guided repaint, and a coarse-label adapter Work package **F3** of the SGH programme (`operations/research/sgh-program-20260908/PLAN.md`), executed 2026-09-08 on spot A100 slot `d` (`sgh-a100-slot-d`, us-central1-f). Slots a, b and c were running other packages and were never touched. The VM was stopped at the end and is **TERMINATED**. Nothing was committed to git. **Engineering evidence only.** Everything below measures geometry, pixel statistics, throughput and one pathology encoder. **No clinical claim is made or implied, no pathologist has seen any of these images, and nothing here says a synthetic field is diagnostically usable.** --- ## 1. Headline Two follow-ups to F2's structure adapter, on the 20 a1 `across` cartoons (4 categories x seeds 11-15). | question | answer | |---|---| | **(1)** Does an adapter-guided img2img repaint keep the layout while reaching two-pass-si3 realism? | **Layout yes, realism no.** Layout IoU is *higher* than F2's from-noise (lumen 0.478-0.523 vs 0.466; ring 0.318-0.336 vs 0.257) against a two-pass si3 of 0.051/0.047, but the envelope distance D stays 0.2-2.8 worse than two-pass si3. The repaint improves D over F2 from-noise by 0.1-1.9 and no further | | **(2)** Does the `--scale 0` ablation show the adapter is doing the holding? | **Yes, decisively and paired.** Same cartoon, same seed, same start image, same donor, same 17 steps: with the adapter the layout IoU is +0.271 (ring) and +0.434 (lumen) on **20 of 20** images; without it the repaint lands on 0.045/0.051 — chance — and on the two-pass envelope. The cost of holding is +1.30 D on 16/20, +3400 nuclei per mm2 on 20/20 and -10.2 um2 of median nucleus area on 20/20 | | **(3)** Does the coarse (5-channel, no nucleus channels) adapter fix nuclear count and size? | **Yes.** Paired against F2's 9-channel adapter on the same cartoons: nuclear density 6655 vs 10070 per mm2 against a real 6376; median nucleus area 24.2 vs 17.6 um2 against a real 23.1. From +58% and -24% error down to +4% and +5%. It costs 15-21% of the layout IoU (lumen 0.395 vs 0.466, ring 0.202 vs 0.257) | | **(4)** Is any single configuration inside the real range on topology, within ~1 D of the two-pass-si3 envelope and above 0.25 layout IoU? | **Yes — the coarse adapter, from noise.** 11 of its 20 images clear all three on the `iou_lumen` instrument the brief names (3/5 normal, 3/5 gastritis, 2/5 IM, 3/5 mixed). No other arm measured here clears more than 7/20, and the three two-pass arms measured on the same cartoons (pass 1, si6, si3) clear 0/20 each. The other `sweep-v1` experiments (a2, a3, d1, d2, e2, b1) were not put on this bar | The single most useful sentence: **F2's adapter was over-conditioned.** The 9-channel one-hot at 2 um per latent cell told it where each nucleus goes, and it obeyed — which is why it controlled layout *and* why it made 1.4-2.1x too many nuclei, each 20-40% too small. Deleting the two nucleus channels (and the mucin/vessel/red-cell detail) removes the defect and keeps most of the control. ### Everything on one line, medians over the 20 cartoons | arm | ring | D | IoU lumen | IoU ring | nuc/mm2 | median nuc um2 | Laplacian | rings/mm2 | |---|---:|---:|---:|---:|---:|---:|---:|---:| | **real held-out** (n=22) | 0.841 | **3.51** | — | — | **6376** | **23.1** | **22.8** | **95** | | two-pass si3 (`sweep-v1`) | 0.481 | 5.77 | 0.051 | 0.047 | 5935 | 29.6 | 19.2 | 119 | | F2 adapter, 9-ch, from noise | 0.899 | 9.03 | 0.466 | 0.257 | 10070 | 17.6 | 19.3 | 204 | | A si6 donor | 0.897 | 7.51 | 0.510 | 0.319 | 9173 | 18.0 | 19.3 | 204 | | A si3 donor | 0.953 | 7.53 | 0.478 | 0.318 | 9311 | 17.8 | 18.7 | 201 | | A si3 shuffled tokens | 0.946 | 7.75 | 0.523 | 0.336 | 9394 | 17.6 | 18.7 | 199 | | A si3 **adapter OFF** | 0.568 | 5.44 | **0.051** | **0.045** | 6054 | 28.0 | 20.6 | 122 | | **B coarse, from noise** | 0.784 | **4.60** | 0.395 | 0.202 | **6655** | **24.2** | 19.9 | 166 | | **B coarse, si3 repaint** | 0.764 | **4.40** | 0.357 | 0.188 | **6351** | **25.8** | 18.2 | 171 | --- ## 2. What was built Everything is under `packages/f3/`. Total upload **0.4 MB of code**: the F2 run root was still on slot d's disk, so the 60 label maps, the segmented `data/`, F2's final checkpoint and F2's 64 from-noise PNGs were read in place and nothing heavy moved in either direction. ### 2.1 `adapter-train/infer_adapter.py` — `--start-image` and `--token-shuffle` - **`--start-image --start-index k`.** The image is VAE-encoded through `pixcell_generate.encode_source` (unchanged: tiling above 2 Mpx, `((mode - shift) * scale)`, cast fp16), noised to schedule step `k` with `scheduler.add_noise(..., timesteps[:1])` after `set_begin_index(k)`, and denoised from `timesteps[k:]` — the same construction `denoise_canvas` already used for `--init cartoon`, now able to start from *any* canvas-sized PNG. **The adapter keeps injecting the label window at every one of the remaining steps**, which is the whole point of the experiment and is asserted by the smoke test (`adapter_calls == denoiser_calls`, per injection point). - **`--token-shuffle`.** `shuffle_tokens(tokens, seed)` permutes the 16 UNI tokens inside each window with `numpy.random.default_rng(seed * 1000 + window_index)` — byte-for-byte the stream `pixcell_generate.patchbag_rng` / `shuffle_token_grids` uses, so an A4-style shuffle and this one are the same draw. The permutations are written into the provenance JSON. - **Provenance** now carries `adapter_sha256`, `start_image`, `start_image_sha256`, `start_index`, `token_shuffle`, `token_shuffle_seed_base`, `token_shuffle_permutations`, `label_groups` and `output_sha256`. - **From-noise behaviour is unchanged when the flags are absent** — the smoke test asserts the latent is bit-identical to the pre-F3 code at the same seed. ### 2.2 `pair_dataset.py` / `train_pairs_adapter.py` — `--label-groups` `parse_label_groups` accepts a preset (`coarse5`) or an explicit partition (`0|1|2,3,4|5,6|7,8`) and *validates that it is a partition* of the 9 tissue3d ids, so the conditioning channels still sum to 1 in every latent cell. The merge happens inside `label_one_hot_latent` when the one-hot is area-pooled to the 128x128 latent grid, so **no label PNG on disk is rewritten** and `groups=None` reproduces F2's 9-channel tensor exactly (asserted). `coarse5` is | channel | tissue3d ids | name | |---|---|---| | 0 | 0 | background above surface | | 1 | 1 | lumen | | 2 | 2, 3, 4 | epithelium (cytoplasm + goblet mucin + epithelial nucleus) | | 3 | 5, 6 | stroma (+ stromal nucleus) | | 4 | 7, 8 | vessel + red cell | The grouping is written into the checkpoint (`label_groups`, `label_groups_spec`, `label_group_names`, `cond_channels`), and `infer_adapter.py` / `infer_jobs.py` rebuild the identical conditioning from the checkpoint rather than from a flag — so an adapter cannot be run against the wrong conditioning by mistake (a channel-count mismatch is a hard error). ### 2.3 `adapter-train/infer_jobs.py` — one process, many generations (new) F2-TRAIN's planner spawned one `infer_adapter.py` per image and paid **79 s wall for 46.3 s of generation** — about 35 minutes of A100 spent re-reading weights. This driver loads PixCell, the SD3 VAE and UNI2-h once and loops, and caches - adapter checkpoints by path, - **UNI tokens by donor field** — one 4096x2048 donor is 21 windows x 16 crops and costs 8.4-8.6 s (measured here, matching SWEEP's 8.3 s), and part A reuses each donor across four arms, - the per-window label conditioning by (label map, rotate180, grouping). It writes the same `.png` + `.png.json` pair, one record per job in a `--records` jsonl, and **skips a job whose output PNG already exists and is non-empty**, so a preemption resumes where it stopped. The model is loaded lazily, on the first job that actually has to run, so a fully-resumed run loads nothing. Measured effect: part A cost **48.1 s per image all-in** against F2's 79 s — 41 minutes and about USD 1.4 saved on this package alone. ### 2.4 `plan_jobs.py` and `worker.sh` `plan_jobs.py` writes the three job manifests on the VM (every path absolute there), resolves the donor as `REFS_ROOT//` with the A1 rotation `(seed - 11) mod 5`, and **exits 2 with `MISSING_INPUTS` if any label map, donor or start image is absent** — the FLEET gotcha-8 failure (a step that "succeeds" having done nothing) made explicit. `worker.sh` runs GPU check -> input check -> scipy guard -> CPU smoke -> plan A -> part A -> part B training -> plan B -> part B from-noise -> part B si3, with F2's evidence pattern (`steps.log`, `exit_code.txt`, `SHA256SUMS`, `WORKER_DONE`) and the 20-minute self-stop under unit `sgh-fleet-stop` with the stop / `reset-failed` / re-arm dance from FLEET gotcha 4 and SWEEP fix 3. F2's `data/` is reached through a symlink because the manifest stores label paths *relative* to its run root. ### 2.5 CPU smoke test — 56 checks, 0 failures, on the Mac and again on the VM `run_cpu_smoke.py` keeps F2's 27 checks (so a regression in the shared code is still caught) and adds 29. It ran **56 passed / 0 failed in 5 s** on the Mac (Python 3.14) before staging and **56 passed in 32 s** on the VM (Python 3.12) before any GPU work. The new sections: ``` 6 coarse label groups [PASS] coarse5 is a 5-way partition of the 9 tissue3d labels [PASS] a non-partition grouping is rejected [PASS] coarse conditioning is (5,128,128) [PASS] each coarse channel is the sum of its fine channels -- max |d| 0.000e+00 [PASS] coarse channels are still a partition (sum == 1 per latent cell) [PASS] the nucleus channels are gone (4 and 6 folded into epithelium/stroma) -- 5 vs 9 [PASS] groups=None reproduces the F2 9-channel one-hot exactly [PASS] a coarse-conditioned adapter trains -- {"steps": 2, "cond_channels": 5} [PASS] the checkpoint records the grouping -- [[0],[1],[2,3,4],[5,6],[7,8]] [PASS] the reloaded adapter takes 5 mask channels [PASS] infer rebuilds the SAME conditioning from the checkpoint's grouping 7 --start-image, --token-shuffle, --scale 0 [PASS] the start image encodes to the canvas latent -- (1, 16, 128, 192) [PASS] start_index 6 of 20 runs 14 steps -- t0=699 [PASS] start_index 3 of 20 runs 17 steps (heavier repaint) [PASS] the adapter is injected at EVERY step of the repaint, not only the first -- [14,14] vs 14 [PASS] an img2img start changes the result vs from noise [PASS] from-noise is bit-identical to before the F3 change (same seed, no new flags) [PASS] the shuffle stream is default_rng(seed*1000 + window_index) [PASS] every window gets its OWN permutation [PASS] shuffling permutes the 16 tokens and invents none [PASS] shuffled tokens change the output [PASS] --scale 0 is a real ablation: no residual is added at all -- [0, 0] [PASS] --scale 0 changes the image (so arm 4 is not arm 2 under another name) 8 infer_jobs single-process driver [PASS] the driver ran every job [PASS] the driver loaded the model ONCE and the donor tokens once [PASS] provenance carries the adapter sha, start-image sha, start_index and permutation [PASS] provenance records the coarse grouping the adapter was trained with [PASS] the scale-0 arm is recorded as scale 0 and differs from the scale-1 arm [PASS] a job whose PNG already exists is skipped, not regenerated ``` `--scale 0` is a genuine ablation, not a small residual: `StructureAdapterAttachment._residual` returns `None` when `scale == 0.0`, so the injection hook adds nothing at all and the reported injection counts are `[0, 0]`. Measured on the GPU, it also runs 0.28 s faster per 4096x2048 canvas (38.92 s vs 39.20 s). --- ## 3. Run timeline (UTC, 2026-09-08) | time | event | |---|---| | 18:56:19 | `fleet.sh up d` — spot A100 acquired first attempt, guest deadline +32400 s | | 18:57:45 | ssh ready, `NVIDIA A100-SXM4-40GB, 40960 MiB, driver 580.173.02` (86 s from start) | | 18:58:16-18:58:48 | `fleet.sh stage d ../packages/f3` — 5 items, 12 files, **32 s** (0.4 MB of code) | | 18:59:01 | launch (returned in 8 s) | | 18:59:11 | inputs verified: 64 F2 from-noise PNGs, 60 label maps, `data/manifest.json`, `latest.txt`, refs | | 18:59:11 | `scipy 1.17.1 skimage 0.26.0 numpy 1.26.4` — **F2-TRAIN's repair survived on this disk**, so the guard installed nothing | | 18:59:11-18:59:43 | CPU smoke on the VM: **56 passed, 0 failed, 32 s** | | 18:59:43 | part A planned: 80 jobs, `all inputs present` | | 19:00:06-19:01:17 | PixCell + UNI2-h loaded once; first image | | 20:03:50 | **part A done: `jobs=80 ok=80 skipped=0 failed=0 seconds=3845`** | | 20:03:52 | part B training starts (coarse 5-channel adapter) | | 20:04:14 | 64 train / 32 val fields, slides held out `['hiesd-324885ef','hiesd-9ded7e41','hiesd-da8667dc']` — **the same split F2 used** | | 20:04:43 | `validation loss @ step 0: 0.1641` — **bit-identical to F2's step 0** (see 5.1) | | 22:20:53 | step 3000, `validation loss 0.1612`, checkpoint `step_003000` | | 22:20:54 | `done: 3000 steps in 8198s` (2 h 17 m, 2.73 s/step) | | 22:20:55-22:45:10 | part B from-noise: **`jobs=24 ok=24 skipped=0 failed=0 seconds=1453`** | | 22:45:11-23:04:22 | part B si3 repaint: **`jobs=20 ok=20 skipped=0 failed=0 seconds=1149`** | | 23:04:23 | `WORKER_DONE`, `exit_code.txt` = 0, **131 PNGs**, `out/SHA256SUMS` (149 files) | | 23:05:37 | the 20-minute self-stop timer stopped by hand so it could not race a 2 GB retrieve | | 23:05:42-23:20:53 | `fleet.sh retrieve d …/packages/f3/retrieved` — 2.0 GB, 911 s, **SHA256SUMS OK (149 files)**, 131 PNGs | | 23:26:38-23:27:34 | `fleet.sh down d`; `fleet.sh status` confirms **TERMINATED** | **Preemptions: none** in 4 h 30 m, so the resume path was again not exercised by this package. **Failures: none.** 124 of 124 planned generations landed, no job was skipped and no job failed. One optimisation worth recording: at 20:08, while part B was training, the 80 part-A PNGs and the evidence directory were pulled with a targeted `tar`-over-ssh (1.1 GB in 4 min, GPU untouched), so the whole part-A evaluation ran during the 2 h 17 m training window instead of after it. It cost about USD 0.13 of duplicated egress. ### Cost and powered time From `fleet/state/d/powered.log` (`UP 1788893822` -> `DOWN 1788910054`). | | | |---|---| | powered window | 2026-09-08 **18:57:02Z -> 23:27:34Z** | | **powered minutes** | **270.5 min (4.509 h)**, one session, zero preemptions | | spot A100 (a2-highgpu-1g) @ USD 2.12/h | **USD 9.56** | | egress, 3.1 GB to the Mac @ ~USD 0.12/GB (2.0 GB retrieve + the 1.1 GB early part-A pull) | ~USD 0.37 | | **total attributable to this package** | **~USD 9.93** | Where the 270 minutes went: 3 boot/stage/smoke, **64 part A**, **137 part B training**, **41 part B generation**, 15 retrieve, 1 shutdown, ~9 of orchestration slack. GPU-busy fraction 89%. VRAM peaked at **6.86 GB of 40 GB** during training (`nvidia-smi` 7658 MiB), so batch 16 would still have fitted; the run was encoder-bound, as F2's was. --- ## 4. What was generated **Part A — adapter-guided pass 2.** The 20 a1 `across` cartoons, F2's final checkpoint (`structure_adapter.safetensors` sha `d3fc7a86f6…`), start image = F2's own from-noise output for that cartoon (**all 80 records' `start_image_sha256` verified equal to the sha of the retrieved F2 PNG**), donor = that category's donor `(seed - 11) mod 5` from `donors.json` resolved under `REFS_ROOT`, which is the A1 sweep's own rotation — so these outputs are **donor-matched with `sweep-v1`'s a1 arms on the same cartoons**, which F2-TRAIN's comparison was not. | arm | start_index | steps run | seed | tokens | adapter | s/image (median) | |---|---|---|---|---|---|---| | `A_si6_donor` | 6 | 14 (84 denoiser calls) | 51002 | donor | scale 1.0 | 32.27 | | `A_si3_donor` | 3 | 17 (102 calls) | 51003 | donor | scale 1.0 | 39.20 | | `A_si3_shuffled` | 3 | 17 | 51003 | donor, 16 tokens permuted per window | scale 1.0 | 39.20 | | `A_si3_adapter_off` | 3 | 17 | 51003 | donor | **scale 0.0** | 38.92 | **Part B — coarse-label adapter.** `cond=labels`, `label_groups=coarse5`, everything else exactly F2's config (3000 steps, batch 4 at 1024, lr 1e-4 cosine + 100 warmup, wd 0.01, clip 1.0, CFG dropout 0.10, conditioning dropout 0.10, bf16, gradient checkpointing, checkpoint every 500, validate every 250, seed 0, `val_split by-slide`). Adapter `width=320 blocks=4 injections=2` at layers [0, 14] of 28 = **8.20M trainable parameters** (F2's 9-channel one was 8.25M; the difference is the 4 fewer input channels). `UNI check: max abs diff 0.000e+00` against `pixcell_generate.uni_tokens`, as in F2. Then, with checkpoint sha `e49b6e1f02…`: from-noise on the same 20 cartoons plus one `--rotate180` per category (seed 51001, the seed F2 used, 20 steps / 120 calls, 45.84 s median), and the adapter-guided si3 repaint of those 20 outputs (seed 51003, start_index 3, 102 calls, 38.99 s median). --- ## 5. Part B training ### 5.1 An instrument check that came free The validation loss at step 0 is **0.1641 — the same four decimals F2 recorded**. It has to be: at step 0 the zero-init heads contribute nothing, so the conditioning cannot matter, and the crops, the noise and the timesteps are fixed (`linspace(50, 950)`, `manual_seed(1234 + batch)`). Getting the same number confirms the by-slide split picked the same three slides, the same 32 val fields and the same 128 fixed crops, so **the two runs' validation curves are directly comparable**. ### 5.2 The curve | step | F2, 9 channels | **B, 5 channels** | |---:|---:|---:| | 0 | 0.1641 | **0.1641** | | 250 | 0.1611 | 0.1619 | | 500 | 0.1604 | 0.1616 | | 750 | 0.1601 | 0.1617 | | 1000 | 0.1598 | 0.1615 | | 1250 | 0.1598 | 0.1614 | | 1500 | 0.1596 | 0.1613 | | 1750 | 0.1595 | 0.1614 | | 2000 | 0.1595 | 0.1612 | | 2250 | 0.1595 | 0.1612 | | 2500 | 0.1595 | 0.1612 | | 2750 | 0.1594 | 0.1612 | | 3000 | **0.1594** (-2.86%) | **0.1612** (-1.77%) | `grad_norm` median 0.0117, min 0.0029, max 0.2483 over all 3000 steps — never zero, so failure criterion 3 does not apply here either. Throughput 2.73 s/step steady state, 8198 s for 3000 steps. **Read honestly: the coarse adapter learns about 62% of what the 9-channel adapter learned on the epsilon-MSE, and it plateaus earlier (flat from step ~1000, against F2's ~1500).** The four channels that were removed — epithelial nucleus, stromal nucleus, goblet mucin and red cell — carry 38% of the total learnable validation gain. That is exactly what you would expect if most of what the fine adapter learned was *where to put nuclei*, and it is the loss-curve counterpart of section 7's morphometry: the fine adapter's extra "knowledge" was the defect. --- ## 6. Part A: does an adapter-guided repaint hold the layout? All numbers are medians over the 20 cartoons unless a table says otherwise; the full tables are `packages/f3/eval/tables.md` and the per-image rows are `features.csv` / `layout-iou.csv`. ### 6.1 Two layout instruments, both reported The programme has been quoting two different numbers under the name "layout IoU" and F3 has to be comparable with both, so every arm carries both: - **`iou_lumen`** — EVAL-RUN's `a1-layout-iou`: `evaluate_sweep.analysis_masks(...)["lumen"]` (pale components at least the sydney minimum lumen area that are not border-connected background) against `label_mask(..., 1)`. Reference ladder on these cartoons: pass 1 0.493, si6 0.026, si3 0.051; the **cartoon's own self-IoU (the ceiling) is 0.709**. - **`iou_ring`** — F2-TRAIN's T3/T4 instrument: `f2_eval.ring_mask` (the accepted ring interiors — holes of the closed hematoxylin band inside the sydney ring-area gate) against the same label lumen mask. Reference: F2 adapter 0.26-0.56, two-pass 0.10-0.13, chance 0.03-0.06. On these 20 cartoons the cartoon's own self-IoU is only **0.211**, so this instrument is *not* bounded by the cartoon and a value above 0.21 is possible (A gastritis reaches 0.464). **Instrument self-check.** Recomputing F2-TRAIN's T4 rotation table with this code reproduces its published numbers to four decimals — gastritis 0.4599 / 0.0417, IM 0.2173 / 0.0397, mixed 0.2634 / 0.0389, normal 0.3494 / 0.0283, and the unrotated column 0.5170 / 0.1982 / 0.2566 / 0.3519. The morphometric side reproduces C1's held-out Mahalanobis distances to max |delta| 4.9e-06. Nothing was re-fitted. **A caveat EVAL-RUN had to carry does not apply here.** EVAL-RUN measured layout IoU against the *Mac* label maps while the sweep's generator saw *VM-rendered* cartoons. F2 uploaded its label maps, so F2 and F3 were conditioned on exactly the bytes measured here — verified by sha256 (`normal_s11_across_labels.png` `7da4ac55…` on the Mac and on the VM). The two-pass reference rows still carry the original caveat. ### 6.2 Layout, per category `iou_lumen`, mean per category, with two independent chance levels (the same map rotated 180, and a different cartoon of the same category): | arm | normal | gastritis | IM | mixed | |---|---|---|---|---| | a1 cartoon itself (ceiling) | 0.858 | 0.841 | 0.409 | 0.549 | | two-pass pass 1 (si12) | 0.628 | 0.659 | 0.347 | 0.462 | | two-pass si3 | 0.054 | 0.057 | 0.048 | 0.053 | | F2 adapter from-noise | 0.703 | 0.466 | 0.371 | 0.468 | | **A si6 donor** | 0.602 | 0.621 | 0.306 | 0.461 | | **A si3 donor** | 0.553 | 0.540 | 0.362 | 0.486 | | **A si3 shuffled** | 0.602 | 0.569 | 0.385 | 0.511 | | **A si3 adapter OFF** | **0.054** | **0.062** | **0.045** | **0.051** | | chance (rot / other), all arms | 0.02-0.06 | 0.00-0.06 | 0.02-0.04 | 0.02-0.05 | `iou_ring`: | arm | normal | gastritis | IM | mixed | |---|---|---|---|---| | a1 cartoon itself | 0.264 | 0.368 | 0.188 | 0.210 | | two-pass si3 | 0.052 | 0.047 | 0.042 | 0.051 | | F2 adapter from-noise | 0.319 | 0.495 | 0.212 | 0.252 | | **A si6 donor** | 0.345 | 0.464 | 0.285 | 0.286 | | **A si3 donor** | 0.342 | 0.453 | 0.275 | 0.283 | | **A si3 shuffled** | 0.356 | 0.478 | 0.300 | 0.298 | | **A si3 adapter OFF** | **0.047** | **0.044** | **0.039** | **0.048** | A heavy repaint with the adapter active **keeps and in three of four categories slightly improves** the layout F2's from-noise pass established: on `iou_ring` the A arms are 0.28-0.48 against F2's 0.21-0.50, with IM going 0.212 -> 0.275-0.300 and mixed 0.252 -> 0.283-0.298. Only gastritis loses a little (0.495 -> 0.453-0.478). The same repaint with the adapter switched off lands on **chance**. ### 6.3 The ablation, fully paired Same cartoon, same seed 51003, same start image, same donor, same 17 steps — the only difference is `--scale 1` vs `--scale 0`. Paired over all 20 cartoons: | quantity | adapter ON | adapter OFF | median paired delta | sign consistency | |---|---:|---:|---:|---| | `iou_lumen` vs own label map | 0.478 | 0.051 | **+0.434** | **20/20** | | `iou_ring` vs own label map | 0.318 | 0.045 | **+0.271** | **20/20** | | envelope distance D | 7.53 | 5.44 | **+1.297** | 16/20 | | nuclei per mm2 | 9311 | 6054 | **+3400** | **20/20** | | median nucleus area um2 | 17.8 | 28.0 | **-10.2** | **20/20** | | mean absolute Laplacian | 18.7 | 20.6 | -1.93 | 20/20 | | within-arm pairwise SSIM | 0.26-0.33 | 0.50-0.58 | +0.22 | all 4 categories | Three independent lines say the same thing. (a) The layout is *entirely* the adapter's doing: without it a 17-step repaint of an adapter-made canvas erases the geometry back to chance, and back to exactly the two-pass si3 number (0.051 / 0.045 vs 0.051 / 0.047). (b) The adapter-off arm also lands on the two-pass si3 *envelope* (paired delta -0.02 to -0.51 D, 3-4 of 5 closer per category), so "adapter off" is not a broken arm — it is the two-pass recipe reproduced from a different pass-1, which is the strongest available check that the ablation changed one thing only. (c) The adapter also preserves **diversity**: five different cartoons stay distinct with it on (SSIM 0.26-0.33, matching F2's 0.25-0.33) and collapse toward one donor-determined appearance with it off (0.50-0.58, matching two-pass si3's 0.49-0.57), which is EVAL-RUN's finding (c) — "by pass 2 the cartoon is gone and the donor decides the canvas" — reproduced and then defeated. And the cost is in the same table: **holding the layout costs +1.3 D, +3400 nuclei per mm2 and -10 um2 of nucleus size.** The adapter is not merely failing to fix the nuclei during the repaint; it is actively re-stamping them at every one of the 17 steps. ### 6.4 The two nulls in part A - **Token shuffling does nothing.** Permuting each window's 16 UNI tokens changes D by +0.24 (median), `iou_ring` by +0.02, Laplacian by +0.03 and the ring fraction by 0.000. Whatever the 16 tokens per window encode about *where* appearance goes inside that window, this pipeline does not use it — which is consistent with the tokens entering as an unordered `encoder_hidden_states` sequence for cross-attention. - **Repaint depth barely matters once the adapter is on.** si6 (14 steps) vs si3 (17 steps): D -0.15, `iou_ring` +0.005, Laplacian +0.42. The adapter re-imposes its conditioning at every step, so more steps do not buy the repaint more freedom. (The one place they differ is the ring fraction for IM and mixed, where si6 is 0.518 / 0.619 and si3 is 0.896 / 0.859.) ### 6.5 Realism: the repaint helps, and stops well short Paired on the same cartoon, median delta in D (negative = closer to the real envelope): | arm | vs F2 from-noise | vs two-pass si3 | |---|---|---| | A si6 donor | -0.12 / -1.80 / -0.46 / -1.87 | +1.20 / +2.27 / +0.27 / +1.64 | | A si3 donor | -0.70 / -1.42 / -0.60 / -1.76 | +0.63 / +2.65 / +0.97 / +2.84 | | A si3 shuffled | -1.13 / -1.66 / -0.36 / -1.51 | +0.20 / +2.74 / +1.07 / +2.74 | | A si3 adapter OFF | -0.87 / -4.74 / -0.78 / -3.00 | -0.02 / -0.51 / -0.11 / -0.04 | (normal / gastritis / IM / mixed.) So **the answer to question (1) is: layout yes, realism no.** The repaint moves the adapter's fields toward the real envelope in every category — by up to 1.9 D — but leaves them 0.2 to 2.8 D short of the two-pass si3 arm, and the shortfall is entirely the nuclei (T5: 8506-10674 per mm2 against a real 5996-6762, median area 14.2-19.6 um2 against a real 21.8-25.1). Ring fraction also *overshoots harder* after the repaint than before (0.86-0.98 against a real 0.71-0.85), so "in the real range" is satisfied for gastritis and mixed and not for normal. --- ## 7. Part B: does dropping the nucleus channels fix the nuclei? Paired against F2's 9-channel adapter, both from noise, on the same 20 cartoons: | quantity | **B, 5 channels** | F2, 9 channels | real held-out | B error | F2 error | |---|---:|---:|---:|---:|---:| | nuclei per mm2 | **6655** | 10070 | 6376 | **+4%** | +58% | | median nucleus area um2 | **24.2** | 17.6 | 23.1 | **+5%** | -24% | | epithelial nuclei per mm2 | **8076** | 12645 | 7302 | **+11%** | +73% | | rings per mm2 | 166 | 204 | 95 | +74% | +115% | | ring_with_lumen_fraction | **0.784** | 0.899 | 0.841 | -0.06 | +0.06 | | envelope distance D | **4.60** | 9.03 | 3.51 | +1.09 | +5.52 | | mean absolute Laplacian | 19.9 | 19.3 | 22.8 | -2.9 | -3.5 | | `iou_lumen` | 0.395 | 0.466 | — | | | | `iou_ring` | 0.202 | 0.257 | — | | | **Yes — the coarse adapter fixes nuclear count and size**, and the fix is large: the single systematic defect F2-TRAIN identified ("1.4x to 2.1x too many nuclei, each 20-40% too small") is gone. The envelope distance halves, from a median 9.03 to 4.60 against a real 3.51, and it does so in **every** category (T2: normal 8.51 -> 4.89, gastritis 10.21 -> 4.34, IM 5.28 -> 3.66, mixed 9.22 -> 5.04). Against the *two-pass si3* arm — the realism reference — the coarse adapter is now **closer to the real envelope in all four categories** (paired median -1.64 / -1.57 / -0.57 / -0.85, 4-5 of 5 cartoons per category), which is the condition F2's section 8 asked for and got only for IM. What it costs: 15-21% of the layout signal (`iou_lumen` 0.395 vs 0.466, `iou_ring` 0.202 vs 0.257 — still 4-8x the two-pass arm's 0.05, and 6-8x chance), and a loss of within-arm diversity (mean pairwise SSIM 0.45-0.51 against F2's 0.25-0.33 and a real floor of 0.03-0.05) — note the B arm's five images per category use five *different* donors and are still more alike than F2's five, which all share one donor, so this is the coarse conditioning being less discriminative between cartoons, not a donor artefact. Ring density is improved but still 1.7x real. **Rotation test, coarse adapter** (one cartoon per category, the conditioning rotated 180 and nothing else): | category | instrument | vs rotated map | vs unrotated map (chance) | ratio | |---|---|---:|---:|---:| | normal | `iou_lumen` | 0.679 | 0.044 | **15.5x** | | normal | `iou_ring` | 0.239 | 0.028 | **8.4x** | | gastritis | `iou_lumen` | 0.330 | 0.065 | **5.0x** | | gastritis | `iou_ring` | 0.356 | 0.031 | **11.5x** | | IM | `iou_lumen` | 0.309 | 0.040 | **7.7x** | | IM | `iou_ring` | 0.106 | 0.039 | 2.7x | | mixed | `iou_lumen` | 0.332 | 0.040 | **8.2x** | | mixed | `iou_ring` | 0.144 | 0.046 | 3.1x | The output rotates with the conditioning in all four categories (2.7x-15.5x separation, against F2's 5.5x-17.8x on the same measurement). H04's failure mode does not reproduce with the coarse conditioning either, though IM and mixed are visibly weaker on the ring instrument. **The si3 repaint on top of the coarse adapter is a wash** (paired, 20 cartoons): D +0.05, nuclei -307 per mm2, median nucleus area +1.3 um2, Laplacian -1.75, `iou_lumen` -0.01, `iou_ring` -0.01. It buys nothing the coarse from-noise pass did not already have. On the medians it edges ahead on D (4.40 vs 4.60) and behind on detail; there is no reason to pay 39 s per image for it. --- ## 8. Copy screens phikon-v2 embedding screen (1024 px tiles, fail line 0.95) against **all 118 real fields**, plus the pixel NCC search (64 px windows, stride 256, threshold 0.92) against each output's own donor and its phikon-nearest reference fields — the harness's `targeted` policy, which is what EVAL-RUN used. | set / arm | n | max cosine | median max cosine | own-source frac | tiles >= 0.95 | NCC pairs | max NCC | windows >= 0.92 | COPIED | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | two-pass si3 | 20 | 0.905 | 0.867 | 1.000 | 0 | 28 | 0.825 | 0 | 0 | | F2 from-noise | 20 | 0.833 | 0.798 | 0.375 | 0 | 44 | 0.599 | 0 | 0 | | A si6 donor | 20 | 0.857 | 0.807 | 0.500 | 0 | 44 | 0.655 | 0 | 0 | | A si3 donor | 20 | 0.855 | 0.805 | 0.375 | 0 | 47 | 0.655 | 0 | 0 | | A si3 shuffled | 20 | 0.824 | 0.794 | 0.250 | 0 | 49 | 0.710 | 0 | 0 | | A si3 adapter OFF | 20 | 0.922 | 0.877 | 1.000 | 0 | 26 | 0.815 | 0 | 0 | | B coarse from-noise | 20 | 0.928 | 0.896 | 1.000 | 0 | 29 | 0.741 | 0 | 0 | | B coarse si3 | 20 | **0.937** | 0.898 | 1.000 | 0 | 28 | 0.836 | 0 | 0 | **Nothing is flagged: 0 tiles at or above the 0.95 embedding fail line and 0 NCC windows at or above 0.92, in 200 candidates and 372 pairs.** The calibration for reference is: a recoloured copy scores 0.987 (embed) / 0.9995 (NCC), a generated field against its own reference 0.83 / 0.85, unrelated real fields 0.66 / 0.88. Two honest readings. First, the arms whose appearance is donor-dominated — two-pass si3, A adapter-off, and both B arms — have **own-source fraction 1.000**: every 1024 tile's nearest neighbour among 118 real fields is its own token donor. For B this is worth stating precisely, because B is generated **from noise** and never saw the donor's pixels, only its 16 UNI tokens per window; the tokens *are* the donor's appearance, so "nearest to its own donor" is the expected behaviour, not evidence of copying, and the pixel NCC (0.741) is far below the copy level. Second, the margin to the fail line is smallest for the B arms (0.937 max, margin 0.013) and largest for the A/F2 adapter arms (0.824-0.857). The coarse adapter buys its realism partly by leaning harder on the donor's appearance statistics, and a future run should watch that margin. --- ## 9. Contact sheets — read with the Read tool and described honestly `packages/f3/eval/sheets/f3-.png` — one row per cartoon seed (s11, s12, s13), ten columns: *cartoon | F2 from-noise | A si6 donor | A si3 donor | A si3 shuffled | A si3 adapter-off | B from-noise | B si3 | two-pass si3 | real held-out*. Every cell is the **same** 768 px window of its 4096x2048 canvas (`evaluate_sweep.CROP_768` = (1664, 640)-(2432, 1408)), so a difference between two cells is a difference between images, not between crops; the real held-out column is a different field, so it is the same rectangle of a different slide. Each cell is labelled `ring / D / lap / IoU_lumen / IoU_ring`. `packages/f3/eval/sheets/f3-zoom-.png` is the same nine sources plus real at **1:1 pixels**, 512 px = 128 um, because a 768 px crop scaled into a sheet cell cannot show whether a nucleus has chromatin. `sheets-small/` is the field-scale sheet at 230 px cells, which is the readable one for layout. **What I see.** At **field scale** the layout claim is visible without any measurement. Across a row — same cartoon, different arm — F2's from-noise column and the three adapter-on A columns put their pale spaces in the same places, and those places track the cartoon's white slots; the *adapter-off* column and the two-pass si3 column are a completely different field with no relation to the cartoon, and (in `mixed` rows 2 and 3) they resemble each other far more than either resembles the cartoon. The B columns sit in between: they carry the cartoon's gland arrangement — in `mixed` s11 and s13 there is a long tall-columnar epithelial band with an open lumen where the cartoon put one — while looking much more like tissue than the A columns do. At **1:1 pixels** the trade is unmistakable. The A and F2 crops are a dense, fairly uniform scatter of small round soft-edged nuclei on washed pink cytoplasm with little internal texture; the epithelium is not organised into a polarised band and the pale holes are walled by a crowd of small nuclei rather than a palisade. That is F2-TRAIN's description and the repaint does not change it. The **adapter-off** and **two-pass si3** crops are the other extreme: large elongated nuclei with visible chromatin, an organised epithelial band, dense eosinophilic stroma — convincing tissue that ignores the label map. The **B coarse** crops are the first column in this programme that looks like both: in gastritis s11 the B from-noise crop has an epithelial surface with basally-oriented elongated nuclei and a red blood cell, at layout IoU 0.348 against the adapter-off crop's 0.064; in IM s11 the B crops have large well-formed nuclei with chromatin and a dark epithelial cluster, at IoU 0.332 / 0.312 against two-pass si3's 0.041. They are still short of the real held-out column, which has speckled chromatin, crisper nuclear membranes, red cells in vessels and a stronger stain contrast, and whose Laplacian is 22.8 against B's 19.9. In `normal` the A arms look plausible as oxyntic mucosa (pale parietal-like cells, small round nuclei) and that is the category where their D penalty is smallest. --- ## 10. Verdicts **(1) Does the adapter-guided repaint keep the layout while reaching two-pass-si3 realism?** **Half.** Layout is kept and slightly improved: `iou_ring` 0.318-0.336 and `iou_lumen` 0.478-0.523, against F2's from-noise 0.257 / 0.466 and the two-pass si3's 0.047 / 0.051, with two independent chance levels at 0.03-0.06. Realism is not reached: D stays 0.2-2.8 above the two-pass si3 arm in every category, and the whole gap is the nuclei. The repaint moves the adapter's fields 0.1-1.9 D toward real and then stops, because the adapter re-imposes its own conditioning at every step of the repaint. **(2) Does the ablation show the adapter is doing the holding?** **Yes, unambiguously.** Paired on 20 images with only `--scale` changed: +0.434 `iou_lumen` and +0.271 `iou_ring`, 20/20 both; with the adapter off the repaint lands on chance and on the two-pass envelope simultaneously. The ablation also localises the *cost*: +1.30 D (16/20), +3400 nuclei per mm2 (20/20), -10.2 um2 median nucleus area (20/20). This is the `--scale 0` vs `--scale 1` control F2-TRAIN could not afford to run, and it is now run. **(3) Does the coarse adapter fix nuclear count and size?** **Yes.** Nuclear density 6655 per mm2 against a real 6376 (F2: 10070); median nucleus area 24.2 um2 against a real 23.1 (F2: 17.6); epithelial nuclear density 8076 against 7302 (F2: 12645). Envelope distance halves from 9.03 to 4.60 against a real 3.51, in all four categories, and it beats the two-pass si3 arm on the same cartoons in all four. The price is about a quarter of the layout IoU, a loss of within-arm diversity (SSIM 0.45-0.51 vs F2's 0.25-0.33), a ring density still 1.7x real, and an embedding-screen margin that narrows to 0.013. **(4) Which single configuration, if any, is inside the real range on topology, within ~1 D of the two-pass-si3 envelope and above 0.25 layout IoU?** **The coarse adapter run from noise** (`B_fromnoise`, seed 51001, adapter `out/adapter-b/checkpoints/step_003000`, sha `e49b6e1f02…`, donor = the A1 seed rotation). On the `iou_lumen` instrument the brief names it clears all three conditions on **11 of 20** images, spread over all four categories (normal 3/5, gastritis 3/5, IM 2/5, mixed 3/5); its coarse-si3 repaint clears the same 11; F2's 9-channel adapter clears 7/20 (5 of them IM); the A repaint arms clear 3-5/20 and the adapter-off arm 0/20; and the three **two-pass arms measured on these cartoons (pass 1, si6, si3) clear 0/20 each**. On the stricter `iou_ring > 0.25` reading the same arm clears 6/20 (normal 3/5, gastritis 3/5, IM 0/5, mixed 0/5), because B's ring-mask IoU for IM (0.149) and mixed (0.173) sits below the bar — so the answer is instrument-dependent for IM and mixed and holds for normal and gastritis either way. --- ## 11. What this package did not do - **No pathologist review and no clinical claim of any kind.** Ring fraction, envelope distance, mask IoU, Laplacian, SSIM and one encoder's cosine are engineering measures of geometry and pixel statistics. - **The coarse grouping was not swept.** `coarse5` is one point. The obvious next probes are 6 channels (mucin split back out) and 4 (vessel folded into stroma), and a nucleus channel at *lower* spatial resolution rather than none. - **One training configuration per adapter.** No sweep over width, injection depth, step count or conditioning dropout; no `cond=cartoon` or `labels+cartoon` arm. - **`across` cut only, 20 cartoons.** F2 measured 60 cartoons over three cuts. EVAL-RUN found `oblique` keeps lumens best and `across` worst at pass 2, so this package measured the harder cut; the numbers are not directly comparable with F2-TRAIN's all-cut tables and the report quotes the `across` subset of those tables wherever a comparison is made. - **The preemption/resume path was not exercised** — no preemption in 4 h 30 m. `infer_jobs.py`'s resume is covered by the smoke test only. - **The mucin/goblet channel still has no dedicated measurement**, as F2_RESULT section 8 asked; the coarse grouping folds it into epithelium and the sheets are the only evidence about it. - **B's within-arm diversity was not chased down.** It is measurably lower than F2's and the cause is not established. - Nothing was committed to git. --- ## 12. Files ``` packages/f3/worker.sh part A -> part B training -> part B generation packages/f3/plan_jobs.py the three job manifests, built on the VM, with MISSING_INPUTS packages/f3/STAGE_LIST code only (0.4 MB); everything heavy is F2's, read in place packages/f3/code/pixcell_generate.py the canonical generator, byte-identical to F2's (sha 64d6103a…) packages/f3/adapter-train/infer_adapter.py + --start-image / --start-index / --token-shuffle packages/f3/adapter-train/infer_jobs.py NEW: one process, model + tokens + conditioning cached packages/f3/adapter-train/pair_dataset.py + parse_label_groups / label_group_names, grouped one-hot packages/f3/adapter-train/train_pairs_adapter.py + --label-groups, recorded in the checkpoint packages/f3/adapter-train/config_coarse.json F2's config + "label_groups": "coarse5" packages/f3/adapter-train/run_cpu_smoke.py 56 checks (F2's 27 + 29 new) packages/f3/adapter-train/structure_adapter.py unedited copy (sha b85d2dba…) packages/f3/retrieved/out/f3a///.png(+.json) 80 part-A canvases (1.1 GB) packages/f3/retrieved/out/f3b/{fromnoise,si3_donor}/… 44 part-B canvases (638 MB) packages/f3/retrieved/out/f3a-records.jsonl 80 records packages/f3/retrieved/out/f3b-{fromnoise,si3}-records.jsonl 24 + 20 records packages/f3/retrieved/out/adapter-b/ loss_log.jsonl, val_loss.json, summary.json, train_config.json, checkpoints/step_{000500..003000} packages/f3/retrieved/out/SHA256SUMS 149 files, verified after transfer packages/f3/retrieved/evidence/ steps.log, exit_code.txt (0), runtime.txt, smoke.log, plan_a.log, plan_b.log, infer_a.log, infer_b1.log, infer_b2.log, train_b.log, WORKER_DONE packages/f3/eval/f3_features.py features.csv via evaluate_sweep._feat_one + C1's envelope packages/f3/eval/f3_layout.py both layout instruments + the rotation test, one pass packages/f3/eval/f3_screens.py phikon-v2 + pixel NCC, the harness's own conventions packages/f3/eval/f3_sheets.py contact sheets and the 1:1 zoom grids packages/f3/eval/f3_tables.py every table above packages/f3/eval/run_eval.sh the whole evaluation, in order packages/f3/eval/{features,layout-iou,screen-embed,screen-ncc}.csv packages/f3/eval/{layout-iou,stage-features,stage-screens,within-arm-diversity}.json packages/f3/eval/tables.md the generated tables packages/f3/eval/sheets/, sheets-small/ 8 + 4 contact sheets fleet/logs/{d.log,f3-watch.out,f3-retrieve.out}, fleet/state/d/powered.log ``` Reproduce with: ```bash cd operations/research/sgh-program-20260908/fleet ./fleet.sh up d && ./fleet.sh stage d ../packages/f3 && ./fleet.sh launch d ../packages/f3 ./fleet.sh watch d ./fleet.sh ssh d 'sudo -n systemctl stop sgh-fleet-stop.timer' ./fleet.sh retrieve d "$PWD/../packages/f3/retrieved" && ./fleet.sh down d && ./fleet.sh status cd .. && bash packages/f3/eval/run_eval.sh packages/f3/retrieved 10 ```