# F2-TRAIN — the structure adapter, trained on an A100 and measured against the two-pass recipe

Work package **F2-TRAIN** (Phase 3) of the SGH programme, executed 2026-09-08 on spot A100 slot `d`
(`sgh-a100-slot-d`, us-central1-f). Engineering evidence only: everything below is a measurement of
geometry, pixel statistics and throughput. **No clinical claim is made or implied, no pathologist saw
any of these images, and nothing here says a synthetic field is diagnostically usable.**

The VM was stopped at the end of the run and is TERMINATED. Nothing was committed to git.

---

## 1. Headline

The adapter **does** control layout, and by a wide margin — that was the open question and the
rotation test answers it unambiguously. It **does not** clear the success bar in `F2_RESULT.md`
section 8, which required *both* an in-range ring fraction *and* a smaller morphometric envelope
distance than the two-pass recipe on the same cartoons. It clears the second condition only for
intestinal metaplasia.

| question | answer |
|---|---|
| Does a from-noise generation follow the cartoon's label map? | **Yes.** IoU(output ring mask, cartoon lumen mask) is 0.26–0.56 against a 0.03–0.06 chance level, and 2.6–5.4x the two-pass recipe's on the same cartoons |
| Does rotating the label map rotate the output? | **Yes.** 5–15x separation between the matched and mismatched overlaps, all four categories |
| Is the ring fraction inside the real held-out range? | **Yes, all four** — but the held-out range is wide (n=4–6 fields); against the held-out *mean* the adapter overshoots everywhere, from +0.016 (IM) to +0.243 (gastritis) |
| Is the morphometric envelope distance better than two-pass? | **Only for intestinal metaplasia** (9/14 cartoons, median -0.23). Worse for normal, gastritis and mixed by +1.3 to +1.9 |
| Did validation loss separate from the zero-init base model? | **Yes, but barely**: 0.1641 -> 0.1594, a 2.9% relative drop, and flat after step 1500 |
| Verdict against section 8 | **Not a success; not a failure either.** No failure criterion is tripped. Partial success on intestinal metaplasia only |

---

## 2. Run timeline (all times UTC, 2026-09-08)

| time | event |
|---|---|
| 14:11:39 | `fleet.sh up d` — spot A100 acquired first attempt, guest deadline +32400 s |
| 14:12:20 | ssh ready, `NVIDIA A100-SXM4-40GB, 40960 MiB, driver 580.173.02` (62 s from start to ready) |
| 14:12:33–14:15:31 | `fleet.sh stage d ../packages/f2-adapter` — 8 items, 20 MB (19 MB of it the 60 a1 label maps), 178 s |
| 14:16:17 | launch #1 |
| 14:16:34 | **FAILURE 1: `pair build FAILED` — `ModuleNotFoundError: No module named 'scipy'`** |
| 14:17–14:18 | fixed on the VM: `pip install "numpy==1.26.4" scipy scikit-image` into the asset-root venv |
| 14:18:35 | launch #2 |
| 14:18:45–14:26:38 | segmentation of the 96 donor fields into label maps, on the VM, **472 s, 96 ok / 0 failed** |
| 14:26:38–14:26:56 | CPU smoke test on the VM: **27 passed, 0 failed in 15 s** |
| 14:26:56 | training starts; DiT cast bf16 straight from the fp32 checkpoint, VAE fp16, UNI2-h fp32 |
| 14:28:10 | `validation loss @ step 0: 0.1641 (48 batches, 28 s) [zero-init adapter == base model]` |
| 14:28:15 | step 1 |
| 16:44:40 | step 3000 |
| 16:45:07 | `validation loss @ step 3000: 0.1594`; checkpoint `step_003000` saved |
| 16:45:09–18:17:22 | 64 from-noise generations at 4096x2048 (60 a1 cartoons + 4 `--rotate180`) |
| 18:17:22 | `WORKER_DONE`, `exit_code.txt` = 0, 64 images, `out/SHA256SUMS` written |
| 18:18 | the worker's 20-minute self-stop timer was stopped by hand so the 1.5 GB retrieve could not race it (the guest-deadline metadata and `--max-run-duration`, both 32400 s, remained armed as the real backstop) |
| 18:19:03–18:28:09 | `fleet.sh retrieve d …/retrieved` — 1.5 GB, 546 s (2.8 MB/s), **SHA256SUMS OK (73 files)** |
| 18:28:21–18:29:32 | `fleet.sh down d`; `fleet.sh status` confirms **TERMINATED** |

**Preemptions: none.** The spot instance ran 4 h 18 m uninterrupted, so the auto-resume path
(`resuming training from …` in `steps.log`) was never exercised in this run. It was exercised by
FLEET's own preemption tests, not here — this package cannot claim to have re-verified it.

### `fleet/slots.json` change (as instructed)

`defaults.deadline_seconds` and `defaults.max_run_duration_seconds` were both raised from **10800 to
32400** before `up`. Both are still 32400 in the file. This lengthens only the *backstop*; the sweep
slots call `down` explicitly, so a longer backstop cannot leave them running.

---

## 3. What failed, and the fix

**FAILURE 1 — no `scipy` on the VM (17 s into the first launch).** `segment_real.py` imports
`scipy.ndimage` and, deeper in, `skimage.feature.peak_local_max` / `skimage.segmentation.watershed`.
The PixCell asset-root venv is a *generation* environment: numpy 1.26.4, pillow, torch 2.9.1+cu129,
diffusers, transformers, timm, peft — and no scipy, no scikit-image. Neither does the system python.
This was invisible to the CPU smoke test, which reads an already-built manifest and never runs the
segmenter, and invisible to F2-PAIRS, which ran the segmenter on the Mac where both are installed.

Fixed on the VM with `pip install --no-input "numpy==1.26.4" scipy scikit-image`, pinning numpy so
the resolver could not upgrade it out from under torch. Result: `scipy 1.17.1`, `scikit-image 0.26.0`,
`imageio 2.37.4`, `lazy-loader 0.5`, `tifffile 2026.3.3`; numpy stayed 1.26.4 and
`torch.cuda.is_available()` stayed True. `segment_real.py --selftest` then passed. **The change lives
on slot d's boot disk only, not in the golden image `sgh-pixcell-a100-20260908`** — any other slot
that runs this package will hit the same error. Either add the two wheels to the image or ship a
`pip install` line in `worker.sh`.

**BUG FOUND AND FIXED BEFORE LAUNCH — the inference donor path.** `worker.sh`'s inference planner read
`donors.json` (which holds bare file names such as `hiesd-28f9b423__hpylori_gastritis__34688_2112__mpp0.25.png`)
and passed the bare name straight to `infer_adapter.py --donor`. `Image.open` would have resolved it
against the run root and raised `FileNotFoundError` on every one of the 64 generations — and because
the planner's loop uses `subprocess.run(..., check=False)`, all 64 would have failed *silently* and
the worker would still have written `WORKER_DONE` with exit 0 and zero PNGs. Patched in
`packages/f2-adapter/worker.sh` to resolve `REFS_ROOT/<category>/<file>`, to log `MISSING_DONORS` if a
path does not exist, and to match the category longest-key-first. Verified on the VM before launch
that all four donor-0 files exist. This is the same class of bug as FLEET gotcha 8: a step that
"succeeds" having done nothing.

**Not fixed (deliberately): the per-image model reload.** The planner spawns one
`infer_adapter.py` process per image, so PixCell + VAE + UNI2-h are loaded 64 times. Each image cost
**79 s wall of which 46.3 s was generation** — roughly 35 minutes of the run, about USD 1.2, spent
re-loading weights. Rewriting the loop mid-run risked the whole 2.5 h of training, so it was left
alone. A single-process driver that loads once and loops is the obvious fix for the next run.

---

## 4. Training

Config as shipped (`adapter-train/config.json`): `cond=labels` (9 soft one-hot channels at 128x128),
crop 1024, batch 4, grad-accum 1, lr 1e-4 cosine with 100 warmup steps, weight decay 0.01, grad-norm
clip 1.0, CFG dropout 0.10, conditioning dropout 0.10, bf16, gradient checkpointing on, 3000 steps,
checkpoint every 500, validate every 250, seed 0.

- **Adapter**: `StructureAdapter width=320 blocks=4 injections=2`, **8.25M trainable parameters**
  (trunk 7.51M + two zero-init heads of 0.37M), fp32 33 MB, injected at blocks `[0, 14]` of 28.
- **Base model**: PixCell DiT 608.1M params, `sample_size=128`, **cast bf16 directly from the fp32
  checkpoint**; SD3 VAE fp16; UNI2-h fp32. The `UNI check` in the trainer confirmed the tokens it
  computes are **bit-identical** to `pixcell_generate.uni_tokens` (max abs diff 0.000e+00).
- **Split**: `val_split: by-slide` as instructed, because the 22 held-out fields are not on the VM.
  Three whole training slides were held out — `hiesd-324885ef`, `hiesd-9ded7e41`, `hiesd-da8667dc` —
  leaving **64 train fields / 32 val fields**, i.e. 512 train crops and 128 val crops per epoch.
  3000 steps at batch 4 is **23 epochs**.
- **Throughput**: **2.61 s/step steady state**, 2.74 s/step including validations and checkpoints;
  3000 steps in **8246 s (2 h 17 m)**.
- **VRAM**: `max_memory_allocated` **6.86 GB**, 7.5 GB reserved (`nvidia-smi` 7660 MiB of 40960).
  That is at the very bottom of F2_RESULT's 11–15 GB estimate: batch 16 would have fitted, and the
  run was encoder-bound rather than memory-bound.
- **Gradients**: `grad_norm` median 0.0150, min 0.0032, max 0.967 — **never zero**, so the zero-init
  heads did leave zero (failure criterion 3 does not apply).

### Loss curve (`packages/f2-adapter/retrieved/loss.png`)

| step | train loss (mean of 25 around) | val loss | grad norm | max VRAM GB | s/step (median of 25) |
|---|---|---|---|---|---|
| 0 | 0.1387 | 0.1641 | 0.014 | 6.8 | 2.60 |
| 1 | 0.1387 |  | 0.014 | 6.8 | 2.60 |
| 250 | 0.1228 | 0.1611 | 0.034 | 6.9 | 2.61 |
| 500 | 0.0883 | 0.1604 | 0.022 | 6.9 | 2.61 |
| 750 | 0.1253 | 0.1601 | 0.019 | 6.9 | 2.61 |
| 1000 | 0.1322 | 0.1598 | 0.022 | 6.9 | 2.61 |
| 1250 | 0.1372 | 0.1598 | 0.022 | 6.9 | 2.61 |
| 1500 | 0.1084 | 0.1596 | 0.023 | 6.9 | 2.61 |
| 1750 | 0.0917 | 0.1595 | 0.021 | 6.9 | 2.61 |
| 2000 | 0.0648 | 0.1595 | 0.010 | 6.9 | 2.61 |
| 2250 | 0.0844 | 0.1595 | 0.018 | 6.9 | 2.61 |
| 2500 | 0.1040 | 0.1595 | 0.008 | 6.9 | 2.61 |
| 2750 | 0.1018 | 0.1594 | 0.011 | 6.9 | 2.61 |
| 3000 | 0.0734 | 0.1594 | 0.007 | 6.9 | 2.61 |

Per-step training loss is dominated by the random diffusion timestep, so it scatters between 0.0002
and 0.42 throughout and its trend is only visible in the smoothed curve; the validation loss (fixed
crops, fixed noise, fixed timesteps `linspace(50, 950)`) is the number that matters. It fell
**0.1641 -> 0.1594 (-2.9% relative)** and was essentially flat from step 1500 onward
(0.1596 -> 0.1594 over the last half of the run). The cosine schedule reached lr 0 exactly at 3000.

Read honestly: the adapter learned *something* the frozen model did not already know, monotonically
and without instability, but the epsilon-MSE it improves is dominated by the frozen DiT's already-good
prediction, so a 2.9% drop is a weak signal on its own. The layout measurements in section 6 are the
evidence that it learned the *right* thing; the loss alone would not carry that claim.

---

## 5. Generation

64 images at 4096x2048, 21 overlapping 1024 windows on one shared latent, 20 DPMSolver++ steps,
CFG 1.5, `--init noise`, seed 51001, adapter scale 1.0, checkpoint `step_003000`, donor 0 of each
category from `donors.json`:

| category | token donor at inference | in the adapter's training set? |
|---|---|---|
| normal | `hiesd-40f3c0cf__normal__23744_3072` | yes |
| hpylori_gastritis | `hiesd-28f9b423__hpylori_gastritis__34688_2112` | yes |
| intestinal_metaplasia | `hiesd-324885ef__intestinal_metaplasia__15616_54528` | **no — held-out slide** |
| mixed | `hiesd-324885ef__mixed__11584_55488` | **no — held-out slide** |

Worth flagging, because `hiesd-324885ef` is one of the three slides the by-slide split removed: the
two categories generated from a slide the adapter never saw are IM (its best category) and mixed (its
worst). Two categories is not enough to say anything about donor generalisation, but the pairing
should not be read as evidence either way.

**Cost per image: 46.3 s of generation** (120 denoiser calls), remarkably constant — sd across the 64
images was 0.023 s, min 46.27 s, max 46.38 s — plus ~33 s of process startup and model loading. 64 images in 92 minutes.

---

## 6. Evaluation

All local, all on the retrieved PNGs, with `research/sgh-synthetic-histopathology/.venv/bin/python`.
Ring topology is `code/ring_topology.py` with `config/sydney_qa_v1.json` (md5 `aa8cdf2b…`, byte-identical
to `code/sydney_qa_v1.json`). Morphometrics is `morphometrics/morphometrics.py` scored against the
stored `morphometrics/envelope.json` through `calibrate.score_rows`, i.e. the same instrument and the
same fitted envelope C1 calibrated — nothing was re-fitted here.

**The two-pass comparison uses `packages/sweep-v1/retrieved/*/out/a1/pass2/<cat>/<stem>.png`** — the
SWEEP package's own base arm (`start_index` 6, seed 51002) on **the same 60 a1 cartoons**, so this is
a like-for-like comparison rather than the cross-cartoon `fullset_pass2` numbers. Two caveats:
**59 of 60** were usable (`intestinal_metaplasia_s14_oblique.png` was zero bytes in that package's
local copy — a truncated retrieval, not a generation failure; the sweep's own `steps.log` reports
`failed=0`), and **the donor matches only for the `s11` cartoons**: the A1 sweep rotates donors by
seed (s11->donor 0 … s15->donor 4) while this run used donor 0 throughout. Table T5 is the strictly
donor-matched subset. The `fullset_pass2` medians from `morphometrics/scores.csv` are quoted in
section 7 for continuity with C1, but they are a different cartoon set.

#### T1 ring_with_lumen_fraction

| category | adapter, from noise (n=15) | two-pass on the same cartoons | a1 cartoon itself | real held-out mean (min-max) | adapter gap to real | two-pass gap to real |
|---|---|---|---|---|---|---|
| normal | 0.9218 +- 0.0886 | 0.3385 +- 0.2346 (n=15) | 0.4420 | 0.8538 (0.671-0.976) | +0.0680 | -0.5153 |
| hpylori_gastritis | 0.9856 +- 0.0158 | 0.0618 +- 0.0716 (n=15) | 0.7430 | 0.7428 (0.297-1.000) | +0.2428 | -0.6810 |
| intestinal_metaplasia | 0.7265 +- 0.1509 | 0.2511 +- 0.1927 (n=14) | 0.6589 | 0.7101 (0.128-0.953) | +0.0164 | -0.4590 |
| mixed | 0.8155 +- 0.1028 | 0.2360 +- 0.1616 (n=15) | 0.6323 | 0.7261 (0.268-1.000) | +0.0894 | -0.4901 |

#### T2 morphometric envelope distance (Mahalanobis, per-category real-train envelope)

| category | adapter median (min-max) | two-pass median | real held-out median | real held-out max | adapter inside band | paired: adapter closer on | median paired delta |
|---|---|---|---|---|---|---|---|
| normal | 8.620 (5.309-10.931) | 7.435 | 3.330 | 4.339 | 0% | 3/15 | +1.371 |
| hpylori_gastritis | 8.280 (7.527-10.646) | 7.760 | 4.193 | 7.974 | 13% | 3/15 | +1.260 |
| intestinal_metaplasia | 5.112 (4.525-5.604) | 5.557 | 2.575 | 4.502 | 0% | 9/14 | -0.228 |
| mixed | 9.188 (7.989-10.457) | 7.355 | 3.660 | 4.377 | 0% | 0/15 | +1.939 |

#### T3 layout adherence: IoU(output ring mask, cartoon lumen mask)

| category | adapter, true label map | adapter, label map rotated (chance) | adapter, other cartoon (chance) | two-pass, true label map | two-pass, chance (rotated) | adapter / two-pass |
|---|---|---|---|---|---|---|
| normal | 0.5020 | 0.0358 | 0.0489 | 0.1267 | 0.0408 | 4.0x |
| hpylori_gastritis | 0.5599 | 0.0282 | 0.0432 | 0.1028 | 0.0399 | 5.4x |
| intestinal_metaplasia | 0.2586 | 0.0565 | 0.0504 | 0.1002 | 0.0439 | 2.6x |
| mixed | 0.3194 | 0.0509 | 0.0501 | 0.1050 | 0.0406 | 3.0x |

#### T4 rotation test (one cartoon per category, both renders)

| category | cartoon | (a) rotated output vs rotated label | (b) rotated output vs unrotated label | (a) unrotated output vs unrotated label | (b) unrotated output vs rotated label | a/b (rotated) | a/b (unrotated) |
|---|---|---|---|---|---|---|---|
| hpylori_gastritis | hpylori_gastritis_s11_across | 0.4599 | 0.0417 | 0.5170 | 0.0345 | 11.0x | 15.0x |
| intestinal_metaplasia | intestinal_metaplasia_s11_across | 0.2173 | 0.0397 | 0.1982 | 0.0410 | 5.5x | 4.8x |
| mixed | mixed_s11_across | 0.2634 | 0.0389 | 0.2566 | 0.0412 | 6.8x | 6.2x |
| normal | normal_s11_across | 0.3494 | 0.0283 | 0.3519 | 0.0328 | 12.3x | 10.7x |

#### T5 donor-matched subset (s11 cartoons only; adapter and sweep both used donor 0)

| category | cartoon | adapter ring | two-pass ring | adapter D | two-pass D | adapter layout IoU | two-pass layout IoU |
|---|---|---|---|---|---|---|---|
| normal | s11_across | 0.9398 | 0.0000 | 8.509 | 7.523 | 0.3519 | 0.0450 |
| normal | s11_along | 0.9688 | 0.0164 | 9.592 | 7.435 | 0.5611 | 0.0875 |
| normal | s11_oblique | 0.9670 | 0.0455 | 8.975 | 7.605 | 0.6755 | 0.0883 |
| hpylori_gastritis | s11_across | 0.9882 | 0.0000 | 10.214 | 7.831 | 0.5170 | 0.0614 |
| hpylori_gastritis | s11_along | 0.9630 | 0.0208 | 8.025 | 9.195 | 0.5788 | 0.1324 |
| hpylori_gastritis | s11_oblique | 1.0000 | 0.1905 | 8.192 | 6.932 | 0.5843 | 0.0709 |
| intestinal_metaplasia | s11_across | 0.7481 | 0.2017 | 5.284 | 5.587 | 0.1982 | 0.0639 |
| intestinal_metaplasia | s11_along | 0.4444 | 0.1742 | 5.604 | 5.757 | 0.1832 | 0.1057 |
| intestinal_metaplasia | s11_oblique | 0.8000 | 0.3622 | 5.134 | 5.535 | 0.3485 | 0.1368 |
| mixed | s11_across | 0.8485 | 0.2333 | 8.917 | 5.872 | 0.2566 | 0.0807 |
| mixed | s11_along | 0.5556 | 0.0575 | 10.457 | 6.380 | 0.2426 | 0.0482 |
| mixed | s11_oblique | 0.8800 | 0.4583 | 8.609 | 5.733 | 0.4296 | 0.1718 |

#### T6 per-image correlation with the cartoon

- Pearson r(cartoon ring fraction, adapter ring fraction) over 60 cartoons = 0.513
- Pearson r(cartoon ring fraction, two-pass ring fraction) over 59 cartoons = -0.045

#### T7 what the envelope distance is actually made of

Median per category, adapter vs the two-pass base arm vs the real held-out fields:

| category | ring area um2 (A / 2P / real) | ring density per mm2 (A / 2P / real) | nuclear density per mm2 (A / 2P / real) | median nucleus area um2 (A / real) |
|---|---|---|---|---|
| normal | 439 / 425 / 442 | 148 / 141 / 101 | **9067** / 4893 / 6350 | **17.3** / 25.1 |
| hpylori_gastritis | 416 / 414 / 365 | 124 / 107 / 89 | **12322** / 5912 / 5996 | **14.2** / 23.1 |
| intestinal_metaplasia | 594 / 432 / 438 | 191 / 170 / 120 | **8835** / 5784 / 6326 | 20.4 / 23.3 |
| mixed | 604 / 463 / 441 | 187 / 143 / 101 | **10499** / 5524 / 6762 | 18.2 / 21.8 |

The three features that dominate the `worst_features` column of `adapter-scores.csv` are the same in
every category and in the same direction: `nuc_density_per_mm2 = +`, `epi_nuc_density_per_mm2 = +`,
and (for IM) `median_ring_area_um2 = +`. **The adapter's single systematic defect is nuclei: it makes
1.4x to 2.1x too many of them and each is 20–40% too small.** Ring *size* is close to real; ring
*density* is 25–70% high, which is a knock-on of the same thing. The two-pass recipe, which starts
from a real image's latent, gets nuclear density almost exactly right and loses on geometry instead.

---

## 7. Reading the tables

**Ring topology (T1).** The a1 two-pass base arm collapses on this cartoon set —
0.06 to 0.34 lumen-bearing ring fraction against a real held-out 0.71–0.85. The adapter is
dramatically closer on every category: gaps of +0.016 (IM), +0.068 (normal), +0.089 (mixed),
+0.243 (gastritis) against the two-pass's -0.46 to -0.68. Note this is *not* the `fullset_pass2`
picture that C1 recorded (normal 0.726, gastritis 0.475, IM 0.917, mixed 0.726 on the fullset
cartoons); the a1 cartoons behave very differently under the two-pass recipe, which is SWEEP's
finding to explain, not this package's. Against `fullset_pass2` the adapter is still closer to the
real mean for normal, gastritis and IM, and further for mixed.

The adapter's failure direction is a consistent **overshoot**: it makes tissue whose rings almost
always contain a pale lumen (gastritis 0.986, i.e. 62.6 of 63.3 detected rings on the average field). Real tissue
is messier. Section 8's condition 1 is satisfied on the letter — every category's mean lies inside
the real held-out min–max — but that range spans 0.13–1.00 for IM and 0.30–1.00 for gastritis on 4–6
fields, so "inside the range" is a weak test and should not be read as "matches real".

**Morphometrics (T2).** Paired on the same cartoon, the adapter is *further* from the real envelope
than the two-pass base arm for normal (3/15 wins, median +1.37), gastritis (3/15, +1.26) and mixed
(0/15, +1.94), and *closer* for intestinal metaplasia (9/14, -0.23). The donor-matched `s11` subset
(T5) tells the same story: IM 3/3 to the adapter, mixed 0/3, normal 0/3, gastritis 1/3. Neither method sits
inside the real held-out band often: the adapter is inside for 0% of fields in three categories and
13% in gastritis; the two-pass base arm is inside for 0% (normal), 0% (mixed), 21% (IM) and 60%
(gastritis) — the one place the two-pass is clearly the better of the two.

**Layout (T3, T4, T6).** This is where the adapter wins and the win is not marginal:

- IoU between the output's ring mask and its own cartoon's lumen mask is **0.26–0.56**, against a
  chance level of **0.03–0.06** measured two independent ways (same label map rotated 180, and a
  *different* cartoon of the same category).
- The same measurement on the two-pass outputs is **0.10–0.13** — above chance, as expected since
  pass 1 is img2img from the cartoon, but **2.6x to 5.4x lower than the adapter's**.
- The rotation test does exactly what F2_RESULT section 8 asked: for the rotated output the overlap
  with the *rotated* label map beats the overlap with the unrotated one by **5.5x–12.3x**, and the
  unrotated output shows the mirror-image separation of **4.8x–15.0x**. H04's failure mode does not
  reproduce.
- Per-image, the adapter's ring fraction tracks its cartoon's ring fraction at Pearson
  **r = 0.513** over 60 cartoons; for the two-pass outputs the same correlation is **r = -0.045**,
  i.e. no relationship at all.

**The `--scale 0` ablation of failure criterion 3 was not run** — it would have cost a second VM boot
after `down`, and the evidence already on hand is strictly stronger: non-zero `grad_norm` at every one
of 3000 steps, a validation loss that moved off the zero-init value, and outputs whose geometry
tracks *which* label map was supplied (rotating it moves the tissue). An adapter doing nothing cannot
produce T3 or T4.

---

## 8. Contact sheets

`packages/f2-adapter/retrieved/adapter-sheet.png` (3790x1950) — one row per category, columns
*cartoon label map | adapter from noise | adapter with the label map rotated 180 | two-pass on the same
cartoon*, each cell the whole 4096x2048 field downscaled so that layout is comparable.
`packages/f2-adapter/retrieved/detail-sheet.png` (3096x1082) — the same IM and gastritis cartoons at
**native 0.25 um/px**, 256x128 um windows, adapter | two-pass | a real held-out field.

**What I see, honestly.**

At field scale the adapter columns clearly carry the label map. In the gastritis row the label map's
dense, gland-free patch at lower-centre-right appears as a correspondingly dense pale region in the
from-noise output, and in the rotated column that region has moved to the upper left; in the IM and
mixed rows the vertical crypt columns of the label map appear as vertical epithelial columns in the
output, and rotate with it. The normal row is the hardest to judge by eye because the label map is a
near-uniform carpet of round gland profiles — but that is also where the IoU is highest (0.50).
The two-pass column is not obviously related to the label map in any row.

At native resolution the ranking reverses. The two-pass crops have convincing one-cell-thick
epithelial bands with elongated, basally-oriented nuclei, visible goblet-type vacuoles in the IM crop
and stromal collagen strands; the real held-out crops have all of that plus crisply defined chromatin
and, in the gastritis field, red blood cells in a vessel. The adapter crops are grainier and flatter:
nuclei are small, round, soft-edged and scattered fairly evenly rather than being organised into a
polarised band at the edge of a gland, the cytoplasm is a washed pink with little internal texture,
and there are no red cells. The pale holes are the right *size* but their walls are made of a crowd
of small nuclei rather than a palisade. This is the same defect the morphometrics found — too many,
too small nuclei — and it is visible without any measurement.

So: the adapter has learned **where** to put tissue and has not yet learned **what** the tissue looks
like up close, while the two-pass recipe is the other way round.

---

## 9. Verdict against `F2_RESULT.md` section 8

**Success** required *both* conditions. It is met for **intestinal metaplasia only**, and even there
condition 2 is a small margin (median -0.23 Mahalanobis, 9 of 14 cartoons).

| condition | normal | gastritis | IM | mixed |
|---|---|---|---|---|
| 1. ring fraction inside the real held-out range | yes (0.922 in 0.671–0.976) | yes (0.986 in 0.297–1.000) | yes (0.727 in 0.128–0.953) | yes (0.816 in 0.268–1.000) |
| 2. morphometric distance below the two-pass on the same cartoon | **no** (+1.37) | **no** (+1.26) | **yes** (-0.23) | **no** (+1.94) |
| **combined** | no | no | **yes** | no |

The "weaker but still reportable" outcome named in section 8 — *the adapter fixes gastritis, where the
two-pass recipe is worst* — is **half true and should be stated carefully**. On ring topology the
adapter moves gastritis from 0.06 (a1 two-pass) or 0.475 (fullset two-pass) to 0.986 against a real
0.743: it stops undershooting and starts overshooting, by a comparable margin. On morphometrics it
makes gastritis worse. Calling that "fixed" would be wrong.

**Failure criteria — none is tripped:**

| criterion | result |
|---|---|
| 1. Layout ignored (ring density / lumen fraction uncorrelated with the cartoon's) | **not tripped.** r = 0.513 over 60 cartoons; layout IoU 4–10x chance |
| 2. Rotation test — output is not the rotation of the unrotated output | **not tripped.** 5.5x–12.3x separation on the rotated renders, 4.8x–15.0x on the unrotated ones, all four categories |
| 3. Adapter does nothing (zero-init heads never left zero) | **not tripped.** `grad_norm` non-zero at all 3000 steps (median 0.0150, min 0.0032); conditioning demonstrably changes the output. The literal `--scale 0` vs `--scale 1` ablation was not run — see section 7 |
| 4. Validation loss does not separate from step 0 | **not tripped, but weakly.** 0.1641 -> 0.1594, -2.9% relative, flat after step 1500 |

**Overall.** Phase 3's question was whether a small zero-init control branch on a frozen PixCell DiT
can make a from-noise generation obey a procedural label map. The answer measured here is **yes for
geometry, no for texture**: the layout signal is strong, unambiguous and reproducible across all four
categories and 60 cartoons, and the resulting fields are further from the real morphometric envelope
than the two-pass recipe in three categories out of four because the adapter over-produces small
nuclei. This is a promising negative: the hard part (control) works, and the part that failed is a
distribution-matching problem with obvious next moves.

**The honest caveat from F2_RESULT section 8 still stands and is now sharper.** The adapter was
trained on *segmenter output*, not on ground truth, so it learned "epithelium wherever
`segment_real.py` said epithelium". The programme cartoons' label maps are cleaner than that training
conditioning, and the adapter's response to a clean map is to produce a lumen in nearly every ring —
gastritis 0.986. Some of the overshoot is the adapter, and some is a train/inference distribution
shift the segmenter created. This run cannot separate the two. The mucin channel specifically was not
measured separately here (no goblet-vacuole detector exists yet); the detail sheet shows the adapter's
IM crops lack convincing goblet vacuoles, but that is a visual impression, not a measurement.

---

## 10. Exact commands

```bash
cd [local]/sgh-program-20260908
P=[local]/python

# 0. before anything: raise both fleet backstops from 10800 to 32400 in fleet/slots.json
#    (deadline_seconds and max_run_duration_seconds), and fix the donor path in the package worker.

# 1. local pre-flight, free: the package's own 27-check CPU smoke against F2-PAIRS' manifest
$P packages/f2-adapter/adapter-train/run_cpu_smoke.py --manifest pairs/data/manifest.json --out-dir /tmp/f2-smoke

# 2. the run (fleet.sh run == up + stage + launch + watch; done as four calls here so that a
#    staging failure could be seen before committing to a multi-hour watch)
cd fleet
./fleet.sh up d
./fleet.sh stage  d ../packages/f2-adapter
./fleet.sh launch d ../packages/f2-adapter
./fleet.sh watch  d                       # backgrounded; polls 60 s, recovers preemptions

# the scipy repair, on the VM, after launch #1 failed
./fleet.sh ssh d '[local]/pip \
    install --no-input "numpy==1.26.4" scipy scikit-image'

# supervision, every ~2 minutes
./fleet.sh ssh d 'R=[local]/f2-adapter; tail -3 $R/evidence/train.log; nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader'

# 3. retrieve, verify, stop
./fleet.sh ssh d 'sudo -n systemctl stop sgh-fleet-stop.timer'   # so the 20-min backstop cannot race a 1.5 GB copy
./fleet.sh retrieve d [local]/retrieved
./fleet.sh down d && ./fleet.sh status

# 4. evaluation
cd ..
$P packages/f2-adapter/eval/plot_loss.py --log packages/f2-adapter/retrieved/out/adapter/loss_log.jsonl \
   --val packages/f2-adapter/retrieved/out/adapter/val_loss.json \
   --out packages/f2-adapter/retrieved/loss.png --table packages/f2-adapter/eval/loss-table.md
$P packages/f2-adapter/eval/cartoon_rings.py                      # ring metrics of the 60 a1 cartoons
$P packages/f2-adapter/eval/f2_eval.py --images packages/f2-adapter/retrieved/out/adapter-infer \
   --labels packages/f2-adapter/cartoons/a1 --out packages/f2-adapter/eval/results --jobs 10
$P packages/f2-adapter/eval/layout_iou.py packages/f2-adapter/retrieved/out/adapter-infer \
   packages/f2-adapter/cartoons/a1 packages/f2-adapter/eval/results/layout-iou-adapter.csv
# the same three on the sweep's two-pass a1 outputs, collected into eval/twopass-a1/ as symlinks
$P packages/f2-adapter/eval/f2_eval.py   --images packages/f2-adapter/eval/twopass-a1 \
   --labels packages/f2-adapter/cartoons/a1 --out packages/f2-adapter/eval/results-twopass --jobs 10
$P packages/f2-adapter/eval/layout_iou.py packages/f2-adapter/eval/twopass-a1 \
   packages/f2-adapter/cartoons/a1 packages/f2-adapter/eval/results-twopass/layout-iou-twopass.csv
$P packages/f2-adapter/eval/final_tables.py > packages/f2-adapter/eval/final-tables.md
$P packages/f2-adapter/eval/make_sheet.py --images packages/f2-adapter/retrieved/out/adapter-infer \
   --labels packages/f2-adapter/cartoons/a1 --twopass packages/f2-adapter/eval/twopass-a1 \
   --out packages/f2-adapter/retrieved/adapter-sheet.png
```

---

## 11. Cost and powered time

Reconstructed from `fleet/state/d/powered.log` (`UP 1788876699` -> `DOWN 1788892172`) and
cross-checked against the orchestrator log.

| | |
|---|---|
| powered window | 2026-09-08 **14:11:39Z -> 18:29:32Z** |
| **powered minutes** | **257.9 min (4.298 h)**, one session, zero preemptions |
| spot A100 (a2-highgpu-1g) @ USD 2.12/h | **USD 9.11** |
| egress, 1.5 GB retrieved to the Mac @ ~USD 0.12/GB | ~USD 0.18 |
| **total attributable to this package** | **~USD 9.29** |

Where the 258 minutes went: 5 boot/stage, 8 segmentation, 0.3 smoke, **137 training**, **92
generation**, 9 retrieve, 1 shutdown, ~5 of orchestration slack. Two obvious savings for a rerun:
**~35 min** of the generation phase is the 64 redundant model loads (one process per image), and the
run used 6.9 GB of 40 GB VRAM at batch 4, so batch 12–16 would cut the 137 training minutes
substantially at the same step count. The other four slots were untouched by this package; `a`, `b`
and `c` were RUNNING under the concurrent SWEEP package when this one finished and are not this
package's to stop.

---

## 12. Files

```
packages/f2-adapter/worker.sh                          patched: donor paths resolved under REFS_ROOT
packages/f2-adapter/retrieved/out/adapter-infer/       64 PNGs at 4096x2048 + 64 .json provenance (944 MB)
packages/f2-adapter/retrieved/out/adapter/             loss_log.jsonl, val_loss.json, summary.json,
                                                       train.log, train_config.json,
                                                       checkpoints/step_{000500..003000}/ (6 x 95 MB:
                                                       structure_adapter.safetensors 33 MB +
                                                       training_state.pt 66 MB), latest.txt
packages/f2-adapter/retrieved/out/smoke/               the on-VM smoke result
packages/f2-adapter/retrieved/out/SHA256SUMS           73 files, all verified locally after transfer
packages/f2-adapter/retrieved/evidence/                steps.log, exit_code.txt (0), runtime.txt,
                                                       build_pairs.log, smoke.log, train.log,
                                                       infer.log, infer_plan.json, WORKER_DONE
packages/f2-adapter/retrieved/loss.png                 loss / validation / VRAM+throughput curves
packages/f2-adapter/retrieved/adapter-sheet.png        label map | adapter | adapter rot180 | two-pass
packages/f2-adapter/retrieved/detail-sheet.png         native-resolution adapter | two-pass | real
packages/f2-adapter/eval/f2_eval.py                    ring + morphometrics + rotation test
packages/f2-adapter/eval/layout_iou.py                 layout adherence with two chance levels
packages/f2-adapter/eval/cartoon_rings.py              ring metrics of the a1 cartoons themselves
packages/f2-adapter/eval/plot_loss.py                  loss.png + loss-table.md
packages/f2-adapter/eval/make_sheet.py                 the contact sheet
packages/f2-adapter/eval/final_tables.py               every table in section 6
packages/f2-adapter/eval/results/                      adapter-ring.csv, adapter-features.csv,
                                                       adapter-scores.csv, rotation-test.csv,
                                                       layout-iou-adapter.csv, summary.json
packages/f2-adapter/eval/results-twopass/              the same instruments on sweep-v1's a1 pass-2 base arm
packages/f2-adapter/eval/twopass-a1/                   59 symlinks into packages/sweep-v1/retrieved/
packages/f2-adapter/eval/a1-cartoon-ring.csv           the 60 a1 cartoons' own ring metrics
packages/f2-adapter/eval/final-tables.md               sections 6's tables as generated
fleet/logs/d.log, fleet/logs/f2-train-poll.log         orchestrator and supervision logs
fleet/slots.json                                       deadline_seconds / max_run_duration_seconds 10800 -> 32400
```

## 13. What this package did not do

- No pathologist review and no clinical claim of any kind. Ring fraction, envelope distance and mask
  IoU are engineering measures of geometry and pixel statistics.
- The `--scale 0` vs `--scale 1` ablation was not run (would have needed a second boot).
- The mucin/goblet channel was not measured separately, as F2_RESULT section 8 asked; only observed
  visually on the detail sheet.
- The preemption/resume path was not exercised — there was no preemption in 4 h 18 m.
- The two-pass comparison borrows another live package's retrieved outputs (SWEEP, `sweep-v1`), one of
  which was zero bytes locally. If SWEEP re-retrieves, these numbers should be recomputed.
- Only one training configuration was run: `cond=labels`, 3000 steps, batch 4, lr 1e-4. No sweep over
  adapter width, injection depth, conditioning type (`cartoon`, `labels+cartoon`) or step count, and
  no `--init cartoon` img2img ablation.
- Nothing was committed to git.