F2-TRAIN — the structure adapter, trained on an A100 and measured against the two-pass recipe

Work package F2-TRAIN (Phase 3) of the SGH programme, executed 2026-09-08 on spot A100 slot d (sgh-a100-slot-d, us-central1-f). Engineering evidence only: everything below is a measurement of geometry, pixel statistics and throughput. No clinical claim is made or implied, no pathologist saw any of these images, and nothing here says a synthetic field is diagnostically usable.

The VM was stopped at the end of the run and is TERMINATED. Nothing was committed to git.

1. Headline

The adapter does control layout, and by a wide margin — that was the open question and the rotation test answers it unambiguously. It does not clear the success bar in F2_RESULT.md section 8, which required both an in-range ring fraction and a smaller morphometric envelope distance than the two-pass recipe on the same cartoons. It clears the second condition only for intestinal metaplasia.

2. Run timeline (all times UTC, 2026-09-08)

Preemptions: none. The spot instance ran 4 h 18 m uninterrupted, so the auto-resume path (resuming training from … in steps.log) was never exercised in this run. It was exercised by FLEET's own preemption tests, not here — this package cannot claim to have re-verified it.

fleet/slots.json change (as instructed)

defaults.deadline_seconds and defaults.max_run_duration_seconds were both raised from 10800 to 32400 before up. Both are still 32400 in the file. This lengthens only the backstop; the sweep slots call down explicitly, so a longer backstop cannot leave them running.

3. What failed, and the fix

FAILURE 1 — no scipy on the VM (17 s into the first launch). segment_real.py imports scipy.ndimage and, deeper in, skimage.feature.peak_local_max / skimage.segmentation.watershed. The PixCell asset-root venv is a generation environment: numpy 1.26.4, pillow, torch 2.9.1+cu129, diffusers, transformers, timm, peft — and no scipy, no scikit-image. Neither does the system python. This was invisible to the CPU smoke test, which reads an already-built manifest and never runs the segmenter, and invisible to F2-PAIRS, which ran the segmenter on the Mac where both are installed.

Fixed on the VM with pip install --no-input "numpy==1.26.4" scipy scikit-image, pinning numpy so the resolver could not upgrade it out from under torch. Result: scipy 1.17.1, scikit-image 0.26.0, imageio 2.37.4, lazy-loader 0.5, tifffile 2026.3.3; numpy stayed 1.26.4 and torch.cuda.is_available() stayed True. segment_real.py --selftest then passed. The change lives on slot d's boot disk only, not in the golden image sgh-pixcell-a100-20260908 — any other slot that runs this package will hit the same error. Either add the two wheels to the image or ship a pip install line in worker.sh.

BUG FOUND AND FIXED BEFORE LAUNCH — the inference donor path. worker.sh's inference planner read donors.json (which holds bare file names such as hiesd-28f9b423__hpylori_gastritis__34688_2112__mpp0.25.png) and passed the bare name straight to infer_adapter.py --donor. Image.open would have resolved it against the run root and raised FileNotFoundError on every one of the 64 generations — and because the planner's loop uses subprocess.run(..., check=False), all 64 would have failed silently and the worker would still have written WORKER_DONE with exit 0 and zero PNGs. Patched in packages/f2-adapter/worker.sh to resolve REFS_ROOT/<category>/<file>, to log MISSING_DONORS if a path does not exist, and to match the category longest-key-first. Verified on the VM before launch that all four donor-0 files exist. This is the same class of bug as FLEET gotcha 8: a step that "succeeds" having done nothing.

Not fixed (deliberately): the per-image model reload. The planner spawns one infer_adapter.py process per image, so PixCell + VAE + UNI2-h are loaded 64 times. Each image cost 79 s wall of which 46.3 s was generation — roughly 35 minutes of the run, about USD 1.2, spent re-loading weights. Rewriting the loop mid-run risked the whole 2.5 h of training, so it was left alone. A single-process driver that loads once and loops is the obvious fix for the next run.

4. Training

Config as shipped (adapter-train/config.json): cond=labels (9 soft one-hot channels at 128x128), crop 1024, batch 4, grad-accum 1, lr 1e-4 cosine with 100 warmup steps, weight decay 0.01, grad-norm clip 1.0, CFG dropout 0.10, conditioning dropout 0.10, bf16, gradient checkpointing on, 3000 steps, checkpoint every 500, validate every 250, seed 0.

• Adapter: StructureAdapter width=320 blocks=4 injections=2, 8.25M trainable parameters (trunk 7.51M + two zero-init heads of 0.37M), fp32 33 MB, injected at blocks [0, 14] of 28.
• Base model: PixCell DiT 608.1M params, sample_size=128, cast bf16 directly from the fp32 checkpoint; SD3 VAE fp16; UNI2-h fp32. The UNI check in the trainer confirmed the tokens it computes are bit-identical to pixcell_generate.uni_tokens (max abs diff 0.000e+00).
• Split: val_split: by-slide as instructed, because the 22 held-out fields are not on the VM. Three whole training slides were held out — hiesd-324885ef, hiesd-9ded7e41, hiesd-da8667dc — leaving 64 train fields / 32 val fields, i.e. 512 train crops and 128 val crops per epoch. 3000 steps at batch 4 is 23 epochs.
• Throughput: 2.61 s/step steady state, 2.74 s/step including validations and checkpoints; 3000 steps in 8246 s (2 h 17 m).
• VRAM: max_memory_allocated 6.86 GB, 7.5 GB reserved (nvidia-smi 7660 MiB of 40960). That is at the very bottom of F2_RESULT's 11–15 GB estimate: batch 16 would have fitted, and the run was encoder-bound rather than memory-bound.
• Gradients: grad_norm median 0.0150, min 0.0032, max 0.967 — never zero, so the zero-init heads did leave zero (failure criterion 3 does not apply).

Loss curve (packages/f2-adapter/retrieved/loss.png)

Per-step training loss is dominated by the random diffusion timestep, so it scatters between 0.0002 and 0.42 throughout and its trend is only visible in the smoothed curve; the validation loss (fixed crops, fixed noise, fixed timesteps linspace(50, 950)) is the number that matters. It fell 0.1641 -> 0.1594 (-2.9% relative) and was essentially flat from step 1500 onward (0.1596 -> 0.1594 over the last half of the run). The cosine schedule reached lr 0 exactly at 3000.

Read honestly: the adapter learned something the frozen model did not already know, monotonically and without instability, but the epsilon-MSE it improves is dominated by the frozen DiT's already-good prediction, so a 2.9% drop is a weak signal on its own. The layout measurements in section 6 are the evidence that it learned the right thing; the loss alone would not carry that claim.

5. Generation

64 images at 4096x2048, 21 overlapping 1024 windows on one shared latent, 20 DPMSolver++ steps, CFG 1.5, --init noise, seed 51001, adapter scale 1.0, checkpoint step_003000, donor 0 of each category from donors.json:

Worth flagging, because hiesd-324885ef is one of the three slides the by-slide split removed: the two categories generated from a slide the adapter never saw are IM (its best category) and mixed (its worst). Two categories is not enough to say anything about donor generalisation, but the pairing should not be read as evidence either way.

Cost per image: 46.3 s of generation (120 denoiser calls), remarkably constant — sd across the 64 images was 0.023 s, min 46.27 s, max 46.38 s — plus ~33 s of process startup and model loading. 64 images in 92 minutes.

6. Evaluation

All local, all on the retrieved PNGs, with research/sgh-synthetic-histopathology/.venv/bin/python. Ring topology is code/ring_topology.py with config/sydney_qa_v1.json (md5 aa8cdf2b…, byte-identical to code/sydney_qa_v1.json). Morphometrics is morphometrics/morphometrics.py scored against the stored morphometrics/envelope.json through calibrate.score_rows, i.e. the same instrument and the same fitted envelope C1 calibrated — nothing was re-fitted here.

The two-pass comparison uses packages/sweep-v1/retrieved/*/out/a1/pass2/<cat>/<stem>.png — the SWEEP package's own base arm (start_index 6, seed 51002) on the same 60 a1 cartoons, so this is a like-for-like comparison rather than the cross-cartoon fullset_pass2 numbers. Two caveats: 59 of 60 were usable (intestinal_metaplasia_s14_oblique.png was zero bytes in that package's local copy — a truncated retrieval, not a generation failure; the sweep's own steps.log reports failed=0), and the donor matches only for the s11 cartoons: the A1 sweep rotates donors by seed (s11->donor 0 … s15->donor 4) while this run used donor 0 throughout. Table T5 is the strictly donor-matched subset. The fullset_pass2 medians from morphometrics/scores.csv are quoted in section 7 for continuity with C1, but they are a different cartoon set.

T1 ring_with_lumen_fraction

T2 morphometric envelope distance (Mahalanobis, per-category real-train envelope)

T3 layout adherence: IoU(output ring mask, cartoon lumen mask)

T4 rotation test (one cartoon per category, both renders)

T5 donor-matched subset (s11 cartoons only; adapter and sweep both used donor 0)

T6 per-image correlation with the cartoon

• Pearson r(cartoon ring fraction, adapter ring fraction) over 60 cartoons = 0.513
• Pearson r(cartoon ring fraction, two-pass ring fraction) over 59 cartoons = -0.045

T7 what the envelope distance is actually made of

Median per category, adapter vs the two-pass base arm vs the real held-out fields:

The three features that dominate the worst_features column of adapter-scores.csv are the same in every category and in the same direction: nuc_density_per_mm2 = +, epi_nuc_density_per_mm2 = +, and (for IM) median_ring_area_um2 = +. The adapter's single systematic defect is nuclei: it makes 1.4x to 2.1x too many of them and each is 20–40% too small. Ring size is close to real; ring density is 25–70% high, which is a knock-on of the same thing. The two-pass recipe, which starts from a real image's latent, gets nuclear density almost exactly right and loses on geometry instead.

7. Reading the tables

Ring topology (T1). The a1 two-pass base arm collapses on this cartoon set — 0.06 to 0.34 lumen-bearing ring fraction against a real held-out 0.71–0.85. The adapter is dramatically closer on every category: gaps of +0.016 (IM), +0.068 (normal), +0.089 (mixed), +0.243 (gastritis) against the two-pass's -0.46 to -0.68. Note this is not the fullset_pass2 picture that C1 recorded (normal 0.726, gastritis 0.475, IM 0.917, mixed 0.726 on the fullset cartoons); the a1 cartoons behave very differently under the two-pass recipe, which is SWEEP's finding to explain, not this package's. Against fullset_pass2 the adapter is still closer to the real mean for normal, gastritis and IM, and further for mixed.

The adapter's failure direction is a consistent overshoot: it makes tissue whose rings almost always contain a pale lumen (gastritis 0.986, i.e. 62.6 of 63.3 detected rings on the average field). Real tissue is messier. Section 8's condition 1 is satisfied on the letter — every category's mean lies inside the real held-out min–max — but that range spans 0.13–1.00 for IM and 0.30–1.00 for gastritis on 4–6 fields, so "inside the range" is a weak test and should not be read as "matches real".

Morphometrics (T2). Paired on the same cartoon, the adapter is further from the real envelope than the two-pass base arm for normal (3/15 wins, median +1.37), gastritis (3/15, +1.26) and mixed (0/15, +1.94), and closer for intestinal metaplasia (9/14, -0.23). The donor-matched s11 subset (T5) tells the same story: IM 3/3 to the adapter, mixed 0/3, normal 0/3, gastritis 1/3. Neither method sits inside the real held-out band often: the adapter is inside for 0% of fields in three categories and 13% in gastritis; the two-pass base arm is inside for 0% (normal), 0% (mixed), 21% (IM) and 60% (gastritis) — the one place the two-pass is clearly the better of the two.

Layout (T3, T4, T6). This is where the adapter wins and the win is not marginal:

• IoU between the output's ring mask and its own cartoon's lumen mask is 0.26–0.56, against a chance level of 0.03–0.06 measured two independent ways (same label map rotated 180, and a different cartoon of the same category).
• The same measurement on the two-pass outputs is 0.10–0.13 — above chance, as expected since pass 1 is img2img from the cartoon, but 2.6x to 5.4x lower than the adapter's.
• The rotation test does exactly what F2_RESULT section 8 asked: for the rotated output the overlap with the rotated label map beats the overlap with the unrotated one by 5.5x–12.3x, and the unrotated output shows the mirror-image separation of 4.8x–15.0x. H04's failure mode does not reproduce.
• Per-image, the adapter's ring fraction tracks its cartoon's ring fraction at Pearson r = 0.513 over 60 cartoons; for the two-pass outputs the same correlation is r = -0.045, i.e. no relationship at all.

The --scale 0 ablation of failure criterion 3 was not run — it would have cost a second VM boot after down, and the evidence already on hand is strictly stronger: non-zero grad_norm at every one of 3000 steps, a validation loss that moved off the zero-init value, and outputs whose geometry tracks which label map was supplied (rotating it moves the tissue). An adapter doing nothing cannot produce T3 or T4.

8. Contact sheets

packages/f2-adapter/retrieved/adapter-sheet.png (3790x1950) — one row per category, columns cartoon label map | adapter from noise | adapter with the label map rotated 180 | two-pass on the same cartoon, each cell the whole 4096x2048 field downscaled so that layout is comparable. packages/f2-adapter/retrieved/detail-sheet.png (3096x1082) — the same IM and gastritis cartoons at native 0.25 um/px, 256x128 um windows, adapter | two-pass | a real held-out field.

What I see, honestly.

At field scale the adapter columns clearly carry the label map. In the gastritis row the label map's dense, gland-free patch at lower-centre-right appears as a correspondingly dense pale region in the from-noise output, and in the rotated column that region has moved to the upper left; in the IM and mixed rows the vertical crypt columns of the label map appear as vertical epithelial columns in the output, and rotate with it. The normal row is the hardest to judge by eye because the label map is a near-uniform carpet of round gland profiles — but that is also where the IoU is highest (0.50). The two-pass column is not obviously related to the label map in any row.

At native resolution the ranking reverses. The two-pass crops have convincing one-cell-thick epithelial bands with elongated, basally-oriented nuclei, visible goblet-type vacuoles in the IM crop and stromal collagen strands; the real held-out crops have all of that plus crisply defined chromatin and, in the gastritis field, red blood cells in a vessel. The adapter crops are grainier and flatter: nuclei are small, round, soft-edged and scattered fairly evenly rather than being organised into a polarised band at the edge of a gland, the cytoplasm is a washed pink with little internal texture, and there are no red cells. The pale holes are the right size but their walls are made of a crowd of small nuclei rather than a palisade. This is the same defect the morphometrics found — too many, too small nuclei — and it is visible without any measurement.

So: the adapter has learned where to put tissue and has not yet learned what the tissue looks like up close, while the two-pass recipe is the other way round.

9. Verdict against F2_RESULT.md section 8

Success required both conditions. It is met for intestinal metaplasia only, and even there condition 2 is a small margin (median -0.23 Mahalanobis, 9 of 14 cartoons).

The "weaker but still reportable" outcome named in section 8 — the adapter fixes gastritis, where the two-pass recipe is worst — is half true and should be stated carefully. On ring topology the adapter moves gastritis from 0.06 (a1 two-pass) or 0.475 (fullset two-pass) to 0.986 against a real 0.743: it stops undershooting and starts overshooting, by a comparable margin. On morphometrics it makes gastritis worse. Calling that "fixed" would be wrong.

Failure criteria — none is tripped:

Overall. Phase 3's question was whether a small zero-init control branch on a frozen PixCell DiT can make a from-noise generation obey a procedural label map. The answer measured here is yes for geometry, no for texture: the layout signal is strong, unambiguous and reproducible across all four categories and 60 cartoons, and the resulting fields are further from the real morphometric envelope than the two-pass recipe in three categories out of four because the adapter over-produces small nuclei. This is a promising negative: the hard part (control) works, and the part that failed is a distribution-matching problem with obvious next moves.

The honest caveat from F2_RESULT section 8 still stands and is now sharper. The adapter was trained on segmenter output, not on ground truth, so it learned "epithelium wherever segment_real.py said epithelium". The programme cartoons' label maps are cleaner than that training conditioning, and the adapter's response to a clean map is to produce a lumen in nearly every ring — gastritis 0.986. Some of the overshoot is the adapter, and some is a train/inference distribution shift the segmenter created. This run cannot separate the two. The mucin channel specifically was not measured separately here (no goblet-vacuole detector exists yet); the detail sheet shows the adapter's IM crops lack convincing goblet vacuoles, but that is a visual impression, not a measurement.

10. Exact commands

cd [local]/sgh-program-20260908
P=[local]/python

# 0. before anything: raise both fleet backstops from 10800 to 32400 in fleet/slots.json
#    (deadline_seconds and max_run_duration_seconds), and fix the donor path in the package worker.

# 1. local pre-flight, free: the package's own 27-check CPU smoke against F2-PAIRS' manifest
$P packages/f2-adapter/adapter-train/run_cpu_smoke.py --manifest pairs/data/manifest.json --out-dir /tmp/f2-smoke

# 2. the run (fleet.sh run == up + stage + launch + watch; done as four calls here so that a
#    staging failure could be seen before committing to a multi-hour watch)
cd fleet
./fleet.sh up d
./fleet.sh stage  d ../packages/f2-adapter
./fleet.sh launch d ../packages/f2-adapter
./fleet.sh watch  d                       # backgrounded; polls 60 s, recovers preemptions

# the scipy repair, on the VM, after launch #1 failed
./fleet.sh ssh d '[local]/pip \
    install --no-input "numpy==1.26.4" scipy scikit-image'

# supervision, every ~2 minutes
./fleet.sh ssh d 'R=[local]/f2-adapter; tail -3 $R/evidence/train.log; nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader'

# 3. retrieve, verify, stop
./fleet.sh ssh d 'sudo -n systemctl stop sgh-fleet-stop.timer'   # so the 20-min backstop cannot race a 1.5 GB copy
./fleet.sh retrieve d [local]/retrieved
./fleet.sh down d && ./fleet.sh status

# 4. evaluation
cd ..
$P packages/f2-adapter/eval/plot_loss.py --log packages/f2-adapter/retrieved/out/adapter/loss_log.jsonl \
   --val packages/f2-adapter/retrieved/out/adapter/val_loss.json \
   --out packages/f2-adapter/retrieved/loss.png --table packages/f2-adapter/eval/loss-table.md
$P packages/f2-adapter/eval/cartoon_rings.py                      # ring metrics of the 60 a1 cartoons
$P packages/f2-adapter/eval/f2_eval.py --images packages/f2-adapter/retrieved/out/adapter-infer \
   --labels packages/f2-adapter/cartoons/a1 --out packages/f2-adapter/eval/results --jobs 10
$P packages/f2-adapter/eval/layout_iou.py packages/f2-adapter/retrieved/out/adapter-infer \
   packages/f2-adapter/cartoons/a1 packages/f2-adapter/eval/results/layout-iou-adapter.csv
# the same three on the sweep's two-pass a1 outputs, collected into eval/twopass-a1/ as symlinks
$P packages/f2-adapter/eval/f2_eval.py   --images packages/f2-adapter/eval/twopass-a1 \
   --labels packages/f2-adapter/cartoons/a1 --out packages/f2-adapter/eval/results-twopass --jobs 10
$P packages/f2-adapter/eval/layout_iou.py packages/f2-adapter/eval/twopass-a1 \
   packages/f2-adapter/cartoons/a1 packages/f2-adapter/eval/results-twopass/layout-iou-twopass.csv
$P packages/f2-adapter/eval/final_tables.py > packages/f2-adapter/eval/final-tables.md
$P packages/f2-adapter/eval/make_sheet.py --images packages/f2-adapter/retrieved/out/adapter-infer \
   --labels packages/f2-adapter/cartoons/a1 --twopass packages/f2-adapter/eval/twopass-a1 \
   --out packages/f2-adapter/retrieved/adapter-sheet.png

11. Cost and powered time

Reconstructed from fleet/state/d/powered.log (UP 1788876699 -> DOWN 1788892172) and cross-checked against the orchestrator log.

Where the 258 minutes went: 5 boot/stage, 8 segmentation, 0.3 smoke, 137 training, 92 generation, 9 retrieve, 1 shutdown, ~5 of orchestration slack. Two obvious savings for a rerun: ~35 min of the generation phase is the 64 redundant model loads (one process per image), and the run used 6.9 GB of 40 GB VRAM at batch 4, so batch 12–16 would cut the 137 training minutes substantially at the same step count. The other four slots were untouched by this package; a, b and c were RUNNING under the concurrent SWEEP package when this one finished and are not this package's to stop.

12. Files

packages/f2-adapter/worker.sh                          patched: donor paths resolved under REFS_ROOT
packages/f2-adapter/retrieved/out/adapter-infer/       64 PNGs at 4096x2048 + 64 .json provenance (944 MB)
packages/f2-adapter/retrieved/out/adapter/             loss_log.jsonl, val_loss.json, summary.json,
                                                       train.log, train_config.json,
                                                       checkpoints/step_{000500..003000}/ (6 x 95 MB:
                                                       structure_adapter.safetensors 33 MB +
                                                       training_state.pt 66 MB), latest.txt
packages/f2-adapter/retrieved/out/smoke/               the on-VM smoke result
packages/f2-adapter/retrieved/out/SHA256SUMS           73 files, all verified locally after transfer
packages/f2-adapter/retrieved/evidence/                steps.log, exit_code.txt (0), runtime.txt,
                                                       build_pairs.log, smoke.log, train.log,
                                                       infer.log, infer_plan.json, WORKER_DONE
packages/f2-adapter/retrieved/loss.png                 loss / validation / VRAM+throughput curves
packages/f2-adapter/retrieved/adapter-sheet.png        label map | adapter | adapter rot180 | two-pass
packages/f2-adapter/retrieved/detail-sheet.png         native-resolution adapter | two-pass | real
packages/f2-adapter/eval/f2_eval.py                    ring + morphometrics + rotation test
packages/f2-adapter/eval/layout_iou.py                 layout adherence with two chance levels
packages/f2-adapter/eval/cartoon_rings.py              ring metrics of the a1 cartoons themselves
packages/f2-adapter/eval/plot_loss.py                  loss.png + loss-table.md
packages/f2-adapter/eval/make_sheet.py                 the contact sheet
packages/f2-adapter/eval/final_tables.py               every table in section 6
packages/f2-adapter/eval/results/                      adapter-ring.csv, adapter-features.csv,
                                                       adapter-scores.csv, rotation-test.csv,
                                                       layout-iou-adapter.csv, summary.json
packages/f2-adapter/eval/results-twopass/              the same instruments on sweep-v1's a1 pass-2 base arm
packages/f2-adapter/eval/twopass-a1/                   59 symlinks into packages/sweep-v1/retrieved/
packages/f2-adapter/eval/a1-cartoon-ring.csv           the 60 a1 cartoons' own ring metrics
packages/f2-adapter/eval/final-tables.md               sections 6's tables as generated
fleet/logs/d.log, fleet/logs/f2-train-poll.log         orchestrator and supervision logs
fleet/slots.json                                       deadline_seconds / max_run_duration_seconds 10800 -> 32400

13. What this package did not do

• No pathologist review and no clinical claim of any kind. Ring fraction, envelope distance and mask IoU are engineering measures of geometry and pixel statistics.
• The --scale 0 vs --scale 1 ablation was not run (would have needed a second boot).
• The mucin/goblet channel was not measured separately, as F2_RESULT section 8 asked; only observed visually on the detail sheet.
• The preemption/resume path was not exercised — there was no preemption in 4 h 18 m.
• The two-pass comparison borrows another live package's retrieved outputs (SWEEP, sweep-v1), one of which was zero bytes locally. If SWEEP re-retrieves, these numbers should be recomputed.
• Only one training configuration was run: cond=labels, 3000 steps, batch 4, lr 1e-4. No sweep over adapter width, injection depth, conditioning type (cartoon, labels+cartoon) or step count, and no --init cartoon img2img ablation.
• Nothing was committed to git.

Download public Markdown export