F2-PAIRS: (label map, cartoon, real) pairs and a runnable structure-adapter package

Phase 1, CPU only. Nothing was trained here; this package builds the dataset and the code the fleet runs in Phase 3. Engineering evidence only, no clinical claim.

Everything below lives under operations/research/sgh-program-20260908/pairs/.

1. Why this exists

Every SGH fine-tune so far trained the generator on real fields unconditionally, so the only thing it could learn was the marginal appearance of real tissue -- including the filled-in gland failure. The paired task we actually want is

label map (or cartoon) + appearance tokens -> real tissue

and that training set can be manufactured from real data: segment real fields into the tissue model's 9-label vocabulary, re-render those label maps with the programme's own renderer, and train the zero-initialised control branch from operations/research/pixcell-topology-slot-20260906/code/structure_adapter.py on the triples. The earlier attempt (H04) trained a tiny adapter on ONE pair and failed the rotation test; this package supplies 96 training fields + 22 held-out fields and a trainer that consumes them.

2. Segmentation (pairs/segment_real.py, 790 lines, self-contained)

Input a 4096x2048 RGB PNG at 0.25 um/px; output exactly what tissue3d.SliceResult carries: labels uint8 (the 9 ids, same order and palette), cell_id int32, nucleus_id int32, wall_t uint8 (0 apical .. 254 basal inside the wall, 255 elsewhere) and a meta dict. It is built so render_cartoon.render(res, ...) accepts it unchanged (Segmentation.as_slice_result() returns a real T3.SliceResult when tissue3d is importable).

Method, in order:

1. Stain. optical_density and reference_deconvolution are inlined copies of scripts/sydney_qa.py's (same H_REFERENCE/E_REFERENCE, same formula) so the file ships to the VM alone; --selftest asserts they are bit-identical to sydney_qa.py (it passes). tissue = OD sum above the sydney optical_density_floor 0.15 and mean RGB < 245.
2. Glass (0) vs lumen (1). Bright components (~tissue or mean RGB > sydney's lumen_brightness_floor 220) are labelled: connected to the image border and >= 2000 um^2 -> glass; enclosed, in 80..60000 um^2, with a 2 px ring that is >= 55% tissue -> lumen.
3. Nuclei. Hematoxylin above a percentile, opened, hole-filled, split by a distance-transform watershed (skimage, markers = local maxima >= 1.6 um apart), then filtered to sydney's nucleus area range 5..180 um^2 and elongation <= 6. Deviation from the brief: sydney's nuclear_hematoxylin_percentile 72 marks 28% of all tissue pixels. Used directly for nucleus instances it produces long slivers off smooth-muscle and collagen bundles, so percentile 72 is kept for the band construction (which is what ring_topology.py uses it for) and instances are cut at percentile 78 (nucleus_instance_percentile).
4. Epithelium band (2). ring_topology.py's construction -- the nuclear mask morphologically CLOSED with a one-cell-height (6 um) disc -- clipped to the geometric band (within 25 um of a gland-sized lumen or the glass boundary), plus an unconditional 6 um apical strip, the two bridged by a 10 um "wall closing", then closed 2 um, hole-filled and de-specked. Two deviations, both forced by what the images actually look like:
5. closing the raw percentile-72 mask at 6 um closes 86% of a 0.25 um/px gastric field into one blob, so the closing runs on the validated nucleus instances instead;
6. without the 10 um wall closing the band is only the nuclear palisade plus a thin apical rim and the pale mid-cytoplasm of tall foveolar/metaplastic cells falls into stroma (measured on the QA crops: gastritis epithelium 0.168 -> 0.226, IM 0.357 -> 0.496);
7. the band is seeded only by lumina >= 400 um^2, because a scatter of pale spaces in inflamed lamina propria otherwise each induces its own 25 um band and the field closes up into "epithelium" (measured: one mixed crop went epithelium 0.606 / stroma 0.006 before that fix, 0.392 / 0.177 after).
8. Goblet mucin (3). Pale (mean RGB > 205), round (sydney's roundness proxy >= 0.30), 45..230 um^2 blobs inside the band; plus first-pass "lumen" components in the same size range whose 3 px halo is >= 45% band. Small pale components whose halo is < 30% band are dropped as open spaces in loose lamina propria (they are not glands).
9. Stroma (5) / nuclei (4, 6). Remaining tissue is stroma; nuclei inside the band are epithelial (4), outside it stromal (6).
10. Vessel (7) / red cell (8). Red cells = small (12..90 um^2), round, strongly eosinophilic, low-hematoxylin, R-B >= 22 discs in stroma; vessels = pale elongated (axis ratio >= 3) stromal spaces 60..3000 um^2 plus a 1.5 um halo around red cells.
11. Instances. cell_id = geodesic watershed of the band seeded by its own epithelial nuclei (orphan band components get their own ids); nucleus_id = the watershed nucleus labels plus red-cell components; wall_t = d_apical / (d_apical + d_basal) across the band x 254.

Cost: 20.7 s mean per 4096x2048 field for the segmentation (p90 34.8 s) and 16.2 s for the cartoon re-render, single core, peak RSS 2.0 GB.

2.1 Visible failure modes (from pairs/qa-sheet.png, read and described honestly)

The sheet is 8 rows (2 fields per category, different slides where possible) x 3 columns (real | label map in a QA palette | cartoon re-render), 1024 px crops. The crop shown is the most gland-rich 1024 window of the field by construction, so its fractions are systematically higher in epithelium than the field means in section 3.

1. Goblet mucin is heavily under-called. Real segmented mucin is 0.004-0.009 of tissue; the procedural cartoons carry 0.049-0.056 for IM/mixed. Even the gland-rich QA crops only reach 0.009-0.011 mucin, and in row 8 (mixed, hiesd-324885ef) the large pale goblet cells along the surface are plainly visible in the real column and are mostly labelled lumen or stroma. Cause: a goblet vacuole that opens apically merges with the gland lumen into one bright component, and vacuoles below the 80 um^2 sydney lumen floor are shattered by the opening. This is the worst failure and it is not fixed. Consequence for training: the conditioning tells the adapter "pale here", not "goblet cell here" -- the geometry survives, the class does not. Any A1 target derived from the mucin column must be treated as a lower bound.
2. Gland epithelium with no visible lumen is labelled stroma. Row 2 of the sheet (normal, hiesd-40f3c0cf, oxyntic mucosa cut tangentially) is almost entirely packed glands, yet the label map calls it epithelium 0.154 / stroma 0.655 -- the band is seeded from lumina and this field has almost none (lumen 0.036 in a crop chosen to maximise gland content). This is why the category means for epithelium (0.19-0.23 of tissue) sit below what a pathologist would call the epithelial fraction of gastric mucosa (roughly 0.5-0.7). It is a structural limit of a lumen-seeded band, not a threshold that can be tuned.
3. Loose or oedematous lamina propria reads as lumen. The _reclassify_lumen rule removes the spaces whose halo is stroma (54-97 components per field in the fields inspected), but inflamed gastritis fields still contribute lumen that is not gland lumen. Real lumen p10-p90 spans 0.006-0.19 within a single category, an order-of-magnitude spread.
4. Glass vs lumen is only decided by border connectivity. An enclosed space between two surface folds is called lumen even when it is really outside the tissue. Glass is 0.5-6.4% of a field on average and its p90 is much higher.
5. The basal nuclear palisade often merges into a continuous dark ribbon rather than separate nuclei; the watershed under-splits elongated columnar nuclei. cell_id therefore under-counts epithelial cells (2756-3809 per mm^2, which is low for one-cell-thick walls).
6. Red cells are essentially never labelled (0.0001 of tissue) -- the colour rule is conservative and most visible red cells end up as vessel (0.014-0.019). Vessels are a crude pale-elongated-shape proxy, not an endothelial call.
7. Fixed in this package (recorded because the intermediate states are in the git-ignored history): goblet vacuoles were originally all classified as lumen; the band originally over-reached and swallowed the lamina propria; nucleus instances were originally cut at percentile 72 and produced fibre slivers.

3. Real label fractions per category (pairs/real_label_fractions.json, pairs/fraction-comparison.{json,md})

All 118 fields segmented, 0 failures. Fractions are of tissue (glass removed) so they compare with cartoons rendered surface=False. Cartoon column = the 20 existing fullset cartoons (operations/research/pixcell-fullset-20260908/cartoons/); re-run compare_fractions.py --cartoons ../cartoons/a1 when A1 delivers its set.

category               split      n    lumen      epi    mucin   stroma   nuclei   nuc/mm2   glass
normal                 real train  24    0.067    0.186    0.005    0.582    0.145      7759   0.031
normal                 real heldout 4    0.067    0.193    0.006    0.559    0.161      8198   0.004
normal                 cartoon      5    0.113    0.232    0.006    0.525    0.118         -   0.011
hpylori_gastritis      real train  24    0.070    0.189    0.004    0.557    0.162      8442   0.033
hpylori_gastritis      real heldout 6    0.082    0.209    0.006    0.529    0.157      7505   0.063
hpylori_gastritis      cartoon      5    0.113    0.232    0.006    0.434    0.208         -   0.011
intestinal_metaplasia  real train  24    0.082    0.230    0.007    0.505    0.161      7718   0.064
intestinal_metaplasia  real heldout 6    0.078    0.210    0.007    0.535    0.156      7887   0.022
intestinal_metaplasia  cartoon      5    0.112    0.185    0.056    0.511    0.130         -   0.011
mixed                  real train  24    0.086    0.204    0.006    0.530    0.161      8186   0.023
mixed                  real heldout 6    0.075    0.209    0.009    0.532    0.158      8257   0.014
mixed                  cartoon      5    0.112    0.191    0.049    0.451    0.191         -   0.011

Within-category spread is wide (p10-p90 of the field means, train split):

Nucleus density per mm^2 of tissue, split epithelial / stromal, against the tissue model's preset:

category               split      epi nuc  stroma nuc   p10-p90 stroma  epi cells  preset str/mm2
normal                 train         2426        5719     [2682, 8434]       2756            3500
hpylori_gastritis      train         2737        6121     [3400, 8121]       3345            9000
intestinal_metaplasia  train         2918        5199     [1358, 7918]       3809            4300
mixed                  train         2759        5847     [2164, 8730]       3439            8000

What this means for A1 (write it into the tissue-model targets):

• The cartoons make ~50% too much lumen. Real lumen is 0.067-0.086 of tissue in every category; the cartoons are at 0.112-0.113. Since the whole programme is chasing ring_with_lumen_fraction, a renderer that over-draws lumen makes the target easier for the generator to hit on the cartoon and harder to match against real fields. Lowering the tube lumen radius so the cartoon lands near 0.08 is a concrete, measurable A1 change.
• Category separation by area fraction does not exist. Lumen 0.067-0.086, epithelium 0.186-0.230, stroma 0.505-0.582, nuclei 0.145-0.162 -- the four categories are within one within-category standard deviation of each other on every coarse fraction. Whatever separates the categories in the real data is not area composition; it is goblet cells and the identity of the inflammatory infiltrate. A1 should not tune presets to separate categories on these numbers.
• The stroma_nuclei_per_mm2 presets are wrong in the discriminating direction. Measured stromal nuclear density is 5199-6121 /mm^2 in every category, with overlapping p10-p90 bands; the presets are 3500 (normal) / 9000 (gastritis) / 4300 (IM) / 8000 (mixed). Normal and IM are set too low and gastritis/mixed too high. Caveat: this measurement over-calls stromal nuclei in fields where lumen-less glands were labelled stroma (failure mode 2), so the true normal/IM numbers are probably a little lower than 5200-5700, not higher.
• Mucin is the one place the cartoon is plausibly right and the segmentation wrong (0.05 vs 0.007). Do not lower the cartoon's goblet_frac on the strength of this table.

4. QA sheet

pairs/qa-sheet.png (3072 x 8192) plus readable slices qa-sheet-rows1-3.png, qa-sheet-rows4-6.png, qa-sheet-rows7-8.png, and qa-sheet.json recording which crop of which field each row shows with its per-crop fractions. Regenerate with python make_qa_sheet.py. The label column uses a QA palette (glass mid-grey, mucin saturated cyan) because the tissue3d palette draws glass (245,245,245) and lumen (255,255,255) indistinguishably -- and glass/lumen confusion is exactly what the sheet has to expose.

Read and described in section 2.1. What the sheet gets right: gland rings with a pale lumen, a one-cell-thick epithelial band, a basal nuclear palisade at the outer edge of that band, and lamina propria with scattered nuclei -- i.e. the ring/lumen topology the whole programme is trying to impose is present in the manufactured conditioning.

5. Adapter training package (pairs/adapter-train/)

The task. conditioning = the segmented label map of a real 1024 crop, area-pooled to the 128x128 latent grid as 9 soft one-hot channels (--cond labels, default), or the re-rendered cartoon crop's VAE latent (--cond cartoon, 16 channels), or both (--cond labels+cartoon, 25 channels). Appearance = the UNI2-h tokens of the same real crop, 16 x 1536, exactly as in generation. Target = that real crop's SD3-VAE latent, 16 x 128 x 128, with the generator's convention ((encode(x).latent_dist.mode() - shift) * scale). Loss = epsilon-MSE under a DDPMScheduler with linear betas 1e-4..0.02 over 1000 steps (the trainer asserts the VM's scheduler_config.json agrees). CFG dropout 0.10 on the tokens (to the learned uncond_embedding) and 0.10 conditioning dropout to zeros, so unconditional generation still works after training.

The adapter. StructureAdapter(width=320, blocks=4, injections=2) = 8.25M trainable parameters (trunk 7.51M + 2 zero-init heads of 0.37M), fp32 33 MB; 8.43M for the 25-channel variant. Injected after pos_embed (block 0) and before block 14 by pure forward hooks -- the publisher's file is never edited. Zero-init means step 0 is bit-identical to the base model, and the smoke test asserts exactly that.

Crops. Random 1024x1024 crops of the 4096x2048 pairs, crops_per_field 8 -> 768 crops per epoch over 96 fields, resampled every epoch; 3000 steps at batch 4 is ~15.6 epochs. Validation uses a fixed evenly-spaced grid (4 crops per held-out field), fixed noise and fixed timesteps (linspace 50..950), so the validation loss is comparable across steps and runs.

Validation. --val-split heldout uses the 22 held-out fields (slides no SGH run has seen). When those are not staged on the VM, --val-split by-slide holds out 3 whole training slides instead; the trainer falls back to that automatically and says so in the log.

Memory on an A100 40 GB (estimate, extrapolated from the calibrated numbers in pixcell_lora/README.md -- the run's own loss_log.jsonl records max_mem_gb per step):

So batch 4 at 1024 fits comfortably with checkpointing (which config.json enables) and batch 8 would too. Throughput: ~1.5-3 s/step for the DiT plus ~1-2 s for the on-the-fly UNI and ~0.5-1 s for the VAE, i.e. 3-6 s/step and 2.5-5 h for 3000 steps (USD 5-11 at spot A100 2.12/h). If that is too slow, the encoders are the cost, not the DiT: switch to a fixed crop grid (--crops-per-field 8 is already deterministic in the val dataset) and precompute the latents and tokens once with scripts/pixcell_lora/precompute_pixcell_conditioning.py.

Inference (infer_adapter.py). pixcell_generate.denoise_canvas re-expressed with one addition: before each transformer call the adapter is conditioned on the label window that matches the latent window. The generator interleaves [uncond, cond] per window, so the adapter features are built in the same interleaved order and handed to the attachment directly rather than through its CFG shortcut. 21 overlapping 1024 windows on one shared 4096x2048 latent, 20 DPMSolver++ steps, CFG 1.5, starting from noise (--init noise, default); --init cartoon gives the img2img ablation and --rotate180 runs the rotation test on the conditioning only.

6. CPU smoke test (pairs/adapter-train/smoke.log, smoke_result.json)

python run_cpu_smoke.py -- 27 passed, 0 failed in 5 s on the Mac, no PixCell weights.

1  dataset
  [PASS] real crop is (1024,1024,3) uint8 -- (1024, 1024, 3)
  [PASS] UNI patches are (16,3,224,224) fp16
  [PASS] conditioning is (9,128,128)
  [PASS] label channels are a partition (sum == 1 per latent cell) -- min 1.0000 max 1.0000
  [PASS] rotating the label map rotates the conditioning
  [PASS] conditioning carries structure (>=3 labels present)
  [PASS] collate stacks a batch
2  adapter zero-init and injection
  [PASS] zero-init adapter leaves the base model bit-identical -- max |d| 0.000e+00
  [PASS] both injection points fired -- [1, 1]
  [PASS] a non-zero adapter changes the output -- max |d| 3.647e-01
  [PASS] the output depends on WHICH conditioning is passed -- max |d| 3.430e-01
3  two training steps with stubs
  [PASS] 2 optimiser steps ran        val 1.9180 -> 1.9177 -> 1.9176, train 1.6629 / 1.8815
  [PASS] final loss is finite
  [PASS] validation loss recorded and finite
  [PASS] loss log written
4  checkpoint round trip
  [PASS] checkpoint directory exists
  [PASS] checkpoint records the conditioning -- {"cond": "labels", "cond_channels": 9}
  [PASS] state dict reloads exactly
  [PASS] loaded adapter produces (1, tokens, hidden) residuals
  [PASS] residuals are finite
5  inference plumbing (1536x1024 canvas, 2 windows, 2 steps)
  [PASS] window grid for a 1536x1024 canvas is 2 windows -- [(0, 0), (0, 512)]
  [PASS] per-window conditioning is (n,9,128,128)
  [PASS] latent has the canvas shape -- (1, 16, 128, 192)
  [PASS] latent is finite
  [PASS] the adapter was injected on every denoiser call -- [2, 2] vs 2 calls
  [PASS] decoded image is the canvas size -- (1536, 1024)
  [PASS] rotating the conditioning changes the output (rotation test wiring) -- max |d| 4.676e+01

The stub DiT reproduces the interfaces the adapter hooks depend on (pos_embed, transformer_blocks[k] taking hidden_states first, caption_projection.uncond_embedding, config.patch_size/sample_size, learned-sigma output). The real DiT is loaded only on the VM.

Two bugs the smoke test did not catch and that were found by reading, both of which would have crashed step 1 on the GPU: pixcell_generate.load_pipeline hard-codes fp16 while the config trains in bf16 (fixed by load_pixcell(), which casts the fp32 checkpoint once and keeps the missing-key contract), and the frozen VAE/UNI must be fed their own parameter dtype (fp16 and fp32 respectively), not the training dtype. The stubs are single-dtype so they cannot see this.

7. What the fleet runs (pairs/STAGE_LIST)

Total upload ~0.4 MB of code. The 96 training fields are already on the VM at [local]/train<cat>/, so build_pairs_vm.sh segments them in place (~6 min on 8 cores) and only the 33 MB of label PNGs is produced there; the 1.2 GB of cartoons is never uploaded and is only re-rendered on the VM if a --cond cartoon arm is wanted. The 22 held-out fields are not on the VM: either pull them from IOFlood directly or run --val-split by-slide. Exact commands:

# on the VM, RUN_ROOT=[local]/f2-pairs
#            ASSET_ROOT=[local]/pixcell-reference-20260905-v1
bash worker.sh $RUN_ROOT $ASSET_ROOT

which runs, in order: GPU runtime check -> build_pairs_vm.sh -> run_cpu_smoke.py -> train_pairs_adapter.py --config adapter-train/config.json (3000 steps, checkpoint every 500) -> infer_adapter.py from noise on every cartoons/a1/*_labels.png with that category's donor 0, plus one --rotate180 per category. Evidence lands in evidence/ (steps.log, exit_code.txt, runtime.txt, train.log, loss_log.jsonl, val_loss.json, smoke.log, infer.log) and out/SHA256SUMS; the VM self-shuts 3 minutes after WORKER_DONE. Before starting: refresh sgh-pixcell-guest-deadline-epoch and set --max-run-duration to at least 8 h. On preemption, restart and pass --resume out/adapter/checkpoints/step_XXXXXX to the training step; the segmentation step is a no-op once data/manifest.json exists.

Pull back ~120 MB: the final checkpoint (33 MB), the logs, out/adapter-infer/*.png (+ their .json provenance) and out/smoke/smoke_result.json.

8. What counts as success and what counts as failure

Judged on the a1 cartoons, from noise, against the pass-1/pass-2 outputs on the same cartoons (current pass-2 vs real held-out ring_with_lumen_fraction: IM 0.917/0.710, mixed 0.726/0.726, normal 0.726/0.854, gastritis 0.475/0.743).

Success requires both of: 1. ring_with_lumen_fraction of the adapter's from-noise output lands inside the real held-out range for its category (measured with the same pixcell-fullset-20260908/code/ring_topology.py and config/sydney_qa_v1.json), and 2. the morphometric distance to the real envelope (C1's measure) is smaller than the two-pass output's on the same cartoon and donor.

Condition 1 alone is not enough: the two-pass recipe already overshoots on IM (0.917 vs 0.710) and undershoots on gastritis (0.475 vs 0.743), so "in range" and "closer on morphometrics" have to hold together. A weaker but still reportable outcome is that the adapter fixes gastritis specifically, where the two-pass recipe is worst and pass 2 makes pass 1 worse.

Failure is any of: 1. Layout ignored. The adapter's output has no relationship to the label map: ring density or lumen fraction uncorrelated with the cartoon's across the 20 a1 cartoons. 2. Rotation test. --rotate180 produces an output whose gland layout is not the 180-degree rotation of the un-rotated output (compare the two label-space overlaps, or simply put the two PNGs side by side). This is exactly how H04 failed with one training pair, and it is the cheapest discriminator: it costs two generations. 3. Adapter does nothing. --scale 0 and --scale 1 give visually and metrically identical fields, i.e. the zero-init heads never left zero. The grad_norm column of loss_log.jsonl catches this during training. 4. Validation loss does not separate. The held-out epsilon-MSE at step 3000 is not below the step-0 (zero-init == base model) value. Step 0's number is recorded first thing in the run for exactly this comparison.

An honest caveat to carry into Phase 4: the conditioning the adapter is trained on is this segmentation, with the failure modes in section 2.1. If the adapter learns to reproduce "epithelium wherever the segmenter said epithelium", it inherits the segmenter's lumen-less-gland blind spot. A programme cartoon's label map is cleaner than the training conditioning, which is a train/inference distribution shift in the adapter's favour for lumens and against it for mucin -- report the mucin channel's behaviour separately.

9. Files

pairs/segment_real.py                 the segmenter (--selftest passes: stain maths == sydney_qa.py)
pairs/build_pairs.py                  multiprocessing driver (118 fields, 7 workers, 658 s, 0 failures)
pairs/build_pairs_vm.sh               the same on the VM against refs/train
pairs/make_qa_sheet.py                the QA sheet
pairs/compare_fractions.py            real vs procedural-cartoon fractions and nucleus densities
pairs/data/{train,heldout}/<cat>/     96 + 22 x {_labels.png, _cartoon.png, _pair.json}  (1.3 GB:
                                      33 MB of label maps, 1.2 GB of cartoons)
pairs/data/manifest.json              every field, its paths, fractions, counts, densities, timings
pairs/real_label_fractions.json       per split/category aggregates (A1's ground truth)
pairs/fraction-comparison.{json,md}   section 3's tables
pairs/qa-sheet.png + -rows*.png + .json
pairs/build_pairs.log                 the run log of the build
pairs/STAGE_LIST                      exactly what to scp and how to launch
pairs/adapter-train/                  the training package (section 5) + smoke.log, smoke_result.json

Nothing was committed to git.

Download public Markdown export