GEN-CODE result, 8 September 2026

Work package GEN-CODE of the SGH programme (operations/research/sgh-program-20260908/). Everything below is engineering evidence: code, tests and job manifests. No image was generated (no GPU in this package), nothing here is clinically validated, and nothing here is "solved".

Everything runs from the programme root. Nothing in research/sgh-synthetic-histopathology/ was edited; the canonical generator, sydney_qa.py, sydney_qa_v1.json and pixcell_stain_match.py were copied into code/ and only the copies were changed. Nothing was committed to git.

1. What was built

2. Generator changes (code/pixcell_generate.py)

Line numbers are in the new file.

The canonical variable token_source (a PIL image) was renamed token_image so the record key token_source (a string) is unambiguous; no behaviour changed.

Job schema additions

Precedence, asserted by a test: token_file > token_library > token_blend > token_reference > reference. token_reference behaviour is untouched.

Record additions

Always: token_source ("reference", "token_reference", "token_library", "token_blend", "token_file", or null for reconstruct). Then, only for the source that won:

• token_file, token_file_sha256
• token_library, token_library_entries, token_library_same_category, labels, labels_sha256, and token_library_picks: one entry per window with origin, library_file, library_origin, library_category, library_entry, distance, window_composition, library_composition
• token_blend: one entry per source with source, weight, sha256, token_sha256; plus token_blend_weight_sum

Every existing key keeps its name and value. token_sha256 is still the per-window list, now hashed from whatever grid was actually handed to the denoiser.

3. Composition vector, and the one deviation from the brief

[lumen, epithelium, stroma, stroma_nucleus_density_norm], computed two ways that must land on the same scale: from a cartoon label map (pixcell_generate.label_composition, ids 1 / 2,3,4 / 5,6, with 0, 7, 8 ignored and density = label 6 / (label 5 + label 6)) and from a real RGB window by classical segmentation (build_token_library.window_composition). Both divide the density by 0.35 and clip.

The brief said to copy ring_topology.py's epithelial band, which closes the 72nd-percentile hematoxylin mask with a disk of CLOSING_RADIUS_UM = 6.0. Measured, that collapses the descriptor. On 8 real fields, 9 windows each, mean stroma fraction at 6 um:

At 6 um the band swallows the lamina propria and the four-vector degenerates to "lumen fraction" alone (a 4-field, 84-window smoke library had stroma between 0.000 and 0.007). At 2 um the same windows span lumen 0.077-0.485, epithelium 0.429-0.660, stroma 0.025-0.458. Everything else is ring_topology.py and sydney_qa_v1.json unchanged: the OD tissue floor, the 72nd-percentile hematoxylin threshold, the disk closing, the 100 um^2 speck drop, the lumen brightness floor. --epithelium-closing-um 6 reproduces the literal ring_topology band. The radius is recorded in build-meta.json and in composition-summary.md.

Donor side vs cartoon side, per category (donor: 3 fields x 9 windows at 2 um; cartoon: the 15 non-empty windows of an s11_across render from code/render_cartoon.py):

Read honestly: IM overlaps well; gastritis and normal keep an offset on stroma, because the closing still absorbs an inflamed or densely glandular lamina propria. A2 searches within one category, so a constant per-category offset shifts every candidate equally and the ranking is driven by the window-to-window spread, which is large on both sides. Note also that the current renderer's four categories differ only in goblet fraction and stromal nuclear density, so on the cartoon side the first three coordinates are nearly identical across categories - the vector discriminates windows, not categories, which is exactly what A2 needs. None of these numbers means anything clinically.

Two other recorded coarsenesses: region (the composition denominator) is the tissue mask closed at 12 um and hole-filled, so gland lumens count as field rather than as background glass; and disk morphology is computed by Euclidean distance transform rather than ndimage.binary_closing, which is the same disk at O(N) instead of O(N pi r^2) (2 s -> 20 ms per window at r = 24 px) and differs only in not eroding against an implicit background ring at the array border.

4. Tests

code/test_generate_cpu.py, run with research/sgh-synthetic-histopathology/.venv/bin/python (torch 2.9.1, numpy 2.5.2, CPU). Stub encoder returns deterministic pseudo-tokens from the window's pixel bytes; stub pipeline does no diffusion and returns the reference image.

PASS  old-schema jobs are byte-for-byte unchanged against the canonical generator
      4 old-schema jobs identical (bytes + record) to canonical pixcell_generate.py
PASS  token_library picks the nearest composition per window
      2 windows -> lumen.png (d=0.3536), stroma-dense.png (d=0.6124); 5-entry library
PASS  token_blend weights and 'self' resolve to the weighted sum
      alpha=0.75 self+donor exact to 1e-6 on 2 windows; alpha=1.0 equals self; size mismatch refused
PASS  token_file broadcasts one grid to every window, both accepted shapes
      (1,16,1536) and (16,1536) both broadcast to 4 windows; wrong shape refused
PASS  precedence is token_file > token_library > token_blend > token_reference > reference
      token_file > token_library > token_blend > token_reference > reference
PASS  every record carries the provenance fields evaluation needs
      4 records: ['canvas', 'category', 'elapsed_seconds', 'extra']... plus source-specific keys, all present
PASS  resume skips ids already in records.jsonl and adds nothing on a rerun
      2 jobs -> rerun adds 0 records -> manifest grown by 1 adds exactly 1
PASS  resolve_jobs resolves PASS1 against out/ and drops jobs whose inputs are missing
      PASS1 -> out/ absolute, 2 of 3 dropped (reference, token_library), empty manifest exits 3
PASS  split_manifests keeps every pass-2 job on its pass-1 slot and places every job
      2/4/7 slots: 335 jobs placed, 0 broken chains, worst weight spread <25%

9/9 passed

The first test is the strong one: it loads the canonical repo generator and this one as two modules, runs the same four old-schema jobs (img2img on the reference, img2img with token_reference, generate, reconstruct) through both under identical stubs, and demands identical output PNG bytes and identical records apart from exactly one new key, token_source.

The A2 test builds a library whose answer is arithmetic: entries at pure lumen, pure epithelium, pure stroma and nucleus-packed stroma, each with a token grid filled with its own constant, plus a decoy entry of another category on the pure-lumen point. A label map that is 3/4 lumen in window 0 and half nucleus-packed stroma in window 1 must pick lumen.png then stroma-dense.png, and the tokens handed to the denoiser must be those constants. A category with no entries raises rather than falling back silently.

5. Worker rehearsal (CPU, no GPU, no model weights)

worker.sh was rehearsed twice on this Mac with SKIP_GPU_CHECK=1 SELF_STOP=0 DEVICE=cpu and a shim that patches the real generator's encoder and pipeline.

1. Everything missing (no cartoons, no render_set.py, no donor root): 6 manifests, all skipped with a reason in evidence/steps.log, evidence/manifest-summary.json = {"ran":0,"skipped":6, "failed":0}, exit 0, WORKER_DONE written. A slot that arrives before its inputs does not fail.
2. Full chain with two fabricated cartoons, a 4-window token library, token stats and a local donor root: a1-pass1 (2) -> a1-pass2 (2, PASS1/ resolved to out/) -> a2-pass1 (2, library retrieval) -> a3-pass1 (4, token_file) -> d2-pass2 (8, blend on a1's pass-1 output) = 18 records, 18 PNGs, ran=5 skipped=0 failed=0. Every record's token_source equalled its extra.expected_token_source. Rerunning the same command added zero records (diff of records.jsonl empty) and rebuilt neither library nor stats.

build_token_library.py was also run for real on 4 local HiESD training fields (84 windows, --composition-only, no encoder) and token_stats.py on an 84-window library: shapes (1,16,1536), (8,16,1536), (1,16,1536), token-stats.json written with distances and cluster sizes. Composition timing is ~0.3 s per 1024 window on this CPU, so the full library (4 categories x 24 fields x 21 windows = 2016 windows) is roughly 10 CPU-minutes of segmentation plus 2016 x 16 UNI2-h forward passes on the GPU.

6. Manifests

jobs/ holds 15 manifests, 335 jobs, 335 distinct outputs, 0 duplicate ids, 0 unsatisfied PASS1/ references (python code/make_jobs.py --verify). Full table and dependencies in jobs/README.md.

At the fullset run's measured 40-60 s per 4096x2048 pass, 335 jobs is roughly 4-6 GPU-hours; split 4 ways, ~1-1.5 h per slot (split_manifests.py --slots 4 gives 82-87 jobs and weight 19064-19194 per slot, 0 broken chains).

7. Deviations from PLAN.md and the brief

1. Epithelial closing radius 2 um, not ring_topology.py's 6 um - measured, section 3. The brief's literal construction is one flag away (--epithelium-closing-um 6).
2. E2 is 15+15, not the table's 30+30. The ladder is levels 0-3 x 5 seeds and level 2 is the current renderer, so those five cells are already a1-pass1/a1-pass2 (intestinal_metaplasia_s*_across). The brief said exactly this ("fidelity 0,1,3 (f2 is a1)"); the full ladder is 20+20 counting the a1 cells.
3. B1 is 5+5+10, not the table's 10+10+10. The brief fixes five b1 cartoons (hpylori_gastritis_s21..25_oblique).
4. d1's donor0 arm duplicates a1's pass-1 cell (same cartoon, same donor, same seed). Kept so the layout factorial is self-contained and can run on a slot that has no a1.
5. Pass-2 references carry the arm suffix. The brief wrote PASS1/<exp>/pass1/<category>/<stem>.png; the actual pass-1 output is <stem>[_<arm>].png, so every pass-2 reference is PASS1/ + the exact pass-1 output.
6. Two helper files beyond the named deliverables: code/make_jobs.py (so 335 manifest entries are regenerable and auditable rather than hand-written) and code/resolve_jobs.py (the missing-input check worker.sh needs, kept out of the shell so it is unit-testable).
7. b1-native.json is written against a naming convention (organisms/library/organism_NN.png), because package B1 has not landed. worker.sh rewrites it from the real organisms/library/*.png listing before running it.
8. code/render_set.py does not exist yet (package A1). worker.sh tries --set <name> --out cartoons/<name> then --sets <name> --out cartoons, and if both fail it logs RENDER FAILED and lets the affected manifests be skipped. A1's tuning loop (code/tune.py, added while this package ran) has presets that split normal into normal_body / normal_antrum; if the rendered stems use those names, regenerate the manifests with python code/make_jobs.py --presets normal_body:normal normal_antrum:normal hpylori_gastritis intestinal_metaplasia mixed (375 jobs). Until then the mismatch degrades to logged skips, not a crash.
9. worker.sh gained DEVICE and SKIP_GPU_CHECK so the orchestration is rehearsable on CPU. Defaults are cuda and the hard CUDA assertion, exactly as the fullset worker.
10. cartoons/ was empty at the end of this package, so make_jobs.py --verify could only check ids, outputs and PASS1/ closure, not cartoon existence. Rerun --verify after A1 delivers.

8. What is not done

• No image was generated: this package had no GPU. Every number above is CPU-side.
• The token library, token stats and organism library have never been built with the real UNI2-h encoder; only the composition half and the file formats were exercised.
• code/render_set.py (A1), cartoons/ (A1) and organisms/library/ + organism-library/ (B1) are external dependencies that were absent.
• The composition descriptor's usefulness for retrieval is argued from distributions, not from a generated image. Whether A2 beats a fixed donor is the experiment, and it has not run.

Download public Markdown export