# GEN-CODE result, 8 September 2026

Work package GEN-CODE of the SGH programme (`operations/research/sgh-program-20260908/`). Everything
below is engineering evidence: code, tests and job manifests. No image was generated (no GPU in this
package), nothing here is clinically validated, and nothing here is "solved".

Everything runs from the programme root. Nothing in `research/sgh-synthetic-histopathology/` was
edited; the canonical generator, `sydney_qa.py`, `sydney_qa_v1.json` and `pixcell_stain_match.py`
were copied into `code/` and only the copies were changed. Nothing was committed to git.

## 1. What was built

| file | lines | what it is |
|---|---:|---|
| `code/pixcell_generate.py` | 667 | the canonical generator (438 lines, md5 `ab317dd1...`) plus three token sources |
| `code/build_token_library.py` | 411 | composition-indexed UNI2-h token library from real donor fields (GPU) |
| `code/token_stats.py` | 152 | per-category grand mean, k-means centroids and nearest-real-window (torch only) |
| `code/test_generate_cpu.py` | 576 | 9 CPU tests, stub encoder + stub pipeline, no weights |
| `code/make_jobs.py` | 428 | writes all 15 manifests + `jobs/README.md`, deterministically |
| `code/resolve_jobs.py` | 103 | resolves `PASS1/` and drops jobs whose inputs are missing |
| `code/split_manifests.py` | 171 | balanced N-slot partition that never splits a pass-1 -> pass-2 chain |
| `code/worker.sh` | 197 | one GPU slot: cartoons, libraries, stats, manifests, evidence, self-stop |
| `code/sydney_qa.py`, `code/sydney_qa_v1.json`, `code/pixcell_stain_match.py` | - | unmodified copies the above import |

## 2. Generator changes (`code/pixcell_generate.py`)

Line numbers are in the new file.

| lines | change |
|---|---|
| 11-38 | module docstring: the three new sources, their job fields and the precedence rule |
| 328-332 | `TokenCache.library()` - one `TokenLibrary` per path per process, so 20 A2 jobs load `tokens.pt` once |
| 335-379 | composition vector: `COMPOSITION_FIELDS`, the label-id groups, `STROMA_NUCLEUS_DENSITY_SCALE = 0.35`, `normalise_stroma_nucleus_density`, `label_composition`, `load_label_map` |
| 382-426 | `TokenLibrary`: loads `index.json` (list, or dict with `entries`) + `tokens.pt`, validates `(N, 16, 1536)` and the entry count, indexes by category, `nearest()` by exact L2 |
| 429-441 | `load_token_file`: accepts `(1, 16, 1536)` or `(16, 1536)`, rejects anything else and non-finite tokens |
| 444-474 | `tokens_from_library`: per-window composition of the job's label map -> nearest library entry -> its grid; builds the per-window provenance list |
| 477-508 | `tokens_from_blend`: weighted sum of per-window grids, `self` = the job's own `reference`, size and shape asserted |
| 511-528 | `resolve_tokens`: the single precedence point, returns `(tokens, hashes, provenance)` |
| 562 | `reconstruct` records `token_source: null` (a VAE round trip uses no conditioning) |
| 564-567 | `run_job` calls `resolve_tokens` and merges the provenance into the record |

The canonical variable `token_source` (a PIL image) was renamed `token_image` so the record key
`token_source` (a string) is unambiguous; no behaviour changed.

### Job schema additions

| field | experiment | meaning |
|---|---|---|
| `token_file: <path.pt>` | A3 | one `(1,16,1536)` / `(16,1536)` grid used for every window |
| `token_library: <dir>` + `labels: <label png>` | A2, B1 | per-window retrieval by composition; `token_library_same_category` (default `true`) restricts to the job's `category` |
| `token_blend: [{"source": "<path>\|self", "weight": w}, ...]` | D2 | weighted sum of per-window grids; weights need not sum to 1 |

Precedence, asserted by a test: `token_file` > `token_library` > `token_blend` > `token_reference` >
`reference`. `token_reference` behaviour is untouched.

### Record additions

Always: `token_source` (`"reference"`, `"token_reference"`, `"token_library"`, `"token_blend"`,
`"token_file"`, or `null` for `reconstruct`). Then, only for the source that won:

- `token_file`, `token_file_sha256`
- `token_library`, `token_library_entries`, `token_library_same_category`, `labels`,
  `labels_sha256`, and `token_library_picks`: one entry per window with `origin`, `library_file`,
  `library_origin`, `library_category`, `library_entry`, `distance`, `window_composition`,
  `library_composition`
- `token_blend`: one entry per source with `source`, `weight`, `sha256`, `token_sha256`; plus
  `token_blend_weight_sum`

Every existing key keeps its name and value. `token_sha256` is still the per-window list, now hashed
from whatever grid was actually handed to the denoiser.

## 3. Composition vector, and the one deviation from the brief

`[lumen, epithelium, stroma, stroma_nucleus_density_norm]`, computed two ways that must land on the
same scale: from a cartoon label map (`pixcell_generate.label_composition`, ids 1 / 2,3,4 / 5,6, with
0, 7, 8 ignored and density = label 6 / (label 5 + label 6)) and from a real RGB window by classical
segmentation (`build_token_library.window_composition`). Both divide the density by 0.35 and clip.

The brief said to copy `ring_topology.py`'s epithelial band, which closes the 72nd-percentile
hematoxylin mask with a disk of `CLOSING_RADIUS_UM = 6.0`. **Measured, that collapses the
descriptor.** On 8 real fields, 9 windows each, mean stroma fraction at 6 um:

| closing radius | normal | gastritis | IM | mixed |
|---|---|---|---|---|
| 6 um (ring_topology) | 0.011 | 0.001 | 0.057 | 0.068 |
| **2 um (the default here)** | **0.211** | **0.101** | **0.277** | **0.295** |

At 6 um the band swallows the lamina propria and the four-vector degenerates to "lumen fraction"
alone (a 4-field, 84-window smoke library had stroma between 0.000 and 0.007). At 2 um the same
windows span lumen 0.077-0.485, epithelium 0.429-0.660, stroma 0.025-0.458. Everything else is
`ring_topology.py` and `sydney_qa_v1.json` unchanged: the OD tissue floor, the 72nd-percentile
hematoxylin threshold, the disk closing, the 100 um^2 speck drop, the lumen brightness floor.
`--epithelium-closing-um 6` reproduces the literal ring_topology band. The radius is recorded in
`build-meta.json` and in `composition-summary.md`.

Donor side vs cartoon side, per category (donor: 3 fields x 9 windows at 2 um; cartoon: the 15
non-empty windows of an `s11_across` render from `code/render_cartoon.py`):

| category | side | lumen | epithelium | stroma | density |
|---|---|---|---|---|---|
| intestinal_metaplasia | cartoon | 0.119 | 0.546 | 0.335 | 0.300 |
| intestinal_metaplasia | donor | 0.188 | 0.426 | 0.386 | 0.138 |
| hpylori_gastritis | cartoon | 0.120 | 0.545 | 0.335 | 0.543 |
| hpylori_gastritis | donor | 0.319 | 0.587 | 0.094 | 0.029 |
| normal | cartoon | 0.120 | 0.545 | 0.335 | 0.220 |
| normal | donor | 0.153 | 0.689 | 0.157 | 0.012 |

Read honestly: IM overlaps well; gastritis and normal keep an offset on stroma, because the closing
still absorbs an inflamed or densely glandular lamina propria. A2 searches **within one category**,
so a constant per-category offset shifts every candidate equally and the ranking is driven by the
window-to-window spread, which is large on both sides. Note also that the current renderer's four
categories differ only in goblet fraction and stromal nuclear density, so on the cartoon side the
first three coordinates are nearly identical across categories - the vector discriminates *windows*,
not *categories*, which is exactly what A2 needs. None of these numbers means anything clinically.

Two other recorded coarsenesses: `region` (the composition denominator) is the tissue mask closed at
12 um and hole-filled, so gland lumens count as field rather than as background glass; and disk
morphology is computed by Euclidean distance transform rather than `ndimage.binary_closing`, which
is the same disk at O(N) instead of O(N pi r^2) (2 s -> 20 ms per window at r = 24 px) and differs
only in not eroding against an implicit background ring at the array border.

## 4. Tests

`code/test_generate_cpu.py`, run with `research/sgh-synthetic-histopathology/.venv/bin/python`
(torch 2.9.1, numpy 2.5.2, CPU). Stub encoder returns deterministic pseudo-tokens from the window's
pixel bytes; stub pipeline does no diffusion and returns the reference image.

```
PASS  old-schema jobs are byte-for-byte unchanged against the canonical generator
      4 old-schema jobs identical (bytes + record) to canonical pixcell_generate.py
PASS  token_library picks the nearest composition per window
      2 windows -> lumen.png (d=0.3536), stroma-dense.png (d=0.6124); 5-entry library
PASS  token_blend weights and 'self' resolve to the weighted sum
      alpha=0.75 self+donor exact to 1e-6 on 2 windows; alpha=1.0 equals self; size mismatch refused
PASS  token_file broadcasts one grid to every window, both accepted shapes
      (1,16,1536) and (16,1536) both broadcast to 4 windows; wrong shape refused
PASS  precedence is token_file > token_library > token_blend > token_reference > reference
      token_file > token_library > token_blend > token_reference > reference
PASS  every record carries the provenance fields evaluation needs
      4 records: ['canvas', 'category', 'elapsed_seconds', 'extra']... plus source-specific keys, all present
PASS  resume skips ids already in records.jsonl and adds nothing on a rerun
      2 jobs -> rerun adds 0 records -> manifest grown by 1 adds exactly 1
PASS  resolve_jobs resolves PASS1 against out/ and drops jobs whose inputs are missing
      PASS1 -> out/ absolute, 2 of 3 dropped (reference, token_library), empty manifest exits 3
PASS  split_manifests keeps every pass-2 job on its pass-1 slot and places every job
      2/4/7 slots: 335 jobs placed, 0 broken chains, worst weight spread <25%

9/9 passed
```

The first test is the strong one: it loads the **canonical** repo generator and this one as two
modules, runs the same four old-schema jobs (img2img on the reference, img2img with
`token_reference`, `generate`, `reconstruct`) through both under identical stubs, and demands
identical output PNG bytes and identical records apart from exactly one new key, `token_source`.

The A2 test builds a library whose answer is arithmetic: entries at pure lumen, pure epithelium,
pure stroma and nucleus-packed stroma, each with a token grid filled with its own constant, plus a
decoy entry of another category on the pure-lumen point. A label map that is 3/4 lumen in window 0
and half nucleus-packed stroma in window 1 must pick `lumen.png` then `stroma-dense.png`, and the
tokens handed to the denoiser must be those constants. A category with no entries raises rather than
falling back silently.

## 5. Worker rehearsal (CPU, no GPU, no model weights)

`worker.sh` was rehearsed twice on this Mac with `SKIP_GPU_CHECK=1 SELF_STOP=0 DEVICE=cpu` and a
shim that patches the real generator's encoder and pipeline.

1. **Everything missing** (no cartoons, no `render_set.py`, no donor root): 6 manifests, all skipped
   with a reason in `evidence/steps.log`, `evidence/manifest-summary.json` = `{"ran":0,"skipped":6,
   "failed":0}`, exit 0, `WORKER_DONE` written. A slot that arrives before its inputs does not fail.
2. **Full chain** with two fabricated cartoons, a 4-window token library, token stats and a local
   donor root: `a1-pass1` (2) -> `a1-pass2` (2, `PASS1/` resolved to `out/`) -> `a2-pass1` (2,
   library retrieval) -> `a3-pass1` (4, `token_file`) -> `d2-pass2` (8, blend on a1's pass-1
   output) = 18 records, 18 PNGs, `ran=5 skipped=0 failed=0`. Every record's `token_source` equalled
   its `extra.expected_token_source`. Rerunning the same command added **zero** records (`diff` of
   `records.jsonl` empty) and rebuilt neither library nor stats.

`build_token_library.py` was also run for real on 4 local HiESD training fields (84 windows,
`--composition-only`, no encoder) and `token_stats.py` on an 84-window library: shapes
`(1,16,1536)`, `(8,16,1536)`, `(1,16,1536)`, `token-stats.json` written with distances and cluster
sizes. Composition timing is ~0.3 s per 1024 window on this CPU, so the full library (4 categories x
24 fields x 21 windows = 2016 windows) is roughly 10 CPU-minutes of segmentation plus 2016 x 16
UNI2-h forward passes on the GPU.

## 6. Manifests

`jobs/` holds 15 manifests, **335 jobs**, 335 distinct outputs, 0 duplicate ids, 0 unsatisfied
`PASS1/` references (`python code/make_jobs.py --verify`). Full table and dependencies in
`jobs/README.md`.

| manifest | jobs | manifest | jobs |
|---|---:|---|---:|
| `a1-pass1` | 60 | `a3-pass1` | 20 |
| `a1-pass2` | 60 | `a3-pass2` | 20 |
| `a1-pass2-si9` | 15 | `d2-pass2` | 20 |
| `d1-pass1` | 25 | `e2-pass1` | 15 |
| `d1-pass2` | 25 | `e2-pass2` | 15 |
| `a2-pass1` | 20 | `b1-pass1` (optional) | 5 |
| `a2-pass2` | 20 | `b1-pass2` (optional) | 5 |
| | | `b1-native` (optional) | 10 |

At the fullset run's measured 40-60 s per 4096x2048 pass, 335 jobs is roughly 4-6 GPU-hours; split
4 ways, ~1-1.5 h per slot (`split_manifests.py --slots 4` gives 82-87 jobs and weight 19064-19194
per slot, 0 broken chains).

## 7. Deviations from PLAN.md and the brief

1. **Epithelial closing radius 2 um, not `ring_topology.py`'s 6 um** - measured, section 3. The
   brief's literal construction is one flag away (`--epithelium-closing-um 6`).
2. **E2 is 15+15, not the table's 30+30.** The ladder is levels 0-3 x 5 seeds and level 2 *is* the
   current renderer, so those five cells are already `a1-pass1`/`a1-pass2`
   (`intestinal_metaplasia_s*_across`). The brief said exactly this ("fidelity 0,1,3 (f2 is a1)");
   the full ladder is 20+20 counting the a1 cells.
3. **B1 is 5+5+10, not the table's 10+10+10.** The brief fixes five b1 cartoons
   (`hpylori_gastritis_s21..25_oblique`).
4. **`d1`'s `donor0` arm duplicates a1's pass-1 cell** (same cartoon, same donor, same seed). Kept so
   the layout factorial is self-contained and can run on a slot that has no a1.
5. **Pass-2 references carry the arm suffix.** The brief wrote
   `PASS1/<exp>/pass1/<category>/<stem>.png`; the actual pass-1 output is
   `<stem>[_<arm>].png`, so every pass-2 `reference` is `PASS1/` + the exact pass-1 `output`.
6. **Two helper files beyond the named deliverables**: `code/make_jobs.py` (so 335 manifest entries
   are regenerable and auditable rather than hand-written) and `code/resolve_jobs.py` (the
   missing-input check `worker.sh` needs, kept out of the shell so it is unit-testable).
7. **`b1-native.json` is written against a naming convention** (`organisms/library/organism_NN.png`),
   because package B1 has not landed. `worker.sh` rewrites it from the real `organisms/library/*.png`
   listing before running it.
8. **`code/render_set.py` does not exist yet** (package A1). `worker.sh` tries
   `--set <name> --out cartoons/<name>` then `--sets <name> --out cartoons`, and if both fail it logs
   `RENDER FAILED` and lets the affected manifests be skipped. A1's tuning loop
   (`code/tune.py`, added while this package ran) has presets that split `normal` into `normal_body`
   / `normal_antrum`; if the rendered stems use those names, regenerate the manifests with
   `python code/make_jobs.py --presets normal_body:normal normal_antrum:normal hpylori_gastritis
   intestinal_metaplasia mixed` (375 jobs). Until then the mismatch degrades to logged skips, not a
   crash.
9. **`worker.sh` gained `DEVICE` and `SKIP_GPU_CHECK`** so the orchestration is rehearsable on CPU.
   Defaults are `cuda` and the hard CUDA assertion, exactly as the fullset worker.
10. **`cartoons/` was empty at the end of this package**, so `make_jobs.py --verify` could only check
    ids, outputs and `PASS1/` closure, not cartoon existence. Rerun `--verify` after A1 delivers.

## 8. What is not done

- No image was generated: this package had no GPU. Every number above is CPU-side.
- The token library, token stats and organism library have never been built with the real UNI2-h
  encoder; only the composition half and the file formats were exercised.
- `code/render_set.py` (A1), `cartoons/` (A1) and `organisms/library/` + `organism-library/` (B1)
  are external dependencies that were absent.
- The composition descriptor's usefulness for retrieval is argued from distributions, not from a
  generated image. Whether A2 beats a fixed donor is the experiment, and it has not run.