A4-TOKENS: do layout-free tokens let the cartoon survive pass 2?

Work package A4-TOKENS of the SGH programme (operations/research/sgh-program-20260908/PLAN.md), 8-9 September 2026. One spot A100 (slot a), 110 generations, 0 failures. Slot d (F2 training) was never touched; slots b and c were not used. Nothing was committed to git.

Engineering evidence only. Every instrument below measures shape, spacing, texture, colour or one pathology encoder. None of them diagnoses, grades or stages anything, no pathologist has seen any of it, and nothing here is "solved". Every "better" or "worse" names its instrument and its n.

One-sentence answer: the hypothesis is refuted. At start_index 3 not one of the five layout-free arms beats the single-real-donor arm on cartoon lumen IoU — every arm sits at 0.041-0.051 against a single-donor 0.051 and a cartoon ceiling of 0.625 — and the three arms whose tokens no longer depend on the cartoon collapse to producing one image five times (within-arm pairwise SSIM 0.98 against 0.03-0.05 for real fields). The run also produced the control that explains why: at si3 five different start images conditioned on the same tokens are 0.98-similar, while one start image under five different token sets is 0.60-0.65 similar — the pass-2 canvas is decided by the conditioning tokens, and the start image contributes almost nothing. Removing the donor's spatial arrangement does not hand layout control back to the start image; it removes the last thing that differed between cartoons.

0. What was built and run

Commands, in order, all from the programme root (P=../../../research/sgh-synthetic-histopathology/.venv/bin/python):

$P code/test_generate_cpu.py                      # 11/11
python3 code/make_a4_jobs.py                      # 110 jobs, verified against a1
cd fleet && ./fleet.sh up a && ./fleet.sh stage a ../packages/a4
./fleet.sh ssh a "... ln -s <slot0>/{cartoons/a1,token-library,token-stats} ..."
./fleet.sh launch a ../packages/a4 a4-pass1 a4-pass2 && ./fleet.sh watch a
./fleet.sh retrieve a <abs>/packages/a4/retrieved && ./fleet.sh down a
cd .. && python3 packages/a4/merge_records.py
$P packages/sweep-v1/evaluate_sweep.py features --records packages/a4/retrieved/records.jsonl --out packages/a4/eval --jobs 12
$P packages/sweep-v1/evaluate_sweep.py screens  --records packages/a4/retrieved/records.jsonl --out packages/a4/eval --jobs 12
$P packages/sweep-v1/evaluate_sweep.py sheets   --records packages/a4/retrieved/records.jsonl --out packages/a4/eval --jobs 12
$P packages/a4/analyse_a4.py --records packages/a4/retrieved/records.jsonl --out packages/a4/eval --jobs 12

1. The code change

Two additions to code/pixcell_generate.py, both backward compatible. The 16 tokens of a window are a 4x4 grid of 256-px UNI2-h patch embeddings, so their order is the donor's spatial arrangement; both constructions keep real per-patch appearance and destroy that arrangement.

token_patchbag — a primary source (lines 538-596)

"token_patchbag": {"library": "token-library/", "category": "<cat>", "per_window": true}

From the same 2016-window library token_library reads, restricted to category (default: the job's own), draw 16 (library window, patch) pairs uniformly at random and stack them into one (1, 16, 1536) grid. Every token is a real patch embedding of real tissue; no two need come from the same window, so the grid carries no coherent 4x4 layout. per_window: true draws a fresh bag for every canvas window; per_window: false draws once (with window index 0) and broadcasts.

The draw is numpy.random.default_rng(seed * 1000 + window_index): rng.integers(0, n_candidates, 16) then rng.integers(0, 16, 16). Recorded: token_patchbag (the spec as resolved), token_patchbag_candidates, token_patchbag_library_entries, token_patchbag_seed_base, and token_patchbag_picks — one entry per window carrying all 16 draws as {slot, library_entry, library_file, library_origin, library_category, patch_index}.

token_shuffle — a post-step over any source (lines 599-616)

"token_shuffle": true

After the primary source resolves a window's grid, permute its 16 tokens with the identical RNG, numpy.random.default_rng(seed * 1000 + window_index).permutation(16). The multiset of tokens is unchanged — same donor, same appearance statistics — only their positions in the 4x4 grid move. It applies to every source. The record keeps the primary token_source and adds token_shuffle, token_shuffle_seed_base and token_shuffle_permutations (one 16-int permutation per window). When the flag is absent the keys are absent, so old records keep their exact shape.

Precedence (line 619)

token_file > token_patchbag > token_library > token_blend > token_reference > reference, then token_shuffle as a post-step. The pre-existing precedence assertion is unchanged and still passes. token_sha256 is still hashed from whatever grid was actually handed to the denoiser, so it reflects the shuffle.

Two supporting edits so the fleet plumbing sees the new field: code/resolve_jobs.py resolves and existence-checks token_patchbag.library the same way it does token_library, and code/worker.sh counts a patch-bag library when deciding which libraries to build.

Tests — 11/11 pass

code/test_generate_cpu.py under research/sgh-synthetic-histopathology/.venv/bin/python, stub encoder and stub pipeline, no weights.

PASS  old-schema jobs are byte-for-byte unchanged against the canonical generator
      4 old-schema jobs identical (bytes + record) to canonical pixcell_generate.py
PASS  token_library picks the nearest composition per window
PASS  token_blend weights and 'self' resolve to the weighted sum
PASS  token_file broadcasts one grid to every window, both accepted shapes
PASS  token_shuffle permutes the 16 tokens of every window, from the documented RNG
      window perms [7, 10, 2, 11].../[9, 8, 6, 5]... from rng(seed*1000+w); applies to
      token_reference and token_file; absent key when off
PASS  token_patchbag assembles 16 real patches drawn from the library, per window or once
      4 windows x 16 draws from 3 candidates, every (file, patch) verified; per_window=False
      broadcasts window 0's bag; category restriction and precedence hold
PASS  precedence is token_file > token_library > token_blend > token_reference > reference
PASS  every record carries the provenance fields evaluation needs
PASS  resume skips ids already in records.jsonl and adds nothing on a rerun
PASS  resolve_jobs resolves PASS1 against out/ and drops jobs whose inputs are missing
PASS  split_manifests keeps every pass-2 job on its pass-1 slot and places every job

11/11 passed

The two new tests are arithmetic, not aesthetic. The shuffle test computes the expected permutation independently from numpy.random.default_rng(seed*1000+w), checks the record's claim against it window by window, checks the tensors actually handed to the stub denoiser are the unshuffled ones under exactly that permutation, checks the two windows get different permutations, checks a fixed token_file grid shuffled two ways yields two distinct per-window hashes, and checks that with the flag off the keys are absent. The patch-bag test builds a library where tokens[n, p, :] == n*100+p, so the value of every slot names its own (library window, patch) — all 4 x 16 draws are verified pair by pair against an independent re-draw, plus per_window: false broadcast, category restriction, an empty category raising rather than falling back, both precedence directions, and composition with the shuffle.

The strongest test is unchanged and still passes: the canonical repo generator and this one run the same four old-schema jobs under identical stubs and produce identical PNG bytes and identical records apart from the documented new keys.

2. The manifests and the run

code/make_a4_jobs.py copies the a1 pass-1 job of each cartoon verbatim and asserts, for all 10, that mode, seed, start_index, reference, token_reference and category are unchanged; only the id and the output path differ. Pass 2 is 10 cartoons x start_index {6, 3} x 5 arms = 100 jobs, all conditioned on the a4 pass-1 output.

Setup: 2 minutes instead of 45

fleet.sh up a returned a ready GPU in 64 s; staging 31 files took 44 s. Cartoons, token-library/ (198 MB, 2016 windows) and token-stats/ were symlinked into the new run root from the previous package root on slot a's own disk ([local]/slot0), so worker.sh logged cartoons/a1 already rendered (60 cartoons), token-library already built, token-stats already built and went straight to generation. Boot to first generation was 2 min 46 s, against 45 minutes for the sweep, saving roughly USD 1.50 of idle A100 and about 100 minutes.

Timeline (UTC, 8 September)

No preemption. fleet.sh watch polled every 60 s (last logged poll 61) and exited on WATCH WORKER_DONE 0 at 19:38:49.

Throughput and cost

Total generation 4469 s = 1.24 GPU-hours over 110 jobs, mean 40.6 s. The 8.4 s that donor-shuffle si6 costs over the other si6 arms is the UNI2-h encode of the donor field — the layout-free arms never touch the encoder, which is a small side benefit.

Powered 18:18:17 to 19:48:34 = 90.3 min = 1.505 h = USD 3.19 at USD 2.12/h, plus about USD 0.20 of egress for the 1.63 GB retrieved. GPU utilisation 1.24 / 1.505 = 82%, against the sweep's 56%.

Coverage and provenance: clean

packages/a4/merge_records.py: 110 records read, 110 written, 0 duplicates. Per manifest, expected = records = PNGs (a4-pass1 10/10/10, a4-pass2 100/100/100), no missing ids. Every one of the 110 PNGs was re-hashed on this Mac and all 110 match their record's output_sha256; no zero-byte file (the fault that cost the sweep one image). All 110 token_source values equal their extra.expected_token_source — token_reference 30, token_patchbag 40, token_library 20, token_file 20 — and all 60 records that should carry token_shuffle: true do, while the other 50 have no token_shuffle key at all.

Pass 1 reproduces the sweep exactly

All 10 a4 pass-1 outputs are byte-identical to the corresponding sweep a1-pass1 outputs, and each job's reference_sha256 also matches the sweep's, confirming the symlinked cartoons are the exact bytes the sweep used. Example: a4_p1_intestinal_metaplasia_s11_across -> dfe57d6f745b2bdeb0269b70211a1b84fbfd6c6d0ed911e75c998069d5289014, identical to a1_p1_intestinal_metaplasia_s11_across. That is FLEET's cross-slot determinism extended to cross-run determinism on the same slot five hours later, and it means every a4 pass-2 arm starts from literally the same image as the a1 arm it is compared against.

Patch-bag provenance on real weights

token_patchbag_candidates = 504 (the library's per-category share of 2016 windows). bag-window writes 21 picks x 16 draws = 336 recorded draws per canvas and touches a median of 24 distinct donor fields per canvas — every training field of the category. bag-fixed writes one pick of 16 draws touching a median of 11 distinct fields. For contrast, a2-style library-shuffle touches a median of 1.5 fields per canvas.

The label maps are the exact ones the generator saw

EVAL_RESULT.md section (c) had to caveat its IoU numbers because the VM cartoons were never retrieved. They were retrieved here (30 files, 125 MB). Comparing them with the local Mac render: the label maps are not pixel-equal (the stromal-nuclei scatter differs, as SWEEP_RESULT.md section 6 found), but the lumen label is identical on all 10 — VM-vs-Mac lumen IoU is exactly 1.000 for every cartoon. So EVAL_RESULT's argument that lumen IoU was safe is now measured rather than argued, and every IoU below is against the exact bytes the generator conditioned on.

3. The table

Per category x start_index x arm, n = 5 cartoons each; medians. IoU is cartoon lumen IoU against the own (VM) label map; SSIM is mean pairwise SSIM within the cell (the repetition check, 10 pairs); cos is phikon-v2 max cosine against 118 real reference fields (fail line 0.95); srcfrac is the fraction of the canvas's phikon tiles whose nearest real field is one the tokens actually came from; ncc is max pixel NCC (flag 0.92). Full file: packages/a4/eval/a4-table.csv.

Cartoon lumen IoU ceiling (the cartoon PNG measured against its own label map): IM 0.407, gastritis 0.843, pooled median 0.625.

Pooled over both categories (n = 10 per cell, packages/a4/eval/a4-headline.json):

Paired deltas, a4 arm minus the a1 single-donor arm on the same cartoon at the same depth

Medians of 10 per-cartoon deltas; n+ is how many of the 10 deltas were positive (packages/a4/eval/a4-paired-vs-a1.csv). For IoU and ring, positive is better; for D, negative is better; lap is the Laplacian detail statistic, higher is closer to real.

4. Repetition, and the control that explains everything

Mean pairwise SSIM inside each cell — five outputs from five different cartoons (packages/a4/eval/a4-within-arm-ssim.json):

The token-vs-start control (new, and free from the design)

bag-window, bag-fixed and nearmean-shuffle seed their draw from (seed, window_index) only, so all five cartoons in one cell receive the identical 21 token grids and differ only in their start image. Verified from the records: those six cells each contain exactly 1 distinct token-grid set across 5 records with 5 distinct reference_sha256 values. donor-shuffle has 5 distinct token-grid sets (one per donor); library-shuffle has 4-5 (the composition pick varies with the cartoon).

That gives a two-way control the sweep never had:

At si3, replacing the entire 4096x2048 start image — a different cartoon, a different pass-1 canvas — moves the output less than a fifth as far as changing the tokens does. The pass-2 canvas is a function of the conditioning tokens; the start image is nearly inert. At si6 the start image matters somewhat more (0.888-0.914 rather than 0.98), which is consistent with the a1 ladder's monotone IoU collapse, but the ordering is the same.

That is the mechanism behind the null result. The hypothesis assumed the layout would revert to the start image once the tokens stopped carrying one. It does not: with no layout in the tokens, the model simply invents its own, driven by the tokens' appearance statistics. And because those tokens no longer depend on the cartoon, five different cartoons produce one image.

5. The contact sheets, described honestly

packages/a4/eval/sheets/a4-intestinal_metaplasia.png and a4-hpylori_gastritis.png: five rows (cartoons s11-s15), nine columns — cartoon | a1 pass1 si12 | a1 si3 donor | donor-shuffle si3 | bag-window si3 | bag-fixed si3 | library-shuffle si3 | nearmean-shuffle si3 | real held-out — 768 px crops at the same canvas position (1664, 640, 2432, 1408), each labelled ring / D / IoU. Four a4-zoom-* sheets show the same nine cells for one cartoon at ~620 px so nuclei are judgeable.

Reading down the columns (this is the decisive view). In the IM sheet, columns 2, 3 and 4 — a1 pass 1, a1 si3 donor and donor-shuffle si3 — are five visibly different fields: row 1 a dark palisade along a gland edge, row 2 open pale lumens with thin strands, row 3 a very pale foamy field, row 4 a magenta inflamed-looking field, row 5 a crowded blue-purple one. Columns 5, 6 and 8 — bag-window, bag-fixed and nearmean-shuffle — are five near-copies of each other: the same pale gland with the same vacuole pattern and the same dark nuclear strand in the same corner, five times. Column 7, library-shuffle, is in between: recognisably the same kind of field five times, but with real variation in where things sit. The gastritis sheet says the same thing more strongly — its bag-window, bag-fixed and nearmean-shuffle columns are five copies of one pale foveolar-looking field. This is exactly what the SSIM column reports, and nothing in the sheets contradicts it.

Reading across a row (what the arms look like). On IM s11 si3: the cartoon is flat lilac with dark bean-shaped nuclei and white lumen blobs; a1 pass 1 is a soft, low-contrast, faintly blurry version of it — smoothed cartoon, not tissue; a1 si3 donor is genuinely tissue-like, with dark columnar nuclei palisaded along a gland, pink fibrillar stroma and red cells; donor-shuffle si3 is of the same character — crowded epithelial clusters with clear vacuoles between nuclear strands; bag-window si3 is paler and more vacuolated with large clear spaces (and has the best envelope distance in the package, D 2.9); bag-fixed si3 is similar but with darker nuclear clumps; library-shuffle si3 is darker, denser, flatter, visibly over-stained; nearmean-shuffle si3 is distinctly different — many small round dark nuclei on a magenta ground with red cells, reading more like inflamed lamina propria than intestinal metaplasia. The real held-out field is paler and more delicate than all of them, with a foamy reticulated cytoplasm and thin strands that none of the generated cells reproduces (lap 23.6 against 17-21).

What is absent from every pass-2 cell. The cartoon's geometry. In none of the 100 pass-2 images can I find the cartoon's lumen blobs or gland circles in their cartoon positions — which is the 0.03-0.06 IoU against a 0.41-0.84 ceiling, seen rather than computed. The a1 pass-1 column, by contrast, visibly is the cartoon (IoU 0.35 IM / 0.66 gastritis).

si6 (a4-zoom-*-si6.png). Gastritis si6 is the worst region of the whole programme and A4 does not repair it: a1 si6 donor is a pale streaky field with bland pink blobs, ring 0.000; donor-shuffle si6 is paler and smoother still; bag-window and bag-fixed si6 are chunky or foamy but structureless; nearmean-shuffle si6 is a uniform vacuolated texture with evenly spaced nuclei that reads as a repeating pattern rather than tissue; library-shuffle si6 is much more purple and denser than any real field, which is its +1.1 D and -4.5 Laplacian.

6. Verdict per arm

Against the hypothesis: tokens that keep real per-patch appearance but no coherent spatial arrangement let the start image keep the layout at pass 2 while realism stays near the single-donor level.

Does any arm dominate the single-donor si3 arm on layout IoU without losing more than ~1 D?

No. Stating it precisely, on the 10 paired cartoons at si3:

• The largest positive paired IoU delta is bag-window at +0.0030 (median), positive on 6 of 10 cartoons — against a within-cell IoU spread of 0.023-0.087. That is not a gain, it is noise around zero. Two arms are negative and nearmean-shuffle is positive on only 2 of 10.
• Every arm is within +-0.013 IoU of the single-donor arm, against a headroom of 0.57 to the cartoon ceiling. There is nothing to trade off.
• The only arm that gains real layout anywhere is library-shuffle at si6 (+0.062 IoU, +0.232 ring), and it costs +1.10 D — right at the ~1 D limit — plus a 26% loss of Laplacian detail and 0/10 inside the envelope band. It fails the constraint.
• bag-window si3 does dominate on the envelope (-0.85 D, better on 10/10, 10/10 inside the band), which is the best realism result in the package, but it buys nothing on layout and is disqualified by SSIM 0.98.

The copy screens are clean everywhere: max phikon cosine over all 800 candidate tiles is 0.881 against a 0.95 fail line (min copy margin 0.069, nearmean-shuffle), max pixel NCC 0.747 against a 0.92 flag, 0 flagged tiles and 0 flagged windows in all 182 pairs. Nothing in A4 reproduces a real field.

7. What this changes, and what to do instead

1. The 4x4 token arrangement is not the lever. donor-shuffle is a null on every axis. Whatever the 16 tokens do, they do it as an unordered set of appearance descriptors, not as a spatial map. The layout collapse at pass 2 is therefore not caused by the donor's coordinates being copied, and A2's and A3's ~0.22 IoU is not evidence that layout-free tokens preserve the cartoon.
2. A2/A3's 0.22 was a different experiment. Verified from the sweep records: a2 and a3 used their layout-free tokens at both passes, so their pass-1 output was already layout-free and their pass-2 IoU is measured against a cartoon that pass 1 had already only loosely followed. A4 holds pass 1 fixed (byte-identical to a1) and changes only pass 2, and under that control the layout-free arms give 0.041-0.051, not 0.22.
3. The start image is nearly inert at si3 (SSIM 0.98 across five different start images with fixed tokens, vs 0.60-0.65 across five token sets with a fixed start image). Any plan that hopes to impose geometry through the img2img start latent at si3 is fighting a factor-of-five asymmetry. Geometry has to enter through something the denoiser reads per position — the F2 structure adapter, or region_img2img's per-label lock (lock_mask/lock_labels, already in the generator and unused by this sweep), not through the token grid.
4. Any conditioning that does not depend on the cartoon collapses the output set. The three arms whose RNG keys on (seed, window) alone produce SSIM 0.98. If patch bags are used again, the seed must include the cartoon (e.g. seed*1000 + window + hash(cartoon)), which is a one-line change to patchbag_rng's caller. That would at least restore diversity; it would not restore layout.
5. bag-window is worth one follow-up for realism alone. D 3.96 pooled (10/10 inside the envelope band, IM 2.84 against a real median of 2.58) is the best envelope result in the programme, and it needs no donor field, no encoder pass and no single slide. Re-run with a cartoon-dependent seed to see whether the envelope gain survives once the five outputs are forced apart.

8. Harness changes made in this package

Three, all in packages/sweep-v1/evaluate_sweep.py, all additive; none of them can change a sweep-v1 number except where noted.

1. _library_origins_for() now also reads token_patchbag_picks, so a patch-bag arm gets an honest source fraction instead of the structural zero EVAL_RESULT.md section 1.2(b) fixed for token_library. No sweep-v1 record has that key.
2. _library_field_count() had the same bug 1.2(b) fixed elsewhere: it read library_origin or library_file, and library_origin is a pixel coordinate ([0, 512]), so Path(str([0, 512])).name returned "[0, 512]" and the function counted distinct window origins, not distinct donor fields. Fixed to read library_file only, and extended to patch bags. This does change n_library_fields for a2 if packages/sweep-v1/eval is regenerated; EVAL_RESULT.md section (d)'s "median 2 distinct donor fields per canvas, range 1-3" was computed with the bug and should be re-derived. It is not used in any headline number.
3. The sheets stage is hard-wired to a1/d1/a2/a3/d2/e2/b1 and produced 0 sheets for a4 (it logs no data: for each and exits 0). The sheets this brief asks for are written by packages/a4/analyse_a4.py instead, which also labels each cell with IoU — something _num_label does not carry.

packages/a4/analyse_a4.py re-implements the two EVAL-RUN measurements that were never saved as scripts (a1-layout-iou, within-arm-diversity) using the harness's own _d1_one, analysis_masks, label_mask, iou and analysis_masks(...)["grey"][::4,::4] SSIM raster, so the numbers are directly comparable. As a check that the re-implementation is faithful, its independent per-tile source fraction reproduces the harness's own_source_fraction exactly on the a1 arms (0.875 / 1.000 IM si6 / si3; 0.750 / 1.000 gastritis).

9. Caveats

• n = 5 per cell, 10 cartoons total, two categories, one cut. Every "median" is a median of 5. The paired deltas are the more trustworthy statistic and they are medians of 10.
• The three collapsed arms have an effective n far below 5: five outputs at SSIM 0.98 are close to one measurement, so their envelope and ring numbers should be read as "this token set gives D ≈ x", not as a distribution over cartoons.
• Layout is measured only as lumen IoU against the cartoon's own label map, on the ring_topology analysis raster. It is a coarse instrument; a rearranged-but-plausible gland field scores 0 exactly as a wash does. The epithelium IoU is in a4-layout-iou.csv and is noisier (the cartoon's own epithelium ceiling is 0.10-0.42).
• The envelope, ring, Laplacian and copy screens are the C1/C2 instruments unchanged; the features self-check reproduced C1's held-out Mahalanobis distances to max |delta| = 0.0000.
• bag-window's per-window bag makes 21 independent draws over one canvas; nothing here measures window-boundary seams, which is the failure mode that construction would be most prone to. Recorded as not measured, not as absent.
• No pathologist has seen any of this. Yash has not reviewed a sheet. Nothing here is a clinical claim.
• Slots b, c and d were RUNNING under other packages while this ran; they were not touched. Slot a is TERMINATED, confirmed by fleet.sh status after down.

Download public Markdown export