# EVAL-RUN: measuring the Phase-2 sweep Work package EVAL-RUN of the SGH programme (`operations/research/sgh-program-20260908/PLAN.md`), 8 September 2026. CPU only, `research/sgh-synthetic-histopathology/.venv/bin/python`, `--jobs 12`. Nothing was committed to git. No GPU was used. **Engineering evidence only.** Every instrument here measures shape, spacing, texture, colour, or one pathology encoder. None of them diagnoses, grades or stages anything, and no number below says an image is diagnostically correct. No pathologist has seen any of this; Yash has not reviewed a sheet. Every "better" or "worse" names its instrument and its n. --- ## 0. What was run ```bash cd [local]/sgh-program-20260908 P=../../../research/sgh-synthetic-histopathology/.venv/bin/python $P packages/sweep-v1/evaluate_sweep.py all \ --records packages/sweep-v1/retrieved/records.jsonl \ --out packages/sweep-v1/eval \ --jobs 12 --probes ``` `--ncc-refs` was **left at its default `targeted`** (own donor field plus up to two phikon nearest-neighbour reference fields per output), as EVAL_BUILD_RESULT.md section 5 documents. No other flag was added. Wall clock: features 142 s, a1 <1 s, d1 9 s, arms+probes 267 s, screens 782 s (embed 168 s + 417 NCC pairs 614 s), sheets 55 s, select 3 s, report <1 s — **about 22 minutes**, plus two stage reruns after the harness fixes in section 1. Deliverables, all under `packages/sweep-v1/eval/`: | file | what | |---|---| | `EVAL_REPORT.md` | the harness's own report (every table, its own verdicts) | | `features.csv` / `scores.csv` | 449 rows: 407 sweep outputs + 22 real held-out + 20 old-baseline | | `a1-table.csv`, `d1-measurements.csv`, `d1-pairwise-ssim.csv`, `arms-pairs.csv` | per-stage tables | | `screen-embed.csv`, `screen-ncc.csv`, `copy-margin.csv` | copy screens | | `candidates.csv` (+ per category), `select-scores.csv` | Pareto selection | | `sheets/*.png` | 26 sheets: 18 from the `sheets` stage + `candidates.png` from `select` + 7 written by this package (section 2) | | `probes/*` | the sweep-augmented C2 probes (C2's own `probes/` verified byte-identical after) | | **`headline.json`** | flat dict of every number a results page would quote | | **`headline-sheet.png`** | per category: real held-out │ old two-pass │ best a1 arm │ a2 │ pass 3 | | `a1-layout-iou.csv/.json`, `cartoon-iou-ceiling.csv/.json`, `within-arm-diversity.json` | extra measurements this package added (section 2) | **Instrument self-check.** The harness rescored the 22 real held-out fields from scratch and reproduced C1's `morphometrics/scores.csv` Mahalanobis distances to **max |delta| = 0.0000**. It is the same instrument. **Hand-check of the harness's arithmetic.** Every one of the 41 sweep rows and 8 reference rows of `a1-table.csv` was recomputed independently with pandas (system `python3` 3.13 / pandas 3.0.2, a different interpreter from the harness venv) straight from `features.csv`: | column | max abs difference, harness vs hand | |---|---| | `ring_with_lumen_fraction` | 5.6e-17 | | `mahalanobis` | 0.0 | | `pct_vs_heldout` | 0.0 | | `det_lap_abs_mean` | 0.0 | | `ring_density_per_mm2` | 0.0 | | `nuc_density_per_mm2` | 0.0 | | `str_nuc_density_per_mm2` | 4.5e-13 | | `n` | 0 | and the reference rows reproduce PLAN.md's recorded baseline exactly: `fullset_pass2` **mean** ring 0.7256 normal / 0.4753 gastritis / 0.9170 IM / 0.7260 mixed, real held-out 0.8538 / 0.7428 / 0.7101 / 0.7261. The tables in this document are safe to quote. --- ## 1. Two things went wrong, and what was done about them ### 1.1 One of the 407 sweep outputs is a zero-byte file (a DATA fault, not a harness fault) `packages/sweep-v1/retrieved/slot0/out/a1/pass2/intestinal_metaplasia/intestinal_metaplasia_s14_oblique.png` is **0 bytes**. Its record (`a1_p2_intestinal_metaplasia_s14_oblique`) carries `output_sha256 = 1779fb07b5f8dda2878a5291c8e864e407c676202be764ecf9ed1a0bd042dbb4`, i.e. the generator wrote and hashed a real 4096x2048 PNG. The VM's own `SHA256SUMS` already records the file as `e3b0c442...` — the SHA-256 of the empty string — so **the file was already empty on the VM before retrieval**, and SWEEP_RESULT.md's "SHA256SUMS verified 143 files OK" verified an already-empty file transferring correctly. Slot a is the slot that was preempted at 15:38:39 UTC; the most likely cause is that the power-off dropped the page cache for a file that had been written but not yet flushed. SWEEP_RESULT.md's coverage check counted PNG *existence*, not size, so it reported 143/143. I verified the rest: **406 of 407 outputs hash exactly to their record's `output_sha256`** (full re-hash of all 5.5 GB, `n_mismatch = 1`). The sweep is otherwise byte-intact. Consequence: `intestinal_metaplasia oblique pass2-si6` is **n = 4, not 5**, and IM `pass2-si6` pooled is n = 14. Every table below carries the real n. The image is excluded everywhere and is not recoverable without regenerating on GPU. A second bookkeeping point found while checking: **8 pairs of sweep records are byte-identical** (same `output_sha256`) — every d1 `donorK` arm whose donor happens to be the a1 donor for that seed duplicates the corresponding a1 cell. EVAL_BUILD_RESULT.md limitation 11 predicted this for `donor0`; it happens for `donor0`, `donor1`, `donor2` and `donor3` depending on the seed. So the 407 records are **398 distinct images**. Two of the IM top-8 candidates (ranks 6 and 7) are the same file under two arm names. ### 1.2 Two harness bugs, both fixed in `packages/sweep-v1/evaluate_sweep.py` **(a) `--probes` aborted on the zero-byte PNG.** `run_probes.py` has no per-image error handling, so one bad PNG in an `--add-set` records file killed the whole probe run (`exit 1`, `PIL.UnidentifiedImageError`). The harness's own `features` stage handles this fine; only the probe subprocess did not. Fixed by adding `unreadable_outputs()` and filtering `probe_specs()`: any record whose `feat_status` is not `ok`, or whose output is missing or zero-length, is dropped from the probe input, logged, and written to `eval/probes-input/excluded.json` and into `stage-arms.json`. Rerun: `arms --probes` exit 0 in 267 s, one record excluded. **(b) the a2 own-source figure was structurally zero.** `_library_origins_for()` read `p.get("library_origin") or p.get("library_file")`, but in the real records `library_origin` is the **pixel coordinate** of the window inside the donor field (`[0, 512]`), not a path. `Path(str([0, 512])).name` is the string `"[0, 512]"`, which never matches a reference basename, so **every library arm reported `library_source_fraction = 0.000`** — silently, and it looked like a real finding ("a2 tiles never land on a conditioning field"). Fixed to prefer `library_file` and to ignore `library_origin` unless it is a string. After the fix a2's figure is **0.756 mean / 0.875 median (range 0.25-1.00)**, which is the number question (d) asks for. The `screens` stage was rerun with `--reuse-screen` (phikon reused, 417 NCC pairs recomputed, 577 s), then `report`. No other harness change was made, and no data was touched. --- ## 2. Extra measurements this package added The harness answers the questions it was built for. Four of the brief's questions needed numbers it does not produce, so these were measured with the harness's **own** functions (`evaluate_sweep._d1_one`, same masks, same config) so they are directly comparable: | output | what it adds | why | |---|---|---| | `eval/a1-layout-iou.csv` / `.json` | lumen IoU against the own cartoon label map for **every a1/a2/a3/d2/e2 arm**, not just d1 | d1 only ran pass 1 and pass-2 si6; the a1 ladder adds si9, si3 and pass 3, and "which arm keeps the cartoon" turned out to be the central question | | `eval/cartoon-iou-ceiling.csv` / `.json` | the IoU ceiling **per cartoon set and per (category, cut)** | the harness computes it for the d1 set only (0.407). The a1 set's ceiling is 0.604 and ranges 0.41-0.85 by cell, so a single ceiling would have been misleading | | `eval/within-arm-diversity.json` | mean pairwise SSIM **inside** each arm, with real held-out fields as the scale | nothing in the harness asks "are these N images, or one image N times" — and for a3, a2 and b1 that is the answer | | `eval/sheets/{b1-overview,b1-zoom-pass2,b1n-zoom,b1n-real-vs-generated,detail-1to1,a2a3-*}.png` | whole canvases, and 1:1-pixel crops at 2x | an H. pylori rod is 8-16 px at 0.25 um/px; the harness sheets downsample a 768 px crop into a ~330 px cell, where a rod cannot survive. Question (h) cannot be answered from those sheets | --- ## 3. Answers ### (a) A1 per category: topology and envelope, versus real and versus the old baseline Medians per category, pooled over the three cuts (n = 15 per arm except where marked). `env pct` is the percentage of that category's real held-out fields at or below this image's envelope distance, so 50 is the real median and **100 means outside the entire real held-out spread**. `D` is the C1 MinCovDet Mahalanobis. `lap` is C2's mean absolute Laplacian (real held-out 20.4-23.6). | category | arm | n | ring (mean) | ring (median) | env pct | D | in band % | lap | |---|---|---:|---:|---:|---:|---:|---:|---:| | **normal** | real held-out | 4 | 0.854 | 0.884 | 62 | 3.33 | 100 | 22.78 | | | OLD fullset pass 2 | 5 | **0.726** | 0.826 | 100 | **8.64** | 20 | 14.00 | | | a1 pass 1 (si12) | 15 | 0.386 | 0.441 | 100 | 8.48 | 0 | 11.90 | | | a1 pass 2 si6 | 15 | 0.339 | 0.473 | 100 | 7.43 | 0 | 15.64 | | | a1 pass 2 si3 | 15 | 0.591 | 0.678 | 100 | **7.10** | 7 | 18.12 | | **gastritis** | real held-out | 6 | 0.743 | 0.807 | 58 | 4.19 | 100 | 20.40 | | | OLD fullset pass 2 | 5 | **0.475** | 0.433 | 83 | **6.32** | 100 | 14.95 | | | a1 pass 1 (si12) | 15 | 0.371 | 0.387 | 100 | 11.53 | 0 | 15.47 | | | a1 pass 2 si9 | 15 | 0.110 | 0.056 | 100 | 9.77 | 7 | 15.78 | | | a1 pass 2 si6 | 15 | 0.062 | 0.026 | 83 | 7.76 | 60 | 17.55 | | | a1 pass 2 si3 | 15 | 0.394 | 0.294 | 83 | **5.77** | 100 | 18.60 | | | a1 pass 3 si15 | 5 | 0.020 | 0.000 | 100 | 8.67 | 20 | 16.76 | | **IM** | real held-out | 6 | 0.710 | 0.774 | 58 | 2.58 | 100 | 23.62 | | | OLD fullset pass 2 | 5 | **0.917** | 0.927 | 100 | **5.28** | 20 | 14.62 | | | a1 pass 1 (si12) | 15 | 0.405 | 0.459 | 100 | 7.81 | 0 | 12.33 | | | a1 pass 2 si6 | **14** | 0.251 | 0.244 | 100 | 5.56 | 21 | 16.90 | | | a1 pass 2 si3 | 15 | 0.645 | 0.506 | 100 | **4.67** | 40 | 19.32 | | | a1 pass 3 si15 | 5 | 0.111 | 0.000 | 100 | 6.36 | 0 | 15.84 | | **mixed** | real held-out | 6 | 0.726 | 0.774 | 58 | 3.66 | 100 | 22.77 | | | OLD fullset pass 2 | 5 | **0.726** | 0.766 | 100 | **6.74** | 0 | 16.64 | | | a1 pass 1 (si12) | 15 | 0.411 | 0.438 | 100 | 9.19 | 0 | 13.87 | | | a1 pass 2 si6 | 15 | 0.236 | 0.233 | 100 | 7.35 | 0 | 18.03 | | | a1 pass 2 si3 | 15 | 0.530 | 0.509 | 100 | **5.78** | 0 | 20.82 | (The bolded old-baseline ring means are exactly PLAN.md's 0.726 / 0.475 / 0.917 / 0.726, and the bolded D values are `morphometrics/scores.csv` set `fullset_pass2` medians 8.64 / 6.32 / 5.28 / 6.74.) **Did the A1 cartoons trade topology for realism? Yes, and the trade is real but partial.** - On **topology**, the new A1 cartoons are worse than the old baseline in three of four categories: ring mean 0.591 vs 0.726 (normal), 0.394 vs 0.475 (gastritis), 0.530 vs 0.726 (mixed) at the best new arm (si3, n = 15 each vs n = 5 baseline). Only **IM improves**, and it improves by moving *down* toward real: 0.645 vs 0.917, against a real held-out mean of 0.710. The old baseline's IM number was above real, not below it. - On the **envelope**, the new arms are better in all four: D 7.10 vs 8.64 (normal), 5.77 vs 6.32 (gastritis), 4.67 vs 5.28 (IM), 5.78 vs 6.74 (mixed). Median improvement 0.55-1.54 D. - On **detail** — the axis C2 called "the largest single lever on the tell" — the improvement is the largest of the three. si3 closes the Laplacian gap to real by **47% (normal), 67% (gastritis), 52% (IM), 68% (mixed)** relative to the old two-pass baseline (medians, n = 15 vs n = 5 vs n = 4-6 real). This is visible: `sheets/detail-1to1.png` at 1:1 pixels shows the old two-pass field as a flat watercolour wash with smooth featureless nuclei, and the new si3/si6 fields with markedly more contrast, granular cytoplasm and some intranuclear texture. They are still short of real, whose nuclei carry speckled chromatin and whose cytoplasm is visibly granular. - **Envelope percentile is a saturated instrument here.** 41 of 249 pass-2/pass-3 outputs are inside their category's held-out band at all; the median of every single a1 cell except three gastritis cells sits at pct = 100. Where the brief asks for "envelope percentile", the honest reading is "at or beyond the top of the real spread", and the *distance* D is the number with resolution. **Which cut keeps lumens best? `oblique`, at every pass-2 depth** (a1, pooled over categories, n = 20 per cut per arm): | cut | pass 1 | pass 2 si9 | pass 2 si6 | pass 2 si3 | |---|---:|---:|---:|---:| | across | 0.439 | 0.020 | 0.095 | 0.481 | | along | 0.091 | 0.034 | 0.174 | 0.496 | | **oblique** | **0.647** | **0.110** | **0.344** | **0.554** | `across` is the worst at pass 2 si6 (0.095) and `along` is the worst at pass 1 (0.091, a geometric consequence — A1_RESULT.md gap 5: an `along` plane spans the whole mucosal depth and shows tapering tubes rather than ring profiles). **Is any arm inside the real range on BOTH instruments?** Yes, but only 23 of 249 pass-2/pass-3 images (9.2%), and they are concentrated: | arm | n | ring inside real range | inside envelope band | **both** | |---|---:|---:|---:|---:| | a1 pass2-si3 | 60 | 39 | 22 | **14** | | a1 pass2-si6 | 59 | 15 | 12 | **3** | | a3 nearmean | 10 | 5 | 4 | **4** (all IM) | | a1 pass2-si9 | 15 | 1 | 1 | **1** | | e2 f3 | 5 | 2 | 1 | **1** | | everything else | 100 | 45 | 1 | **0** | | (reference) OLD fullset pass 2 | 20 | 14 | 7 | **4** | | (reference) real held-out | 22 | 22 | 22 | 22 | So the hit rate is 23/249 = 9.2% for the new sweep against 4/20 = 20% for the old baseline, but the new sweep's best single arm, a1 pass2-si3, is 14/60 = 23%. By category the 23 are **IM 14, gastritis 8, normal 1, mixed 0** — no mixed output anywhere in the sweep is inside both bands, and only one normal output is (`a1_p2_normal_s12_oblique_si3`, which is rank 2 on the normal candidate list). **Normal and gastritis specifically, against the old baseline:** - **normal** — *worse* on topology (ring mean 0.591 vs 0.726, real 0.854; ring-in-range 33% vs 40%), *better* on the envelope (D 7.10 vs 8.64, real 3.33), *better* on detail (lap 18.12 vs 14.00, real 22.78). Only 1 of 35 normal pass-2 images is inside both bands, against 2 of 5 for the baseline's ring and 1 of 5 for its envelope. Normal is the weakest category in the sweep on every count. - **gastritis** — *worse* on topology at si6/si9 (ring mean 0.062 / 0.110 vs 0.475) and roughly level at si3 (0.394 vs 0.475, real 0.743); *better* on the envelope at si3 (D 5.77 vs 6.32, real 4.19, and 100% in band vs 100% in band — note the gastritis band is the wide one, C1 limitation 2); *better* on detail (18.60 vs 14.95, real 20.40). The pass-2-makes-gastritis-worse effect recorded in PLAN.md is not just still present, it is much stronger on the new cartoons: pass 1 0.371 -> si6 0.062. Sheets: `sheets/a1--.png` (12). Looking at them: the A1 cartoons in column 1 are a dense field of small purple ovals on pink with a few white slots — a nuclear scatter, not a gland drawing. Pass 1 is a blurred version of exactly that and looks like nothing histological. Pass 2 at si6 and si3 is the first column that reads as gastric mucosa: columnar epithelium, foveolar pits, mucin-filled cells, capillaries with red cells. The real held-out column is visibly more *organised* than any of them — one or two large glands with an open lumen and a regular basal palisade, where the generated fields show many small irregular lumina and less orderly nuclear rows. ### (b) Does pass 3 (si15) raise detail toward real without moving topology? **No. It lowers detail, and it also lowers topology.** n = 10 pairs (5 gastritis + 5 IM, each pass-3 output against its own pass-2 parent): | quantity | median change | sign consistency | |---|---:|---| | mean absolute Laplacian | **-1.27** | 10/10 negative (-0.84 to -1.36) | | high-frequency energy ratio | -0.0027 | 10/10 negative | | mean saturation | -0.0098 | 10/10 negative | | envelope distance D | **+0.53** | 10/10 positive (+0.17 to +1.26) | | ring fraction | 0.000 | 6/10 unchanged, 4/10 negative (to -0.196) | Real held-out Laplacian is 22.77; pass 2 sits at 16.9 (IM) / 17.5 (gastritis) and pass 3 moves it *away*, to 15.84 / 16.76. Topology "held" only in the sense that the median change is zero — but the pass-3 arm's own ring median is 0.000 for both categories, because its pass-2 parents were already at 0.000-0.091. Pass 3 is a third light img2img (si15, 5 steps) on top of an already-repainted canvas; every step of it removes detail rather than adding it. On this evidence the si15 pass-3 arm should be dropped. ### (c) D1: is layout set by the cartoon and independent of the donor? **One-sentence verdict: at pass 1 the cartoon sets the layout almost completely and the donor sets nothing; by pass 2 (si6) that has reversed — the cartoon is gone and the donor decides what the canvas looks like.** The harness pools the two passes and returns "YES, 3 of 5 checks". That pooled answer is wrong, and the sheet says so at a glance. Split by pass (n = 25 outputs per pass, 5 cartoons x 4 donors + 5 rot180): | measurement | pass 1 (si12) | pass 2 (si6) | control | |---|---:|---:|---| | lumen IoU vs own cartoon label map (median) | **0.353** | **0.060** | ceiling 0.407 (the d1 cartoon PNG vs its own label map) | | — as a fraction of ceiling | 87% | 15% | 100% | | — by donor arm | 0.331-0.360 (spread 0.029) | 0.000-0.089 (spread 0.089) | flat = donor does not set layout | | rot180: IoU vs ROTATED map | **0.355** | 0.062 | | | rot180: IoU vs UNROTATED map | **0.033** | 0.020 | 10.7x preference at pass 1, 3.0x at pass 2 | | donor epithelium-mask NCC, max over shifts | **0.125** | **0.328** | native_generate 0.409, native_img2img 0.642 | | SSIM, same cartoon / different donor | **0.874** (n=30) | 0.505 (n=30) | | | SSIM, same donor / different cartoon | 0.301 (n=40) | **0.870** (n=40) | | | SSIM, unrelated | 0.282 (n=120) | 0.482 (n=120) | | The `sheets/d1-layout-factorial.png` grid (rows = 5 cartoons, columns = 4 donors, last column rot180, all pass 2) is unambiguous: **going down a column — same donor, five different cartoons — the five images are near-copies of each other**; the donor2 column is the same pale mauve field with the same dark diagonal streak in the upper right, five times. **Going across a row — same cartoon, four different donors — the four images have nothing in common.** That is what SSIM 0.870 vs 0.505 means in pixels. The donor NCC number does not contradict this. 0.328 is well below the native positive control (0.409 generate, 0.642 img2img), so the pass-2 canvas is *not* a spatial re-registration of the donor field. What the donor transfers is the tissue's *content and appearance* — which cell types, which stain, which texture — not its coordinates. That is the same structure-inheritance effect C2 measured; probe B here puts 77.2% of a1 tiles nearest their own donor field (old two-pass 89.4%, so the new recipe reduced it, but not by much). The a1 ladder, measured the same way on all 60 a1 cartoons against their own label maps and against a per-(category, cut) ceiling, shows the collapse is monotone in denoising depth: | arm | steps from the reference | lumen IoU | as % of that cell's cartoon ceiling | |---|---:|---:|---:| | pass 1 (si12) | 8 | 0.544 | **89%** (78-109% by cell) | | pass 2 si9 | 11 | 0.129 | 6-23% | | pass 2 si6 | 14 | 0.084 | 1-22% | | pass 2 si3 | 17 | 0.056 | 6-12% | | pass 3 si15 | +5 more | **0.000** | 0% | **This is the most consequential finding in the package**, because it inverts the reading of (a): the arm that scores best on the envelope, `pass2-si3`, is the arm that has thrown the most of the cartoon away. The programme is currently buying realism by abandoning the imposed geometry, so "the cartoon controls the layout" is only true of the pass-1 output, which is also the output that looks least like tissue (D 7.8-11.5, lap 11.9-15.5). **Caveat on the label maps used.** SWEEP_RESULT.md section 6 established that the VM-rendered cartoons are not byte-identical to the local ones (numpy/scipy/Pillow skew; the difference is confined to stromal-nuclei scatter, with lumen labels essentially identical). **The VM cartoons were not retrieved** — `packages/sweep-v1/retrieved/aux/cartoons/{a1,b1,d1,e2}/` contains only `MANIFEST.sha256.json`, no PNGs — and I confirmed the mismatch directly (`cartoons/d1/intestinal_metaplasia_s11_across_cartoon.png` hashes to `e7a6eeef…`; the record's `reference_sha256` is `ac6876dd…`). **Every IoU in this section therefore uses the LOCAL Mac label maps.** Since 99.8% of the label difference is the stroma/stroma-nucleus pair and label 1 (lumen) is essentially identical, the lumen IoU is the right metric to have used and the numbers should be very close to the VM-cartoon values — but they are not measured against the exact bytes the generator saw, and the epithelium IoU (labels 2,3,4) is slightly less safe. Retrieving the VM cartoons would remove this caveat for about 300 MB of transfer. ### (d) A2, composition-matched token library, versus a1 on the same cartoons 20 pass-2 canvases (4 categories x 5 `across` seeds), each paired with the a1 `pass2-si6` cell on the identical cartoon. Medians of the paired deltas: | category | n | a2 ring | Δring | a2 D | ΔD | a2 lap | Δlap | Δsat | |---|---:|---:|---:|---:|---:|---:|---:|---:| | normal | 5 | 0.041 | +0.010 | 6.73 | **-0.93** | 14.67 | -0.85 | +0.059 | | **gastritis** | 5 | 0.131 | +0.121 | 9.29 | **+1.57** | 12.12 | **-5.76** | +0.130 | | IM | 5 | 0.345 | +0.227 | 5.65 | -0.04 | 14.21 | -2.74 | +0.023 | | mixed | 5 | 0.109 | -0.051 | 6.26 | **-1.44** | 14.71 | -3.44 | +0.019 | | **pooled** | 20 | 0.155 | +0.094 | 6.28 | -0.10 | 14.40 | **-3.48** | +0.062 | - **Topology**: a small gain, +0.094 ring pooled, driven by IM (+0.227) and gastritis (+0.121); mixed loses 0.051. 5 of 20 land inside the real ring range. - **Envelope**: a wash pooled (-0.10 D), better for normal and mixed, clearly **worse for gastritis** (+1.57 D). **0 of 20 a2 canvases are inside any envelope band.** - **Detail**: a clear cost, -3.48 Laplacian pooled and -5.76 for gastritis. This is the largest single detail regression of any arm except d2 at high alpha. - **Seams**: not measurable with the instruments in this harness. The library picks tokens per 1024 px window (median 2 distinct donor fields per canvas, range 1-3), so a seam would be a window-boundary discontinuity; nothing here computes one, and at the 768 px sheet crop none is visible by eye. Recorded as not measured, not as absent. - **Own-source figure** (question as asked): a1's tiles land on **its single donor field 86.7% (si6) / 91.7% (si3)** of the time; a2's tiles land on **any** field the library drew from **75.6% mean, 87.5% median (range 25-100%)** of the time. So the library spreads the provenance, but only from ~90% one field to ~76% of two or three. Probe D agrees and is the cleaner statement: a1 sits at the **96.6th percentile** of real held-out manifold distance once its donor slide is removed from the bank, while a2 — which has no single donor slide to remove — sits at **76.1**. That is a genuine reduction in donor dependence. **Did it help gastritis? No.** It lifted the ring fraction off the floor (0.000 -> 0.131) but cost +1.57 D on the envelope and -5.76 Laplacian, and left 0 of 5 inside the band. On `sheets/a2a3-hpylori_gastritis.png` the five a2 canvases are visibly the same crowded field of purple cells with clear vacuoles, five times over, where the five a1 cells are five different fields. Which brings up the thing the harness does not measure: **A2 costs image diversity.** Mean pairwise SSIM *within* an arm (this package's `within-arm-diversity.json`; same coarse-grey SSIM the d1 stage uses): | set | n | mean pairwise SSIM | |---|---:|---:| | real held-out IM | 6 | **0.035** | | real held-out gastritis | 6 | **0.047** | | a1 pass2-si6 IM across | 5 | 0.503 | | a1 pass2-si3 IM across | 5 | 0.491 | | OLD fullset pass 2 IM | 5 | 0.477 | | **a2 library IM across** | 5 | **0.788** | | a3 mean IM / gastritis | 5 / 5 | 0.796 / 0.866 | | **a3 nearmean IM / gastritis** | 5 / 5 | **0.877 / 0.893** | | b1 organism | 5 | 0.864 | Real fields are essentially uncorrelated with each other (0.035). Every generated arm is an order of magnitude more self-similar, and a2/a3/b1 are close to producing one image repeatedly. ### (e) A3, mean and nearest-to-mean tokens: coherent tissue or mush? **Both, split by arm — and both are one image repeated.** | arm | category | n | ring | D | env pct | lap | inside both bands | |---|---|---:|---:|---:|---:|---:|---:| | mean | IM | 5 | 0.140-0.325 | 8.77-9.83 | 100 | 14.7-14.9 | 0/5 | | mean | gastritis | 5 | 0.000-0.016 | 9.49-11.13 | 100 | 17.5-18.3 | 0/5 | | nearmean | IM | 5 | **0.519-0.659** | **4.15-4.67** | 100 | 16.5-16.7 | **4/5** | | nearmean | gastritis | 5 | 0.000 | 9.78-10.50 | 100 | 18.8-19.5 | 0/5 | | (a1 si6 partner) | IM across | 5 | 0.000-0.685 | 5.05-7.75 | 100 | 14.1-19.8 | 1/5 | - **`mean` (grand mean of the category's tokens) is mush**, and the sheet is why I use that word: on `sheets/a2a3-intestinal_metaplasia.png` column 3 every image is the same pale, low-contrast field of loosely-packed cells with evenly-sized round nuclei — no gland, no lumen, no palisade, no goblet cell, no vessel. It is a cell suspension, not tissue. D 8.77-11.13, +2.62 D worse than its a1 partner (n = 10). - **`nearmean` (the real token grid nearest the mean) is coherent, and for IM it is the best-scoring tissue in the entire sweep**: D 4.15-4.67 against a real held-out IM median of 2.58 and max 4.50, ring 0.52-0.66 against a real median of 0.774, 4 of 5 inside the envelope band. Column 4 of that sheet shows clusters of clear goblet-like vacuoles between nuclear strands with pink stroma and occasional red cells — it reads as intestinal metaplasia. - **The caveat that kills it as a result**: the token grid is fixed for the whole arm, so the five IM `nearmean` canvases have mean pairwise SSIM **0.877**. It is one image, generated five times from five different cartoons that had no effect. Its four "inside both bands" images are one image counted four times. And for gastritis the same arm produces ring 0.000 on all five and D 9.78-10.50, so the coherence does not transfer between categories. - A3 `mean` has the second-largest copy-screen margin of any arm (max cosine 0.701, min margin 0.249, behind d2 alpha=1's 0.260), which is consistent: it is the furthest from any specific real field. ### (f) D2, self-conditioned pass 2 by alpha IM `across`, 5 cartoons per alpha, alpha = 0 is the a1 `pass2-si6` cell on the same cartoon. Every trend is monotone: | alpha | n | ring | D | lap | hf energy | cartoon lumen IoU | own-source tiles | max cosine | |---:|---:|---:|---:|---:|---:|---:|---:|---:| | 0 (a1 si6) | 5 | 0.091 | **5.59** | **16.95** | 0.0441 | 0.084 | 0.867 | 0.888 | | 0.25 | 5 | 0.056 | 6.62 | 14.93 | 0.0296 | 0.026 | 0.750 | 0.867 | | 0.50 | 5 | 0.124 | 7.22 | 13.46 | 0.0197 | 0.061 | 0.450 | 0.812 | | 0.75 | 5 | 0.231 | 8.25 | 12.21 | 0.0137 | 0.113 | 0.250 | 0.737 | | **1.00** | 5 | **0.258** | **9.31** | **11.11** | 0.0100 | 0.142 | 0.025 | 0.690 | **Topology rises with alpha, envelope and detail fall, and the envelope/detail loss is much the bigger effect.** From alpha 0 to alpha 1: ring +0.167, D **+3.72** (a 67% increase in distance), Laplacian **-5.90** (a 35% loss of the detail statistic that C2 identified as the biggest component of the tell), HF energy -0.034 (a 77% loss). **Does alpha = 1 (no real donor at pass 2) hold up? No.** `sheets/d2-ladder.png` is a clean visual gradient left to right: the same structures stay in the same places while the rendering gets progressively smoother and flatter, until at alpha = 1 the nuclei are uniform featureless ovals in a flat cytoplasm — the "stamped nuclei, flat wash" appearance C2's probe-B sheet described, in its purest form. The real donor's UNI tokens are doing substantial work at pass 2 and cannot be removed. The one thing alpha = 1 buys is provenance: own-source tiles fall 0.867 -> 0.025 and max phikon cosine 0.888 -> 0.690 (min copy margin 0.260, the **largest** minimum margin of any arm in the sweep). If the goal were minimum donor dependence at any quality cost, alpha = 1 is the arm; on realism it is the worst pass-2 variant except a3 `mean` and b1. ### (g) E2, cartoon fidelity ladder 0/1/2/3 IM `across`, 5 seeds per rung; level 2 is the shipped renderer, i.e. the a1 cell of the same seed. | rung | what the cartoon is | pass-2 ring | pass-2 D | pass-2 lap | pass-1 D | |---|---|---:|---:|---:|---:| | f0 | flat label colours, no texture, no blur | **0.267** | **5.32** | 16.34 | 7.97 | | f1 | + basement rim, per-nucleus jitter | 0.228 | 5.83 | 14.81 | 8.30 | | f2 | the shipped renderer (= a1) | 0.091 | 5.59 | **16.95** | 7.65 | | f3 | + optics, chromatin grain, stain drift | 0.105 | 5.76 | 16.15 | 6.83 | **There is no optimum, because there is no trend.** The envelope distance spread across the whole ladder is 0.51 D (5.32-5.83) on n = 5 per rung, which is smaller than the within-rung spread; the paired deltas against the a1 f2 cell are -0.21 / +0.23 / +0.17 D for f0 / f1 / f3. If anything the *crudest* cartoon (f0, flat colours, no texture at all) is marginally the best on both ring (0.267, the highest of the four) and D (5.32, the lowest) — and it is also the cheapest to render. `sheets/e2-ladder.png` shows why: the four cartoons in each row are visibly different (f0 flat purple ellipses on flat pink; f3 with a greenish optical haze and softer edges), and the four generated outputs in the same row are **visibly the same image**. The row-3 example even carries the same dark diagonal streak in all four outputs. Combined with the layout-IoU numbers (e2 pass-2 IoU 0.020-0.063, i.e. the cartoon is gone by pass 2 whatever its fidelity) the conclusion is that **cartoon photorealism is not a lever on this pipeline at all**; what survives the repaint is the coarse geometry, and even that only through pass 1. ### (h) B1, gastritis surface-compartment donors **Every b1 pass-2 canvas is much worse than a1 gastritis, and none of them shows anything organism-like.** | measurement | b1 organism (n=5) | a1 gastritis oblique si6 (n=5) | real held-out gastritis (n=6) | |---|---:|---:|---:| | envelope D | **11.32-16.13, median 13.45** | 7.65 | 4.19 | | env pct | 100 (5/5) | 83 | 58 | | ring fraction | 0.000-0.149, median 0.015 | 0.089 | 0.807 | | ring count | 46-73 | 56 | 43.5 | | lumen:tissue ratio | 0.105-0.161 | 0.032 | — | | nuclear density /mm2 | 4742-5791 | 5608 | 5996 | | stromal nuclei /mm2 | 650-1163 | 1217 | 4588 | | mean abs Laplacian | 16.7-17.6 | 17.07 | 20.40 | | probe D percentile vs real held-out | **98.9** | 71.0 (a1 pooled) | 50 | The paired delta against the a1 group median is **+5.80 D** (`pair_kind = group`, since a1 never rendered the b1 seeds 21-25). The b1 arm has the worst envelope distance of any arm in the sweep. **What the images actually are.** `sheets/b1-overview.png` shows the five whole canvases downsampled next to three a1 gastritis oblique canvases and two b1 cartoons. The five b1 outputs are **not gastric mucosa**: the upper 80% of each canvas is a mass of horizontal wispy eosinophilic strands — reading as loose fibrin, mucus strands or shredded collagen — in a nearly empty pale field, with isolated elongated dark nuclei, and only a thin strip of actual glandular mucosa along the bottom edge. All five seeds produced the same thing (mean pairwise SSIM 0.864). The a1 gastritis canvases in the same sheet show proper foveolar pits, glands, columnar epithelium and lamina propria. The mechanism is legible and is the same donor-dominance effect as D1: the twelve organism-library windows B1 selected are *surface / mucus-interface* windows (B1_RESULT.md's table: "foveolar pit lumen with apical mucin caps", "surface mucinous exudate", "surface mucus and fibrin band"), so conditioning a whole 4096x2048 canvas on tokens from one of them paints mucus and fibrin over the entire field. Conditioning on a donor whose content is one compartment gives you that compartment everywhere. **Organism-like rods: none.** I looked at 1:1 pixels (2x) on `sheets/b1-zoom-pass2.png` (384 px native crops, 1 px = 0.25 um, across the surface band of all five canvases) and on `sheets/b1n-zoom.png` and `sheets/b1n-real-vs-generated.png` (the 12 native 1024 px organism-donor windows next to the real library windows they came from). **I see no clusters of faint, uniform, gently curved 2-4 um rods lying free in luminal or mucus space, in any b1 or b1-native output.** What is present in the pale space is occasional 1-2 px pink/magenta specks that read as stain speckle, and elongated nuclei attached to cells. This is the expected result and it matches B1's own finding (0 of 60 real candidates graded `likely organisms`): the donors do not demonstrably contain organisms, so the generator has nothing to reproduce. **The b1 arm is a gastritis-surface-compartment-donor experiment and must never be described as an organism experiment.** The b1-native arm (12 windows) is a clean structure-inheritance demonstration: the generated 1024 window is recognisably a redrawing of its real donor window — same cell types, same arrangement, different pixels. Its detail statistics (Laplacian median 19.11) are **not comparable** to the canvas numbers: at 1024x1024 the analysis raster is 0.25 um/px, not the 0.6667 um/px the envelope is calibrated on, so the harness correctly excludes all 12 from every envelope table. ### (i) Copy screens on all pass-2 / pass-3 outputs **Nothing fails. No output needs to be excluded on copy grounds.** phikon-v2 embedding screen, 249 candidates (all pass-2 and pass-3 outputs), 1992 candidate tiles against 118 real reference fields (96 train + 22 held out), fail line 0.95: | arm | n | max cosine | median max | own-source tiles | conditioning-field tiles | flagged tiles | min margin | |---|---:|---:|---:|---:|---:|---:|---:| | a1 pass2-si3 | 60 | **0.905** | 0.868 | 0.917 | 0.917 | 0 | **0.045** | | a1 pass2-si6 | 59 | 0.888 | 0.851 | 0.867 | 0.867 | 0 | 0.062 | | a1 pass2-si9 | 15 | 0.841 | 0.775 | 0.542 | 0.542 | 0 | 0.109 | | a1 pass3-si15 | 10 | 0.887 | 0.839 | 0.838 | 0.838 | 0 | 0.063 | | a2 library | 20 | 0.852 | 0.812 | 0.000 | **0.756** | 0 | 0.098 | | a3 mean | 10 | **0.701** | 0.658 | 0.000 | 0.000 | 0 | **0.249** | | a3 nearmean | 10 | 0.874 | 0.860 | 0.000 | 0.000 | 0 | 0.076 | | b1 organism | 5 | 0.719 | 0.711 | 0.000 | 0.000 | 0 | 0.231 | | d1 donor0/1/2/3/rot180 | 25 | 0.851-0.895 | 0.840-0.887 | 0.675-0.950 | same | 0 | 0.055 | | d2 self0.25 → self1.0 | 20 | 0.867 → 0.690 | 0.792 → 0.636 | 0.750 → **0.025** | same | 0 | 0.083 → 0.260 | | e2 f0/f1/f3 | 15 | 0.871-0.882 | 0.828-0.840 | 0.875-0.900 | same | 0 | 0.068 | | **whole sweep** | **249** | **0.9049** | — | 0.640 overall | — | **0** | **0.0451** | The overall maximum, 0.9049, is on `a1_p2_pass2-si3__normal_s11_across_si3.png`; it is **0.045 below the 0.95 fail line**, which is the tightest margin in the sweep. For reference, the old two-pass fullset run reached 0.8859 on this identical screen and C2 calibrated a pixel-level copy at 0.987 against 0.66 for unrelated real fields. The screen `passed: true`. Pixel NCC search (`pixcell_patch_search.py`, 64 px window, stride 256, threshold 0.92, **targeted references**: each output against its own donor field plus up to two phikon nearest-neighbour reference fields): **417 pairs, max NCC 0.8635, median 0.533, 0 flagged windows, 417/417 `NO_COPIED_PATCHES`, 0 failures.** Highest per arm: a1 si3 0.864, b1 0.775, d2 self0.75 0.741, everything else ≤ 0.73. Two honest limits on that clean result. First, the NCC is a **targeted screen, not a proof** — 29,500 exhaustive pairs would be about 13 hours (EVAL_BUILD_RESULT.md section 5). Second, and more important, **the copy screen is the wrong screen for the risk that is actually present here**: probe A separates every one of the nine sweep sets from real tissue at 100% of fields called generated, and the real risk this sweep exposes is not copying but *donor dependence* (own-source tiles 0.87-0.92 for a1, probe D 96.6th percentile once the donor slide is removed) and *within-arm duplication* (section d). Neither is a copy, and neither is caught by a copy screen. ### (j) Candidates `eval/candidates.csv`, top-8 per category from `morphometrics/pareto_select.py` on (|ring − real held-out median|, envelope distance, copy margin = 0.95 − max cosine). All 32 are on the Pareto front. | category | rank | arm | ring | D | copy margin | pale tissue frac (real) | ring count (real) | |---|---:|---|---:|---:|---:|---:|---:| | normal | 1 | a1 pass2-si6 | 0.595 | 4.63 | 0.144 | 0.118 (0.193) | 37 (52) | | | 2 | a1 pass2-si3 | 0.816 | 3.66 | 0.078 | 0.272 | 38 | | | 3-5 | a1 pass2-si3 | 0.977 | 5.02-5.36 | 0.09-0.10 | 0.256-0.272 | 86-87 | | | 6 | a1 pass2-si3 | 0.818 | 5.34 | 0.083 | 0.235 | 33 | | | 7 | a1 pass2-si6 | 0.606 | 5.89 | 0.136 | 0.149 | 104 | | | 8 | a2 library | 0.190 | 7.31 | 0.189 | 0.029 | 58 | | gastritis | 1-3 | a1 pass2-si3 | 0.964-1.000 | 5.49-5.77 | 0.050-0.066 | **0.354-0.373** (0.229) | **27-29** (43.5) | | | 4 | a1 pass2-si3 | 0.389 | 5.01 | 0.103 | 0.218 | 18 | | | 5-6 | a1 pass2-si9 | 0.190-0.462 | 7.38-8.02 | 0.114-0.193 | 0.091-0.116 | 58-91 | | | 7 | a1 pass2-si3 | 0.294 | 5.33 | 0.107 | 0.201 | 17 | | | 8 | b1 organism | 0.149 | **11.32** | 0.239 | 0.138 | 67 | | IM | 1-2 | a1 pass2-si3 | 0.875-0.912 | **3.17-3.21** | 0.086-0.089 | 0.286-0.303 | 57-64 | | | 3 | a3 nearmean | 0.659 | 4.38 | 0.088 | 0.119 | 88 | | | 4-5 | e2 f1 / e2 f0 | 0.690-0.701 | 5.27-5.28 | 0.068-0.079 | 0.132-0.143 | 97-100 | | | 6-7 | a1 pass2-si6 / **d1 donor3 (same file)** | 0.685 | 5.05 | 0.063 | 0.124 | 89 | | | 8 | a1 pass2-si6 | 0.478 | 4.14 | 0.099 | 0.155 | 69 | | mixed | 1-3 | a1 pass2-si3 | 0.730-0.772 | 5.13-5.38 | 0.095-0.098 | 0.237-0.262 | 57-63 | | | 4-5 | a1 pass2-si3 | 0.306-0.371 | 4.46-4.49 | 0.104-0.110 | 0.117-0.125 | 62 | | | 6-8 | a2 library | 0.107-0.151 | 5.77-6.30 | 0.148-0.157 | 0.088-0.112 | 119-139 | **My honest view of `sheets/candidates.png`.** - **IM and mixed have images I would put in front of a pathologist**, for the narrow question "does this read as gastric tissue of this type". IM ranks 1-2 (`intestinal_metaplasia_s12_across_si3`, `_s12_oblique_si3`, D 3.17 / 3.21 against a real held-out median of 2.58 and max 4.50 — inside the real band) show gland profiles with pale vacuolated cytoplasm and a recognisable epithelial border; mixed ranks 1-3 show a surface epithelium with a columnar band and vessels carrying red cells. IM is also where the sweep concentrates its "inside both bands" images (14 of 23). **Mixed is a split verdict**: it looks the second-best of the four on the sheets, but **0 of its 35 pass-2/pass-3 outputs is inside the envelope band** (its best is D 4.46 against a held-out max of 4.38), so the visual impression and the envelope disagree and only the visual half is favourable. - **Normal is not there.** Ranks 3-5 are three views of the same seed at ring 0.977 and D 5.0-5.4, and what they show is scattered cells with bright red-orange globules in a loose vacuolated matrix — not the parallel, tightly-packed straight gland profiles of oxyntic mucosa that the real held-out column shows. The a1 normal sheets confirm it: pass 2 on normal cartoons produces sheets of cells with variable amounts of blood rather than an ordered glandular field. - **Gastritis is the category that is still clearly not there,** and it is the one where the *selection objective is being gamed*. Ranks 1-3 have ring 0.964-1.000, right on the real median of 0.807, so the Pareto front loves them — but they achieve it with **27-29 detected rings against a real median of 43.5** and a **pale tissue fraction of 0.354-0.373 against a real 0.229**, and on the sheet they are a washed-out hypocellular honeycomb. This is exactly the failure mode C1 demonstrated in its section 7 (fading a real field 40% toward white *raises* its ring fraction from 0.805 to 0.973 while pushing it out of the envelope). The more plausible-looking gastritis candidates are ranks 4-6 (one si3 and two si9), which show nucleated columnar epithelium and which the ranking puts below the pale ones. Separately, the whole category is missing what defines it: stromal nuclear density is 1060-1300/mm2 across the nine a1 gastritis pass-2 cells against a real held-out 4588/mm2, i.e. the inflammatory infiltrate is largely absent (C1 failure mode 3, unchanged by this sweep). - Two structural problems with the list itself: IM ranks **6 and 7 are byte-identical** (section 1.1), and gastritis rank 8 is a b1 canvas at D 11.32 that is on the front only because its copy margin (0.239) is the best in the category — the third objective can promote a visibly broken image. A results page quoting this list should drop the duplicate and say why rank 8 is there. --- ## 4. What I would change next, on this evidence 1. **The cartoon does not survive pass 2.** Any further work on cartoon realism (E2's whole premise) is wasted; the ladder is flat and the outputs are indistinguishable across it. The lever is the *pass-2 conditioning*, not the cartoon. F2's structure adapter, which conditions on the label map directly rather than on a rendered RGB reference, is the arm that actually attacks this. 2. **si3 (17 steps) beats si6 on topology, envelope and detail in all four categories**, and si9 and pass-3-si15 should be dropped (si9 is worse than si6 on the envelope in all three gastritis cuts, and pass 3 is negative on every axis). But si3's advantage comes from abandoning the cartoon (layout IoU 9% of ceiling), so it is a better *image* generator and a worse *layout* generator. Pick which one the programme wants before optimising further. 3. **Diversity needs to become a first-class metric.** Real held-out fields are mutually uncorrelated (SSIM 0.035); every arm here is 0.48-0.89. The Pareto selection has no diversity objective and will happily return five copies of one image (a3 nearmean IM is exactly that). 4. **The gastritis selection objective is being gamed** and needs a cellularity or pale-fraction guard before any gastritis candidate goes to review. 5. **Retrieve the VM cartoons** (~300 MB) so the layout IoUs are measured against the exact bytes the generator saw. --- ## 5. Caveats carried into every number above 1. Not clinical, not a review. No pathologist; Yash has not reviewed a sheet. 2. **Envelope percentile is saturated**: 208 of 249 pass-2/pass-3 outputs sit at pct = 100. Use D. 3. **The envelope calibration is thin** (C1 limitation 2): 4-6 real held-out fields per category from 2-3 slides, so percentiles quantise to 25% (normal) / 16.7% (the rest), and the gastritis band is wide because one held-out field is atypical — "100% of gastritis si3 in band" is a weaker statement than the same phrase for IM. 4. **`ring_with_lumen_fraction` does not separate the four categories on real tissue** (C1 section 6, eta2 = 0.007, p = 0.89). It is a plausibility gate, never a target — and section 3(j) shows what happens when a selection objective treats it as one. 5. **The pixel NCC is targeted, not exhaustive**, and is a screen rather than a proof. 6. **d1's layout IoU uses the LOCAL Mac cartoons**, not the VM cartoons the sweep consumed (section 3c). The lumen labels are essentially identical between the two renders; the epithelium labels are less safe. 7. **`str_nuc_density_per_mm2` is a noisy residual mask** (C1 limitation 5) — comparable across arms because every image is measured identically, not a real stroma segmentation. 8. **Detail statistics cannot certify 0.25 um/px content** (C1 section 7): a half-resolution round trip moves neither the ring metric nor the envelope. A higher Laplacian means "more of something", not "genuine sub-micron detail". 9. **b1's 12 native windows analyse at 0.25 um/px**, not the envelope's 0.6667, and are excluded from every envelope table; their detail statistics are not comparable with the canvas numbers. 10. **B1 is not organisms** (B1_RESULT.md: 0 of 60 candidates graded `likely organisms`). 11. **Category and donor stay confounded** for a1, d1, e2 and d2 by design; only a2 and a3 break it, and probe B cannot attribute a donor for those arms (0.000 by construction), so their structure-inheritance row is blank rather than zero. 12. **Small n throughout**: 5 per (category, cut, arm) cell, 5-6 real held-out fields per category, 5 per d2/e2 rung. There is no significance testing anywhere in this package and there is not enough data for any. 13. `packages/sweep-v1/eval-standin*/` are plumbing fixtures with fabricated provenance and must never be quoted; only `packages/sweep-v1/eval/` is this run.