EVAL-BUILD: a rerunnable evaluation harness for the Phase-2 sweep

Work package EVAL-BUILD of the SGH programme (operations/research/sgh-program-20260908/PLAN.md), 8 September 2026. CPU only, research/sgh-synthetic-histopathology/.venv/bin/python. Nothing was committed to git.

Engineering evidence only. The harness measures shape, spacing, texture, colour and one pathology encoder. It does not diagnose, grade or stage anything, and no number it produces says an image is diagnostically correct.

Status: built, dry-run end to end on real PNGs, and smoke-tested on the first two real sweep outputs to arrive (section 6.5). The merged sweep has not landed, so every number the harness has produced so far is a re-measurement of the existing two-pass fullset run, a single-cell smoke test, or a plumbing fixture with fabricated provenance. Nothing in this document is a Phase-2 result.

1. What was built

Nothing in research/sgh-synthetic-histopathology/ was edited. morphometrics/, probes/, code/ and the two SGH screen scripts are imported or shelled out to, never copied and never modified:

• morphometrics/morphometrics.py -> extract(), load_config(), ENVELOPE_FEATURES
• morphometrics/calibrate.py -> score_rows() (the exact MinCovDet scoring C1 fitted)
• morphometrics/envelope.json -> the per-category envelope, unchanged
• morphometrics/pareto_select.py -> subprocess, for select
• code/cartoon_metrics.py -> measure() (ring topology + composition), prefixed cm_
• probes/run_probes.py -> tile_stats() for the C2 detail statistics; all --add-set as a subprocess
• scripts/pixcell_embed_screen.py and scripts/pixcell_patch_search.py -> subprocess / search()

Self-check that the instrument is the same instrument. The harness rescores the 22 real held-out fields from scratch and compares against C1's morphometrics/scores.csv: max absolute difference in Mahalanobis distance 4.9e-06 over 22 fields. And the phikon copy screen on the 20 fullset pass-2 canvases reproduces pixcell-fullset-20260908/screen-pass2-phikon-fixed.json exactly - max cosine 0.885864794254303, mean 0.798949894681573, 160 candidate tiles, 944 reference tiles.

2. How to run

cd [local]/sgh-program-20260908
P=../../../research/sgh-synthetic-histopathology/.venv/bin/python

# when the sweep lands, this is the whole thing
$P packages/sweep-v1/evaluate_sweep.py all \
     --records packages/sweep-v1/retrieved/records.jsonl \
     --out     packages/sweep-v1/eval

# add the phikon probe re-run (slower; snapshots and restores C2's probes/ outputs)
$P packages/sweep-v1/evaluate_sweep.py all --records ... --out ... --probes

# any stage on its own; `report` reads the stage-*.json the others wrote
$P packages/sweep-v1/evaluate_sweep.py features --records ... --out ...
$P packages/sweep-v1/evaluate_sweep.py a1 --records ... --out ...
$P packages/sweep-v1/evaluate_sweep.py d1 --records ... --out ...
$P packages/sweep-v1/evaluate_sweep.py arms --records ... --out ...      # a2 a3 d2 e2 b1 together
$P packages/sweep-v1/evaluate_sweep.py e2   --records ... --out ...      # or one at a time
$P packages/sweep-v1/evaluate_sweep.py screens --records ... --out ...
$P packages/sweep-v1/evaluate_sweep.py sheets  --records ... --out ...
$P packages/sweep-v1/evaluate_sweep.py select  --records ... --out ...
$P packages/sweep-v1/evaluate_sweep.py report  --records ... --out ...

# partial sweep arriving in stages: features is incremental, so just re-run `all`
$P packages/sweep-v1/evaluate_sweep.py all --records ... --out ... --skip screens

Useful flags: --jobs N (default cpu_count-2), --limit N (first N records, development), --root DIR (if output paths are relative to something other than the records file's directory), --ncc-refs {targeted,all,none} and --ncc-stride (section 5), --ncc-top-refs, --reuse-screen, --keep-screen-dirs, --top-k (default 8), --sheet-rows, --reference-train.

Inputs it expects

Outputs

eval/features.csv (one row per record and per reference image: every cm_*, C1 and det_* feature plus every extra field plus the envelope distance and percentile), eval/scores.csv, eval/a1-table.csv, eval/d1-measurements.csv, eval/d1-pairwise-ssim.csv, eval/arms-pairs.csv, eval/screen-embed.csv, eval/screen-ncc.csv, eval/copy-margin.csv, eval/select-scores.csv, eval/candidates.csv (+ per-category files), eval/sheets/*.png, eval/stage-*.json and eval/EVAL_REPORT.md. The cache lives at eval/cache/features-cache.csv, keyed by the output's sha256, so a rerun after a partial retrieval recomputes only the new images (measured: the second features run over 265 records finished in 0.4 s).

3. What each stage answers

1. features - for every record's PNG: cartoon_metrics.measure() (ring topology, tissue / lumen / nuclear / stroma composition, prefixed cm_), C1's 54 morphometric features, and C2's no-encoder detail statistics averaged over the field's 1024 px tiles (det_lap_abs_mean, det_hf_energy_ratio, det_sat_mean, det_gray_entropy, det_white_fraction). Joined with extra and scored against envelope.json (Mahalanobis, robust-z RMS, percentile against the real held-out fields of that category, inside-band flag, ring gap, three worst features). The 22 real held-out fields and the 20 fullset pass-2 canvases are extracted the same way so the a1 table's reference rows are like-for-like. Images that analyse to a different effective um/px (anything not 4096x2048, e.g. the b1 native 1024 windows) are flagged and excluded from every envelope-based table.
2. a1 - median per (category x cut x arm) over pass1, pass2-si6, pass2-si9, pass2-si3, pass3-si15, with the real held-out row and the fullset pass-2 baseline row. Then: for each category, the pass-2 arms whose median ring fraction is inside the real held-out range, and of those the lowest median envelope distance - reported with which rule fired. Then pass 3 against its own pass-2 parent (from extra.parent, falling back to the same cartoon): median change in Laplacian / HF energy / saturation and in ring fraction, the latter compared against the spread of the pass-2 cells so "moved" means something.
3. d1 - per output: lumen-mask IoU against its own cartoon's label == 1; for rot180 arms the same against the unrotated map; epithelium- and lumen-mask normalised cross-correlation (max over shifts and at zero shift) and grayscale SSIM against the donor field; and pairwise SSIM among all d1 outputs, split into same-cartoon/different-donor, same-donor/different-cartoon and unrelated. Two controls make the numbers readable: the ceiling (each cartoon PNG measured against its own label map - 0.407 lumen IoU on the d1 set, so a generated output can never reach 1.0 on this metric) and the positive control (native slot-3 canvases conditioned on the same donors, split into native_generate and native_img2img). The verdict is five independent yes/no checks, each printed with its control, not one number.
4. arms (also a2, a3, d2, e2, b1 individually) - each record paired with the a1 cell on the same base cartoon and the same pass (_f<level> and _rot180 suffixes are stripped); where a1 never rendered that cartoon (b1 uses seeds 21-25) the delta is against the a1 median for the same category, cut and pass, and the row says so (pair_kind=group). Deltas: ring fraction, envelope percentile, envelope distance, the three detail statistics, nuclear density. For a2, the per-window library provenance from token_library_picks (distinct donor fields per canvas) and the own-source figure. With --probes, run_probes.py all --scales 1024 re-runs with one --add-set per experiment, so probes A, B and D report per experiment rather than lumping the sweep together.
5. screens - pixcell_embed_screen.py (phikon-v2, fail line 0.95) over every pass-2 and pass-3 output against all 118 real fields (96 train + 22 held out), then pixcell_patch_search.py (window 64, threshold 0.92). Per arm: max and median max cosine, own-source tile fraction, any-conditioning-field tile fraction, flagged tiles, minimum copy margin, max NCC, flagged NCC windows. The own-source columns are computed from the per-tile nearest-reference records, not from the screen's own filename heuristic, so library and blend arms are handled correctly.
6. sheets - labelled contact sheets, each cell the same 768 px crop at canvas position (1664, 640, 2432, 1408), sheets at most 2000 px wide: a1 per (category, cut) as cartoon | pass1 | pass2-si6 | pass2-si9 | pass2-si3 | pass3-si15 | real held-out; d1 as a cartoon x donor grid with the rot180 arm in the last column; e2 interleaving each fidelity cartoon with the canvas it produced; d2 alpha ladder; a3 arms; a2 vs a1 pairs; b1.
7. select - writes a scores.csv-shaped file and a copy-margin join, then calls morphometrics/pareto_select.py per category with --n 8 on (|ring - real held-out median|, envelope distance, copy margin = 0.95 - max cosine), and builds a candidates sheet.
8. report - eval/EVAL_REPORT.md: every table, a plain-language verdict per experiment, and nine caveats carried over from C1, C2 and B1.

Every stage that finds nothing prints no data: <what was missing> and continues. Verified: on the faithful stand-in, d1 / a2 / a3 / d2 / e2 / b1 and the pass-3 comparison all report no data and the run still completes and writes a report.

4. Runtime

Measured on this Mac (14 cores, --jobs 12), on the stand-ins:

Rerunning after the first pass is far cheaper: the feature cache is keyed on the output sha256, and --reuse-screen skips the phikon pass. The 405 outputs are all distinct, so unlike the stand-in there is no dedup saving on the first run.

The dominant cost is the pixel NCC search, and it is a deliberate trade (section 5).

5. The one methodological decision worth arguing about

pixcell_patch_search.py costs about 0.10 s per 64 px window per (candidate, reference) pair. The sweep has 250 pass-2/pass-3 outputs; against all 118 reference fields at stride 256 that is 29,500 pairs, about 13 hours on 12 cores (and 4x that at the script's default stride of 128). Batching the FFT across windows was measured and only buys 2.4x, which is not enough, and it is already lost to running pairs in parallel.

The default --ncc-refs targeted therefore searches each output against:

• its own donor field (the only reference the generator was actually shown), and
• the reference fields that are the nearest neighbour of any of its 1024 px tiles in the phikon screen, up to --ncc-top-refs (default 2).

The prefilter is sound on C2's own calibration: a pixel-level copy scores 0.987 embedding cosine against 0.66 for unrelated real fields, so a copy cannot hide from the embedding screen and then be missed by a targeted NCC search. --ncc-refs all restores the exhaustive search; the harness prints the estimated runtime before starting either way. --ncc-refs none skips it.

This is recorded as caveat 5 in the generated report, not buried here.

6. What the dry runs actually did

6.1 The stand-ins

make_standin.py builds both from operations/research/pixcell-fullset-20260908/retrieved/out/records.jsonl (40 records, 30 PNGs present; the 10 pass-1 normal and IM outputs were never retrieved and are deliberately kept in the stand-in so the missing-file path is exercised).

• standin/ - the faithful one the brief asks for: pass / cartoon / category mapped to the Phase-2 schema, exp="a1", arm="baseline", real donors from donors.json, outputs laid out as slot{0,1,2}/out/a1/pass{1,2}/<category>/<stem>.png (symlinks). Every experiment other than a1 has no data, which is what tests the no-data paths.
• standin-rich/ - 265 records over the same 30 PNGs, re-labelled so a1 (pass1 / si6 / si3 / si9 / pass3-si15), d1 (5 cartoons x 4 donors + 5 rot180), a2 (with a fabricated 21-window token_library_picks), a3, d2, e2 and b1 all have records. The pixels are unrelated to the provenance stamped on them. It is a plumbing fixture; every table it produces is meaningless as science and is labelled as such by make_standin.py on stdout.

6.2 Results of the faithful dry run (eval-standin/)

These numbers are real - they are a re-measurement of the existing two-pass fullset run through the new harness, and they reproduce what is already on record:

6.3 Results of the rich dry run (eval-standin-rich/)

Every stage ran; the artifacts are the shape the real sweep will produce.

Two numbers from it are worth keeping even though the provenance is fabricated, because they are properties of the measurements, not of the fixture:

• the d1 lumen-IoU ceiling is 0.407: a cartoon PNG measured against its own label map by the image-only lumen detector. Any generated output's IoU has to be read against that, not against 1.0.
• the donor cross-correlation has usable dynamic range: native_img2img (a light repaint of the donor field itself) 0.642, native_generate (built from the donor's tokens) 0.409, the cartoon-conditioned pass-2 canvases 0.314.

6.4 Broken-input fixture (eval-broken/)

A deliberately hostile records file: a truncated PNG, a zero-byte PNG, a record pointing at a file that does not exist, a record with an experiment nobody has heard of, a record with no category, and a line of plain text instead of JSON. Result: all completed, the non-JSON line was warned about and skipped, the missing file was listed, both unreadable PNGs were recorded in stage-features.json as error: OSError: image file is truncated and error: UnidentifiedImageError: ... and excluded from every table, the unknown experiment passed through as its own arm, and the report was written. Nothing crashed.

6.5 Smoke test on the first two real sweep PNGs (eval-early/)

While this package was being written the fleet pulled back the first two Phase-2 outputs into packages/sweep-v1/early/ (the intestinal_metaplasia_s11_across pass-1 and pass-2 cells from slot a). Running features and a1 on them with the real extra block copied out of jobs/a1-pass{1,2}.json worked with no change to the harness, which is the first evidence that it handles actual sweep provenance and not just the fixture:

$P packages/sweep-v1/evaluate_sweep.py features \
     --records packages/sweep-v1/eval-early/records-early.jsonl \
     --root    packages/sweep-v1/early \
     --out     packages/sweep-v1/eval-early

(the records file lives under eval-early/ so nothing is written into early/, which the retrieval side owns; --root points the harness at the PNGs)

This is n = 1 and means nothing yet. It is reported only to show the pipeline runs. The one thing worth flagging for whoever reads the real run: pass 2 moved the envelope distance the expected way (7.09 -> 5.59) while dropping the ring fraction to 0.202, which is at the bottom of the real held-out IM range and far below the old cartoon set's pass-2 values. If that holds across the 15 IM cells, the a1 stage's "topology inside the real range" gate will be doing real work.

packages/sweep-v1/retrieved/merge_records.py (written by the retrieval side while this package ran) was read for compatibility: it rewrites each record's output to slot<i>/out/<vm output> relative to packages/sweep-v1/retrieved/, keeps extra untouched and adds output_vm, slot and fleet_slot. That is exactly what this harness's default --root and provenance reader expect, so no adapter is needed.

6.6 The probe re-run does not damage package C2

run_probes.py writes its 21 JSON and PNG deliverables straight into probes/, which are C2's outputs. The harness snapshots them, runs the probes, copies the sweep-augmented results into eval/probes/, and restores the originals. Verified by md5 before and after: probes/summary.json, probes/probeB-tell.json and probes/probeD-manifold-1024.json are byte-identical after two --probes runs. The probes/embeddings/*.npz cache does grow (additively), which is the intended behaviour and is what makes the second run 12 s instead of 4 minutes.

7. Bugs found and fixed while dry-running

1. The phikon screen resolved symlinks. pixcell_embed_screen.py calls Path.resolve() on every candidate, so a staging directory of symlinks reported the target's basename; every candidate row failed to join back to its record, and two arms sharing a cartoon stem would have collided silently. Fixed by staging with hard links (which keep their own name and cost no disk), with a copy fallback and a warning if any screened name still fails to join.
2. pass arrives as a float from the CSV round trip, so p{pass} in the screen filename became p2.0. Fixed.
3. sha256 was being coerced to a float (and became nan) when features.csv was read back.
4. b1 has no a1 partner by construction (a1 renders seeds 11-15, b1 uses 21-25), so every b1 delta was blank. Added a two-level group fallback: same category+cut+pass, then same category+pass, with pair_kind recorded per row.
5. The pass-3 verdict said "detail down" for a zero change. Now says "unchanged" below 0.05.
6. The d1 IoU had no ceiling, so 0.046 could not be told apart from "the metric cannot go high". Added the cartoon-against-its-own-label-map control.
7. The a1 sheet mixed cuts in one grid, making rows non-comparable. Now one sheet per (category, cut).
8. The screens stage left 118 + N hard links behind; they are now cleaned up unless --keep-screen-dirs.

8. Limitations

1. Not clinical, and not a review. No pathologist has seen any of this, and Yash has not reviewed a single sheet. The verdicts are arithmetic over classical measurements.
2. The harness has never seen a real Phase-2 output. Every code path has executed on real PNGs, but those PNGs came from the fullset run, not the sweep. Cartoon stems, arm names, token_source values and token_library_picks shapes were taken from jobs/*.json and GEN_RESULT.md; if the sweep emits something else, the affected stage will report no data rather than fail, and the mapping in arm_label() / base_cartoon() is the place to fix it.
3. The pixel NCC screen is targeted, not exhaustive (section 5). It is a screen, not a proof of non-copying.
4. The envelope calibration is thin - 4 to 6 real held-out fields per category from 2 to 3 slides, so pct_vs_heldout quantises to 25% steps for normal and 16.7% for the rest (C1_RESULT.md limitation 2). It is ordinal.
5. ring_with_lumen_fraction does not separate the four categories on real tissue (eta2 = 0.007, p = 0.89, C1_RESULT.md section 6). The a1 stage uses it only as a plausibility gate ("is the topology in the real range") and never as the thing being optimised.
6. The d1 layout verdict is five heuristic checks with a majority vote. The individual checks and their controls are printed; the single boolean is a convenience, not a test statistic. There is no significance testing anywhere in the harness - with 5 cartoons and 4 donors there is not enough data for it.
7. str_nuc_density_per_mm2 is noisy. C1's "stroma" is a small residual of a 6 um-closed hematoxylin band (5-10% of tissue). It is comparable across arms because every image is measured identically, but it is not a real stroma segmentation.
8. Detail statistics cannot certify 0.25 um/px content. They are computed on 1024 px tiles resized to 512, and C1 showed a half-resolution round trip moves neither the ring metric nor the envelope. A pass-3 arm that raises the Laplacian has raised something; whether it is genuine sub-micron detail is not answered here.
9. Probe B cannot attribute a donor to library or mean-token arms - they have no token_reference, so those rows read 0.000 by construction. Their own-source figure comes from the harness's own tile-level join in the A2 block instead.
10. B1 is not organisms. B1_RESULT.md graded 0 of 60 candidates as likely organisms. The b1 arm is labelled throughout as "gastritis surface-compartment donors", and the report repeats the caveat.
11. d1's donor0 arm duplicates a1's pass-1 cell by construction (GEN_RESULT.md deviation 4), so it is not an independent sample.
12. Category and donor stay confounded for a1 and d1: the cartoon's category always equals the donor's. a2, a3 and d2 are the arms that break it, and the harness reports them, but the confound is a property of the experiment design, not something the evaluation can remove.
13. The stand-in outputs left in the tree are fixtures, not results. packages/sweep-v1/eval-standin*/EVAL_REPORT.md must never be quoted as a Phase-2 finding. Delete them, or --out somewhere else, once the sweep lands.

9. What is not done

• The sweep has not run, so packages/sweep-v1/retrieved/records.jsonl does not exist and the harness has produced no Phase-2 answer.
• The --probes path re-runs C2's probes but does not re-run C1's calibrate.py; the envelope is taken as fitted. If the real fields ever change, refit C1 first.
• No F2 (structure adapter) arm is handled. F2's outputs are a Phase-3 package with their own records; adding them is a matter of pointing --records at them, but no f2 branch exists in stage_arms.
• The reviewer pack v2 and the results page (PLAN.md Phase 4) are downstream of this harness and are not built here.

Download public Markdown export