# EVAL-BUILD: a rerunnable evaluation harness for the Phase-2 sweep Work package EVAL-BUILD of the SGH programme (`operations/research/sgh-program-20260908/PLAN.md`), 8 September 2026. CPU only, `research/sgh-synthetic-histopathology/.venv/bin/python`. Nothing was committed to git. Engineering evidence only. The harness measures shape, spacing, texture, colour and one pathology encoder. It does not diagnose, grade or stage anything, and no number it produces says an image is diagnostically correct. **Status: built, dry-run end to end on real PNGs, and smoke-tested on the first two real sweep outputs to arrive (section 6.5). The merged sweep has not landed, so every number the harness has produced so far is a re-measurement of the existing two-pass fullset run, a single-cell smoke test, or a plumbing fixture with fabricated provenance. Nothing in this document is a Phase-2 result.** --- ## 1. What was built | file | lines | what it is | |---|---:|---| | `packages/sweep-v1/evaluate_sweep.py` | 2558 | the harness: 8 stages, multiprocessing, sha256-keyed feature cache | | `packages/sweep-v1/make_standin.py` | 304 | builds the two stand-ins the harness was developed against | | `packages/sweep-v1/standin/` | - | faithful stand-in: 40 records over the 30 retrieved fullset PNGs, `exp="a1"`, `arm="baseline"` | | `packages/sweep-v1/standin-rich/` | - | 265 records over the same 30 PNGs, re-labelled so every experiment has data | | `packages/sweep-v1/eval-standin/` | - | dry-run output of the faithful stand-in (includes the phikon screen) | | `packages/sweep-v1/eval-standin-rich/` | - | dry-run output of the rich stand-in, every stage, `--probes` included | | `packages/sweep-v1/eval-broken/` | - | dry-run output of the deliberately broken fixture (section 6.4) | | `packages/sweep-v1/eval-early/` | - | smoke test on the first two REAL sweep PNGs (section 6.5) | Nothing in `research/sgh-synthetic-histopathology/` was edited. `morphometrics/`, `probes/`, `code/` and the two SGH screen scripts are **imported or shelled out to**, never copied and never modified: - `morphometrics/morphometrics.py` -> `extract()`, `load_config()`, `ENVELOPE_FEATURES` - `morphometrics/calibrate.py` -> `score_rows()` (the exact MinCovDet scoring C1 fitted) - `morphometrics/envelope.json` -> the per-category envelope, unchanged - `morphometrics/pareto_select.py` -> subprocess, for `select` - `code/cartoon_metrics.py` -> `measure()` (ring topology + composition), prefixed `cm_` - `probes/run_probes.py` -> `tile_stats()` for the C2 detail statistics; `all --add-set` as a subprocess - `scripts/pixcell_embed_screen.py` and `scripts/pixcell_patch_search.py` -> subprocess / `search()` **Self-check that the instrument is the same instrument.** The harness rescores the 22 real held-out fields from scratch and compares against C1's `morphometrics/scores.csv`: max absolute difference in Mahalanobis distance **4.9e-06** over 22 fields. And the phikon copy screen on the 20 fullset pass-2 canvases reproduces `pixcell-fullset-20260908/screen-pass2-phikon-fixed.json` exactly - max cosine 0.885864794254303, mean 0.798949894681573, 160 candidate tiles, 944 reference tiles. --- ## 2. How to run ```bash cd [local]/sgh-program-20260908 P=../../../research/sgh-synthetic-histopathology/.venv/bin/python # when the sweep lands, this is the whole thing $P packages/sweep-v1/evaluate_sweep.py all \ --records packages/sweep-v1/retrieved/records.jsonl \ --out packages/sweep-v1/eval # add the phikon probe re-run (slower; snapshots and restores C2's probes/ outputs) $P packages/sweep-v1/evaluate_sweep.py all --records ... --out ... --probes # any stage on its own; `report` reads the stage-*.json the others wrote $P packages/sweep-v1/evaluate_sweep.py features --records ... --out ... $P packages/sweep-v1/evaluate_sweep.py a1 --records ... --out ... $P packages/sweep-v1/evaluate_sweep.py d1 --records ... --out ... $P packages/sweep-v1/evaluate_sweep.py arms --records ... --out ... # a2 a3 d2 e2 b1 together $P packages/sweep-v1/evaluate_sweep.py e2 --records ... --out ... # or one at a time $P packages/sweep-v1/evaluate_sweep.py screens --records ... --out ... $P packages/sweep-v1/evaluate_sweep.py sheets --records ... --out ... $P packages/sweep-v1/evaluate_sweep.py select --records ... --out ... $P packages/sweep-v1/evaluate_sweep.py report --records ... --out ... # partial sweep arriving in stages: features is incremental, so just re-run `all` $P packages/sweep-v1/evaluate_sweep.py all --records ... --out ... --skip screens ``` Useful flags: `--jobs N` (default cpu_count-2), `--limit N` (first N records, development), `--root DIR` (if `output` paths are relative to something other than the records file's directory), `--ncc-refs {targeted,all,none}` and `--ncc-stride` (section 5), `--ncc-top-refs`, `--reuse-screen`, `--keep-screen-dirs`, `--top-k` (default 8), `--sheet-rows`, `--reference-train`. ### Inputs it expects | input | where | what happens if absent | |---|---|---| | merged sweep records | `--records` | hard error naming the path | | per-slot output trees | `/slot{0,1,2}/out/...` from each record's `output` | listed as `PNG not found`, run continues | | real fields | `.artifacts/pixcell-fields-v1/wide/{train,heldout}//*mpp0.25.png` | reference rows and the screens degrade with a logged note | | cartoons + label maps | `cartoons/{a1,d1,e2,b1}/_{cartoon,labels}.png` | d1 IoU cells blank, sheets lose the cartoon column, logged | | C1 envelope | `morphometrics/envelope.json` | hard error (the whole envelope half depends on it) | | fullset baseline | `pixcell-fullset-20260908/retrieved/out/` | the `fullset_pass2_baseline` row is omitted | | native slot-3 canvases | `pixcell-slot3-native025-20260906/retrieved/out/` | D1 positive control reported as no data | ### Outputs `eval/features.csv` (one row per record and per reference image: every `cm_*`, C1 and `det_*` feature plus every `extra` field plus the envelope distance and percentile), `eval/scores.csv`, `eval/a1-table.csv`, `eval/d1-measurements.csv`, `eval/d1-pairwise-ssim.csv`, `eval/arms-pairs.csv`, `eval/screen-embed.csv`, `eval/screen-ncc.csv`, `eval/copy-margin.csv`, `eval/select-scores.csv`, `eval/candidates.csv` (+ per-category files), `eval/sheets/*.png`, `eval/stage-*.json` and `eval/EVAL_REPORT.md`. The cache lives at `eval/cache/features-cache.csv`, keyed by the output's sha256, so a rerun after a partial retrieval recomputes only the new images (measured: the second `features` run over 265 records finished in 0.4 s). --- ## 3. What each stage answers 1. **`features`** - for every record's PNG: `cartoon_metrics.measure()` (ring topology, tissue / lumen / nuclear / stroma composition, prefixed `cm_`), C1's 54 morphometric features, and C2's no-encoder detail statistics averaged over the field's 1024 px tiles (`det_lap_abs_mean`, `det_hf_energy_ratio`, `det_sat_mean`, `det_gray_entropy`, `det_white_fraction`). Joined with `extra` and scored against `envelope.json` (Mahalanobis, robust-z RMS, percentile against the real held-out fields of that category, inside-band flag, ring gap, three worst features). The 22 real held-out fields and the 20 fullset pass-2 canvases are extracted the same way so the a1 table's reference rows are like-for-like. Images that analyse to a different effective um/px (anything not 4096x2048, e.g. the b1 native 1024 windows) are flagged and excluded from every envelope-based table. 2. **`a1`** - median per (category x cut x arm) over `pass1`, `pass2-si6`, `pass2-si9`, `pass2-si3`, `pass3-si15`, with the real held-out row and the fullset pass-2 baseline row. Then: for each category, the pass-2 arms whose median ring fraction is inside the real held-out **range**, and of those the lowest median envelope distance - reported with which rule fired. Then pass 3 against its own pass-2 parent (from `extra.parent`, falling back to the same cartoon): median change in Laplacian / HF energy / saturation and in ring fraction, the latter compared against the spread of the pass-2 cells so "moved" means something. 3. **`d1`** - per output: lumen-mask IoU against its own cartoon's `label == 1`; for `rot180` arms the same against the unrotated map; epithelium- and lumen-mask normalised cross-correlation (max over shifts and at zero shift) and grayscale SSIM against the donor field; and pairwise SSIM among all d1 outputs, split into same-cartoon/different-donor, same-donor/different-cartoon and unrelated. Two controls make the numbers readable: the **ceiling** (each cartoon PNG measured against its own label map - 0.407 lumen IoU on the d1 set, so a generated output can never reach 1.0 on this metric) and the **positive control** (native slot-3 canvases conditioned on the same donors, split into `native_generate` and `native_img2img`). The verdict is five independent yes/no checks, each printed with its control, not one number. 4. **`arms`** (also `a2`, `a3`, `d2`, `e2`, `b1` individually) - each record paired with the a1 cell on the same base cartoon and the same pass (`_f` and `_rot180` suffixes are stripped); where a1 never rendered that cartoon (b1 uses seeds 21-25) the delta is against the a1 median for the same category, cut and pass, and the row says so (`pair_kind=group`). Deltas: ring fraction, envelope percentile, envelope distance, the three detail statistics, nuclear density. For a2, the per-window library provenance from `token_library_picks` (distinct donor fields per canvas) and the own-source figure. With `--probes`, `run_probes.py all --scales 1024` re-runs with **one `--add-set` per experiment**, so probes A, B and D report per experiment rather than lumping the sweep together. 5. **`screens`** - `pixcell_embed_screen.py` (phikon-v2, fail line 0.95) over every pass-2 and pass-3 output against all 118 real fields (96 train + 22 held out), then `pixcell_patch_search.py` (window 64, threshold 0.92). Per arm: max and median max cosine, own-source tile fraction, any-conditioning-field tile fraction, flagged tiles, minimum copy margin, max NCC, flagged NCC windows. The own-source columns are computed from the per-tile nearest-reference records, not from the screen's own filename heuristic, so library and blend arms are handled correctly. 6. **`sheets`** - labelled contact sheets, each cell the same 768 px crop at canvas position (1664, 640, 2432, 1408), sheets at most 2000 px wide: a1 per (category, cut) as cartoon | pass1 | pass2-si6 | pass2-si9 | pass2-si3 | pass3-si15 | real held-out; d1 as a cartoon x donor grid with the rot180 arm in the last column; e2 interleaving each fidelity cartoon with the canvas it produced; d2 alpha ladder; a3 arms; a2 vs a1 pairs; b1. 7. **`select`** - writes a `scores.csv`-shaped file and a copy-margin join, then calls `morphometrics/pareto_select.py` per category with `--n 8` on (|ring - real held-out median|, envelope distance, copy margin = 0.95 - max cosine), and builds a candidates sheet. 8. **`report`** - `eval/EVAL_REPORT.md`: every table, a plain-language verdict per experiment, and nine caveats carried over from C1, C2 and B1. Every stage that finds nothing prints `no data: ` and continues. Verified: on the faithful stand-in, d1 / a2 / a3 / d2 / e2 / b1 and the pass-3 comparison all report no data and the run still completes and writes a report. --- ## 4. Runtime Measured on this Mac (14 cores, `--jobs 12`), on the stand-ins: | stage | measured | projected for the 405-job sweep | |---|---|---| | `features` | 0.28 s/image wall over 148 distinct images (3.2 s CPU each) | 447 images (405 + 42 reference) ~ **2 min** | | `a1` | < 1 s | < 1 s | | `d1` | 50 outputs + 12 controls + 10 ceilings in 9.6 s | 50 d1 outputs + controls ~ **20 s** | | `arms` (no probes) | < 1 s | < 1 s | | `arms --probes` | 12 s warm, ~4 min after new fields are embedded | 405 new fields to embed ~ **8-12 min** first time, ~30 s after | | `screens` embed | 1104 tiles 53 s, 2104 tiles 99 s | 2944 tiles (250 candidates + 118 refs) ~ **2.5 min** | | `screens` NCC | 1.5-1.7 s per (candidate, reference) pair at stride 256 | ~425 pairs ~ **11 min** | | `sheets` | 10 sheets in 27 s | ~18 sheets ~ **50 s** | | `select` | 3 s | ~5 s | | `report` | < 1 s | < 1 s | | **total** | 9 min 39 s end to end on the rich stand-in | **~17 min**, or **~27 min** with `--probes` on a cold embedding cache | Rerunning after the first pass is far cheaper: the feature cache is keyed on the output sha256, and `--reuse-screen` skips the phikon pass. The 405 outputs are all distinct, so unlike the stand-in there is no dedup saving on the first run. The dominant cost is the pixel NCC search, and it is a deliberate trade (section 5). --- ## 5. The one methodological decision worth arguing about `pixcell_patch_search.py` costs about 0.10 s per 64 px window per (candidate, reference) pair. The sweep has **250 pass-2/pass-3 outputs**; against all 118 reference fields at stride 256 that is 29,500 pairs, about **13 hours** on 12 cores (and 4x that at the script's default stride of 128). Batching the FFT across windows was measured and only buys 2.4x, which is not enough, and it is already lost to running pairs in parallel. The default `--ncc-refs targeted` therefore searches each output against: - its **own donor field** (the only reference the generator was actually shown), and - the **reference fields that are the nearest neighbour of any of its 1024 px tiles** in the phikon screen, up to `--ncc-top-refs` (default 2). The prefilter is sound on C2's own calibration: a pixel-level copy scores 0.987 embedding cosine against 0.66 for unrelated real fields, so a copy cannot hide from the embedding screen and then be missed by a targeted NCC search. `--ncc-refs all` restores the exhaustive search; the harness prints the estimated runtime before starting either way. `--ncc-refs none` skips it. This is recorded as caveat 5 in the generated report, not buried here. --- ## 6. What the dry runs actually did ### 6.1 The stand-ins `make_standin.py` builds both from `operations/research/pixcell-fullset-20260908/retrieved/out/records.jsonl` (40 records, 30 PNGs present; the 10 pass-1 normal and IM outputs were never retrieved and are deliberately **kept** in the stand-in so the missing-file path is exercised). - **`standin/`** - the faithful one the brief asks for: `pass` / `cartoon` / `category` mapped to the Phase-2 schema, `exp="a1"`, `arm="baseline"`, real donors from `donors.json`, outputs laid out as `slot{0,1,2}/out/a1/pass{1,2}//.png` (symlinks). Every experiment other than a1 has no data, which is what tests the no-data paths. - **`standin-rich/`** - 265 records over the same 30 PNGs, re-labelled so a1 (pass1 / si6 / si3 / si9 / pass3-si15), d1 (5 cartoons x 4 donors + 5 rot180), a2 (with a fabricated 21-window `token_library_picks`), a3, d2, e2 and b1 all have records. **The pixels are unrelated to the provenance stamped on them.** It is a plumbing fixture; every table it produces is meaningless as science and is labelled as such by `make_standin.py` on stdout. ### 6.2 Results of the faithful dry run (`eval-standin/`) These numbers are real - they are a re-measurement of the existing two-pass fullset run through the new harness, and they reproduce what is already on record: | measurement | harness | already on record | |---|---|---| | real held-out envelope distance, 22 fields | max abs delta 4.9e-06 vs C1 | `morphometrics/scores.csv` | | real held-out ring fraction medians | 0.884 / 0.807 / 0.774 / 0.774 | C1_RESULT.md section 3 | | pass-2 envelope distance (median per category) | normal 8.64, gastritis 6.32, IM 5.28, mixed 6.74 | C1 `fullset_pass2` D50 6.50 pooled | | phikon copy screen, 20 pass-2 canvases | max cosine 0.8859, 0 tiles at or above 0.95 | `screen-pass2-phikon-fixed.json`, identical | | own-source tiles, pass-2 | 0.938 | C2 measured 0.894 nearest their own donor field | | pixel NCC, 30 targeted pairs | max 0.7735, 0 flagged windows, 0 copy verdicts | fullset `patch-search/*.json`, 8 own-donor pairs, max 0.52-0.76, 0 flagged | ### 6.3 Results of the rich dry run (`eval-standin-rich/`) Every stage ran; the artifacts are the shape the real sweep will produce. | output | rows / files | |---|---| | `features.csv` | 307 (265 records + 42 reference images), 0 extraction failures | | `a1-table.csv` | 23 | | `d1-measurements.csv` / `d1-pairwise-ssim.csv` | 72 / 380 | | `arms-pairs.csv` | 140 | | `screen-embed.csv` / `screen-ncc.csv` / `copy-margin.csv` | 160 / 265 / 160 | | `candidates.csv` | 32 (8 per category) | | `sheets/` | 11 PNGs, each <= 2000 px wide | | `EVAL_REPORT.md` | 20 KB, 7 sections, every table populated | | `probes/` | 21 sweep-augmented C2 probe outputs, per-experiment sets `sweep_a1 ... sweep_e2` | Two numbers from it are worth keeping even though the provenance is fabricated, because they are properties of the **measurements**, not of the fixture: - the d1 lumen-IoU **ceiling** is 0.407: a cartoon PNG measured against its own label map by the image-only lumen detector. Any generated output's IoU has to be read against that, not against 1.0. - the donor cross-correlation has usable dynamic range: `native_img2img` (a light repaint of the donor field itself) 0.642, `native_generate` (built from the donor's tokens) 0.409, the cartoon-conditioned pass-2 canvases 0.314. ### 6.4 Broken-input fixture (`eval-broken/`) A deliberately hostile records file: a truncated PNG, a zero-byte PNG, a record pointing at a file that does not exist, a record with an experiment nobody has heard of, a record with no category, and a line of plain text instead of JSON. Result: `all` completed, the non-JSON line was warned about and skipped, the missing file was listed, both unreadable PNGs were recorded in `stage-features.json` as `error: OSError: image file is truncated` and `error: UnidentifiedImageError: ...` and excluded from every table, the unknown experiment passed through as its own arm, and the report was written. Nothing crashed. ### 6.5 Smoke test on the first two real sweep PNGs (`eval-early/`) While this package was being written the fleet pulled back the first two Phase-2 outputs into `packages/sweep-v1/early/` (the `intestinal_metaplasia_s11_across` pass-1 and pass-2 cells from slot a). Running `features` and `a1` on them with the real `extra` block copied out of `jobs/a1-pass{1,2}.json` worked with no change to the harness, which is the first evidence that it handles actual sweep provenance and not just the fixture: ```bash $P packages/sweep-v1/evaluate_sweep.py features \ --records packages/sweep-v1/eval-early/records-early.jsonl \ --root packages/sweep-v1/early \ --out packages/sweep-v1/eval-early ``` (the records file lives under `eval-early/` so nothing is written into `early/`, which the retrieval side owns; `--root` points the harness at the PNGs) | cell | ring | ring/mm2 | nuc/mm2 | str nuc/mm2 | env D | env pct | laplacian | hf energy | saturation | |---|---|---|---|---|---|---|---|---|---| | a1 pass1 | 0.545 | 252 | 6454 | 1313 | 7.09 | 100 | 12.12 | 0.0133 | 0.295 | | a1 pass2-si6 | 0.202 | 227 | 5400 | 1099 | 5.59 | 100 | 14.14 | 0.0167 | 0.355 | | real held-out IM (6 fields) | 0.128-0.953 | 74-162 | 5517-6895 | 996-6879 | 1.50-4.50 | - | 17.1-27.1 | 0.031-0.071 | 0.264-0.346 | | old fullset IM pass-2 (5 fields) | 0.832-0.980 | 94-250 | 5158-6190 | 1362-6059 | 3.64-7.19 | - | 12.5-17.5 | 0.015-0.085 | 0.175-0.329 | **This is n = 1 and means nothing yet.** It is reported only to show the pipeline runs. The one thing worth flagging for whoever reads the real run: pass 2 moved the envelope distance the expected way (7.09 -> 5.59) while dropping the ring fraction to 0.202, which is at the bottom of the real held-out IM range and far below the old cartoon set's pass-2 values. If that holds across the 15 IM cells, the a1 stage's "topology inside the real range" gate will be doing real work. `packages/sweep-v1/retrieved/merge_records.py` (written by the retrieval side while this package ran) was read for compatibility: it rewrites each record's `output` to `slot/out/` relative to `packages/sweep-v1/retrieved/`, keeps `extra` untouched and adds `output_vm`, `slot` and `fleet_slot`. That is exactly what this harness's default `--root` and provenance reader expect, so no adapter is needed. ### 6.6 The probe re-run does not damage package C2 `run_probes.py` writes its 21 JSON and PNG deliverables straight into `probes/`, which are C2's outputs. The harness snapshots them, runs the probes, copies the sweep-augmented results into `eval/probes/`, and restores the originals. Verified by md5 before and after: `probes/summary.json`, `probes/probeB-tell.json` and `probes/probeD-manifold-1024.json` are byte-identical after two `--probes` runs. The `probes/embeddings/*.npz` cache does grow (additively), which is the intended behaviour and is what makes the second run 12 s instead of 4 minutes. --- ## 7. Bugs found and fixed while dry-running 1. **The phikon screen resolved symlinks.** `pixcell_embed_screen.py` calls `Path.resolve()` on every candidate, so a staging directory of symlinks reported the *target's* basename; every candidate row failed to join back to its record, and two arms sharing a cartoon stem would have collided silently. Fixed by staging with hard links (which keep their own name and cost no disk), with a copy fallback and a warning if any screened name still fails to join. 2. **`pass` arrives as a float from the CSV round trip**, so `p{pass}` in the screen filename became `p2.0`. Fixed. 3. **`sha256` was being coerced to a float** (and became `nan`) when features.csv was read back. 4. **b1 has no a1 partner by construction** (a1 renders seeds 11-15, b1 uses 21-25), so every b1 delta was blank. Added a two-level group fallback: same category+cut+pass, then same category+pass, with `pair_kind` recorded per row. 5. **The pass-3 verdict said "detail down" for a zero change.** Now says "unchanged" below 0.05. 6. **The d1 IoU had no ceiling**, so 0.046 could not be told apart from "the metric cannot go high". Added the cartoon-against-its-own-label-map control. 7. **The a1 sheet mixed cuts in one grid**, making rows non-comparable. Now one sheet per (category, cut). 8. The screens stage left 118 + N hard links behind; they are now cleaned up unless `--keep-screen-dirs`. --- ## 8. Limitations 1. **Not clinical, and not a review.** No pathologist has seen any of this, and Yash has not reviewed a single sheet. The verdicts are arithmetic over classical measurements. 2. **The harness has never seen a real Phase-2 output.** Every code path has executed on real PNGs, but those PNGs came from the fullset run, not the sweep. Cartoon stems, arm names, `token_source` values and `token_library_picks` shapes were taken from `jobs/*.json` and GEN_RESULT.md; if the sweep emits something else, the affected stage will report no data rather than fail, and the mapping in `arm_label()` / `base_cartoon()` is the place to fix it. 3. **The pixel NCC screen is targeted, not exhaustive** (section 5). It is a screen, not a proof of non-copying. 4. **The envelope calibration is thin** - 4 to 6 real held-out fields per category from 2 to 3 slides, so `pct_vs_heldout` quantises to 25% steps for normal and 16.7% for the rest (C1_RESULT.md limitation 2). It is ordinal. 5. **`ring_with_lumen_fraction` does not separate the four categories on real tissue** (eta2 = 0.007, p = 0.89, C1_RESULT.md section 6). The a1 stage uses it only as a plausibility gate ("is the topology in the real range") and never as the thing being optimised. 6. **The d1 layout verdict is five heuristic checks with a majority vote.** The individual checks and their controls are printed; the single boolean is a convenience, not a test statistic. There is no significance testing anywhere in the harness - with 5 cartoons and 4 donors there is not enough data for it. 7. **`str_nuc_density_per_mm2` is noisy.** C1's "stroma" is a small residual of a 6 um-closed hematoxylin band (5-10% of tissue). It is comparable across arms because every image is measured identically, but it is not a real stroma segmentation. 8. **Detail statistics cannot certify 0.25 um/px content.** They are computed on 1024 px tiles resized to 512, and C1 showed a half-resolution round trip moves neither the ring metric nor the envelope. A pass-3 arm that raises the Laplacian has raised *something*; whether it is genuine sub-micron detail is not answered here. 9. **Probe B cannot attribute a donor to library or mean-token arms** - they have no `token_reference`, so those rows read 0.000 by construction. Their own-source figure comes from the harness's own tile-level join in the A2 block instead. 10. **B1 is not organisms.** B1_RESULT.md graded 0 of 60 candidates as likely organisms. The b1 arm is labelled throughout as "gastritis surface-compartment donors", and the report repeats the caveat. 11. **d1's `donor0` arm duplicates a1's pass-1 cell** by construction (GEN_RESULT.md deviation 4), so it is not an independent sample. 12. **Category and donor stay confounded** for a1 and d1: the cartoon's category always equals the donor's. a2, a3 and d2 are the arms that break it, and the harness reports them, but the confound is a property of the experiment design, not something the evaluation can remove. 13. **The stand-in outputs left in the tree are fixtures, not results.** `packages/sweep-v1/eval-standin*/EVAL_REPORT.md` must never be quoted as a Phase-2 finding. Delete them, or `--out` somewhere else, once the sweep lands. --- ## 9. What is not done - The sweep has not run, so `packages/sweep-v1/retrieved/records.jsonl` does not exist and the harness has produced no Phase-2 answer. - The `--probes` path re-runs C2's probes but does not re-run C1's `calibrate.py`; the envelope is taken as fitted. If the real fields ever change, refit C1 first. - No F2 (structure adapter) arm is handled. F2's outputs are a Phase-3 package with their own records; adding them is a matter of pointing `--records` at them, but no `f2` branch exists in `stage_arms`. - The reviewer pack v2 and the results page (PLAN.md Phase 4) are downstream of this harness and are not built here.