C2-PROBES: pathology-encoder probes on the generated fields

Package C2 of the SGH programme (operations/research/sgh-program-20260908/PLAN.md). Everything here is engineering evidence from one pathology encoder. None of it is a clinical judgement, and no number here says an image is diagnostically correct. Run locally on the Mac (MPS), 8 September 2026.

Deliverables: probes/run_probes.py, probes/*.json, probes/*.png, embeddings cached under probes/embeddings/. Nothing was committed to git.

What was measured

phikon-v2 (owkin/phikon-v2, DINOv2 ViT-L/16, 224 px input, pathology-pretrained, non-commercial licence, internal evaluation only) embeds every 1024 px tile of every 4096x2048 field (8 tiles per field), CLS token, L2-normalised — the same code path as scripts/pixcell_embed_screen.py. A second scale, 512 px tiles (32 per field, also resized to 224, so twice the effective magnification), was embedded and every probe was run again on it.

292 fields, 2336 tiles at 1024 px and 9344 at 512 px, in 13 sets:

Provenance (category, pass, start_index, arm, cartoon, donor, seed) is read from each run's records.jsonl, never inferred from a filename. Full per-tile manifest: probes/manifest-fields.json.

Question 1. Can a pathology encoder tell our generated images from real ones?

Yes, perfectly, with no overlap at all.

A logistic regression on phikon embeddings, trained on real train tiles versus two-pass tiles from cartoons 0-2, tested on real held-out tiles versus two-pass tiles from cartoons 3-4 (split by cartoon, so no tile of a test image was ever seen):

Chance is 0.5. The separation is not marginal, it is a gap: across all 944 real tiles the highest score is 0.087, and across all 160 two-pass tiles the lowest is 0.606. Nothing lies in between. On the log-odds scale real tiles sit near -9 and two-pass tiles near +9 (probes/probeA-score-histograms-1024.png).

Three controls say the result is real and not an artefact of the setup:

• Real versus real. The same probe trained to separate real train slides from real held-out slides (slide-disjoint 5-fold) gets AUC 0.401 at 1024 px and 0.528 at 512 px, i.e. chance. So the probe is not reading a slide or batch effect; the held-out real slides look like the train slides to it.
• Permuted labels. Ten label permutations give mean AUC 0.498 (sd 0.145). The 1024-dimensional embedding with 864 training tiles is always linearly separable, so this control matters; it comes out at chance.
• Reverse direction. A probe trained only on real train versus native canvases — it has never seen a two-pass image — still separates real held-out from two-pass at AUC 1.000, accuracy 0.991.

The tell is shared across every generator variant we have. A transfer matrix (train against set X, test against set Y, probes/probeA-transfer-matrix-1024.png) has no entry below 0.754; a probe trained only against cartoons detects two-pass images at AUC 0.948.

One arm needs care. Applying the two-pass probe to lock_lora (the LoRA-finetuned arm) gives a mean score of 0.437 — apparently undecided. That is not evidence the LoRA images are more real. A probe trained against lock_lora separates it from real at AUC 1.000, and probe D puts those same images at the 98.9th percentile of the real distance distribution. The LoRA arm is off the real manifold in a different direction from the one the two-pass probe learned. Reading a low probe score as realism would be a mistake.

What the tell is (probe B)

probes/tell-sheet.png (four rows of 512 px thumbnails: the 10 most generated-looking two-pass tiles, their nearest real train tiles, the 10 least generated-looking, their nearest real train tiles, each labelled with its score) and probes/tell-zoom.png (the top four pairs at 1:1 pixels).

Looking at them, the generated tiles differ from their real nearest neighbours in texture, not architecture:

• Nuclei are stamped. Generated nuclei are smooth, evenly saturated blue-purple ovals with hard edges and essentially no internal chromatin texture. Real nuclei have speckled chromatin, lighter centres, occasional nucleoli, and softer, more irregular outlines.
• Cytoplasm is a flat wash. Real eosinophilic cytoplasm is granular and varies within a single cell; generated cytoplasm is a near-uniform pink field.
• Cell borders are too regular. In the generated tiles the pale lines between cells are continuous and of near-constant width, giving an over-tidy honeycomb; real borders vary in width and break up.
• No acquisition noise. Real tiles carry stain granularity and scanner noise everywhere; generated tiles are clean.
• Architecture, by contrast, matches well. The nearest real neighbours have visibly similar gland and lumen layout — which is the known structure-inheritance effect, quantified below.

The same tell is measurable without any encoder. A logistic regression on twelve hand-made colour and texture statistics, on the identical train/test split, reaches AUC 0.932 at 1024 px and 0.914 at 512 px (probes/probeA-pixel-statistics-1024.json). The single strongest feature is mean absolute Laplacian, i.e. local detail:

So a large part of the tell is that the two-pass images carry about 30% less fine detail and are paler than real tissue. It is a real deficiency, not a stain-normalisation artefact: native_gen025 (21.09) and native_i2i025 (20.43) sit at real levels (real train 21.40), so the detail loss belongs to the cartoon-conditioned two-pass path, not to PixCell in general. Note also that phikon still beats the pixel probe (1.000 vs 0.932), so there is a second component of the tell that the twelve statistics do not capture.

Package C1's morphometrics (morphometrics/features.csv, joined on 20 two-pass fields) agree: the field-level probe score correlates negatively with nuclear density (r = -0.56), nucleus count (-0.55), epithelial nucleus count (-0.53), stromal structure energy (-0.46) and mean colour saturation (-0.39). The more generated a field looks to phikon, the fewer and smoother the nuclei it contains.

Structure inheritance, re-measured on the two-pass recipe

For 89.4% of two-pass tiles the nearest real train tile lies inside the very field whose UNI tokens conditioned it (chance is about 1/24 = 4%); 98.1% land somewhere on the donor's slide. native_gen025 93.1%, native_i2i025 95.6%, finals 88.7%. This replicates the finding in operations/research/pixcell-native025-program/REPORT.md on the new recipe: the two-pass recipe did not reduce structure inheritance. lock_lora is the exception at 0.0% — but it is also the furthest off-manifold.

Question 2. Do generated images classify as the category we labelled them?

The honest answer is: the instrument is too weak to say, and once the leakage is removed the generated images are at the real ceiling, which is itself low.

A 4-way logistic regression on real train tiles only, slide-disjoint (GroupKFold on 13 slide ids), chance 0.25:

The ceiling is the headline. phikon can barely read these four categories off 1024 px gastric fields: 0.51 against a chance of 0.25, and the errors are the clinically sensible ones — mixed scatters into intestinal_metaplasia and hpylori_gastritis (it contains both), and hpylori_gastritis leaks into normal. Any statement about generated images is bounded by that.

Applying the classifier to the generated sets first looks flattering, then does not survive its control:

Every generated field was conditioned on UNI tokens taken from one real train field whose category equals the label, and the classifier was trained on that same field. So the uncorrected 0.713 partly measures the classifier recognising the donor. Refitting per field with the donor's entire slide removed drops gen_pass2 to 0.456 tile / 0.450 field, against a matched real held-out ceiling of 0.472 / 0.500. The same classifier gives a generated field the same label as its own donor field 65% of the time.

Read plainly: two-pass images are about as category-consistent as real held-out fields, on an instrument that is only slightly better than chance at the task. That is a weak positive, not evidence of disease truth. It does not distinguish "the generator drew intestinal metaplasia" from "the generator reproduced the donor's appearance, and the donor had intestinal metaplasia".

The cartoons control behaves as it should: 0.250 accuracy, with 158 of 160 tiles predicted hpylori_gastritis — a degenerate prediction carrying no category information at all.

Two cautions on the table. Accuracy on the lock_* sets (all one category) is not a balanced measure: a classifier that always answered intestinal_metaplasia would score 1.0, which is roughly what happens at 512 px (lock_ladder 0.979). And 512 px is the worse scale for this question (real ceiling 0.412), so the 1024 px numbers are the ones to quote.

Question 3. How far are generated tiles from the real-tissue manifold?

For every tile: mean cosine distance to its k=5 nearest real train tiles of the same category. Real held-out tiles (slides absent from every bank) define the scale. probes/probeD-knn-distance-1024.png.

Calibration, real held-out at 1024 px: median 0.211, p90 0.273, p99 0.355, max 0.518.

Three things to take from it.

1. The two-pass images are inside the real manifold's spread, at its outer edge. With the full real-train bank the median two-pass tile sits at the 63rd percentile of real held-out variation and only 1.3% of tiles exceed the held-out p99. But part of that closeness is the donor: remove the donor's slide from the bank and the median moves to the 95th percentile. The right summary is that two-pass tiles are further from unrelated real tissue than real tissue is, but still within its range.
2. Pass 2 is what does the work. Pass 1 sits at the 99th percentile with 100% of its tiles beyond the held-out p99. Comparing only the two categories where pass 1 was retrieved, the same cartoons move from the 95.8th percentile (gastritis) and 100th (mixed) after pass 1 to the 62.5th and 77.1st after pass 2. On this axis the second pass is the difference between clearly off-manifold and inside the distribution — worth stating alongside the ring-topology result, where pass 2 makes gastritis worse (ring-summary.json: 0.628 -> 0.475). The two instruments disagree about pass 2 for gastritis, and both numbers should be carried forward rather than one being chosen.
3. The LoRA arm and the light-repaint lock arms are off-manifold (99th percentile), even though the two-pass probe scores the LoRA arm as undecided. Probe A and probe D disagree about lock_lora, and probe D is the one that is calibrated in real units.

Does phikon have dynamic range here? Yes. Median distances span 0.155 (native_i2i025) to 0.604 (cartoons), with real held-out at 0.211 — a factor of about 4 between "light repaint of a real field" and "renderer cartoon", and clear ordering of every arm in between. This is the opposite of the DINOv2 saturation recorded in operations/research/pixcell-native025-program/REPORT.md, where DINOv2-giant at 448 px scored unrelated real fields at 0.987 cosine and exact copies at 1.000 and was called unusable. phikon-v2 at 1024 px tiles separates real-versus-real (0.66 cosine in that report's calibration) from copies (0.987) with room in between, and the numbers above use that room. The same holds at 512 px (held-out median 0.253, cartoons 0.60).

Caveats

• phikon-v2 was trained on public pathology data. It is a well-chosen but arbitrary instrument. Probe accuracy is a statement about this encoder, not about how tissue looks to a pathologist. A different encoder could rank the arms differently.
• A probe accuracy is not a clinical judgement. "Classified as intestinal metaplasia by a linear probe on phikon embeddings" is not "shows intestinal metaplasia". The disease-truth gate is still open and still requires a pathologist; nothing here fills it.
• The category ceiling is low. phikon reaches 0.506 on real held-out tiles (chance 0.25). Every statement in Question 2 is bounded by that, so "generated matches the ceiling" is a weak claim.
• Donor conditioning contaminates the naive numbers, and both corrections point the same way: probe C 0.713 -> 0.456, probe D 63rd -> 95th percentile. Quote the corrected numbers.
• Category and donor are confounded in every set here. Cartoon category always equals donor category, so no experiment in this package can separate "the cartoon put the disease there" from "the donor did". The planned A2 (token library) and D1 (cartoon x donor) arms are what would break that confound; the script is set up to re-run on them.
• Small n. 20 two-pass fields (5 cartoons per category), 22 real held-out fields from 3 slides, 4 held-out normal fields only. The probe-A test split is 8 generated fields. Effect sizes are large enough that this does not change the direction of any conclusion, but the percentile figures carry real uncertainty.
• The lock_* sets are all intestinal metaplasia and mostly share one donor field, so their per-arm category accuracies are close to meaningless; use them only for probe A and probe D.
• gen_pass1 is only gastritis and mixed — the other two categories' pass-1 outputs were not retrieved.
• Licensing. phikon-v2 is non-commercial; this is internal evaluation only, consistent with the parked PixCell/UNI licence decision. Real fields are public HiESD (CC BY 4.0); no SGH patient data was touched.

What this changes for the programme

1. Detectability is not a marginal question and should stop being framed as one. Any claim of the form "these pass a machine screen" must name the screen: the two-pass set passes the copy screen (pixcell-fullset-20260908/screen-pass2-phikon-fixed.json: max cosine 0.886, per-image 0.79-0.89, against the 0.95 fail line) and fails a real-versus-generated screen at AUC 1.000.
2. The largest single lever on the tell is fine detail. Two-pass output carries ~30% less local Laplacian energy than real tissue while native canvases carry the same amount as real. Whatever the two-pass path does — the cartoon reference, the si6 repaint, or the 0.5 native_mpp habit — it costs detail, and recovering it would move the biggest measurable component of the tell. E2 (fidelity ladder) and F2 (structure adapter) are the arms that test this.
3. Structure inheritance is unchanged by the two-pass recipe (89% of tiles nearest their own donor field). A2 (token library) and A3 (mean tokens) remain the right experiments, and the disclosure requirement in the native025 report still stands.
4. Probe A alone must not be used to select images. lock_lora scores 0.44 on it and 99th percentile on probe D. Selection needs probe D (calibrated in real units) plus the morphometric envelope from C1.

Exact commands

Everything runs from the repository root with the SGH venv and the cached phikon weights, offline.

cd [local]/fund-raise
V=research/sgh-synthetic-histopathology/.venv/bin/python
P=operations/research/sgh-program-20260908/probes/run_probes.py

# full run: build the manifest, embed at both scales, run probes A-D, write json and png
HF_HUB_OFFLINE=1 $V $P all --scales 1024,512

# stages separately
HF_HUB_OFFLINE=1 $V $P embed  --scales 1024,512     # writes probes/embeddings/<set>__<size>.npz
HF_HUB_OFFLINE=1 $V $P probes --scales 1024,512     # reuses the cache
HF_HUB_OFFLINE=1 $V $P probes --scales 1024 --force-embed   # ignore the cache

# re-apply to a Phase-2 run by pointing at its records.jsonl (NAME=<records>[:<output root>][:<kind>])
HF_HUB_OFFLINE=1 $V $P all --scales 1024 \
  --add-set "a1_pass2=$PWD/operations/research/sgh-program-20260908/retrieved/out/records.jsonl:$PWD/operations/research/sgh-program-20260908/retrieved/out"

Runtime on the Mac (MPS, M-series, mps device): embedding all 292 fields from scratch takes 3 min 8 s at 1024 px and 8 min 30 s at 512 px; the probe stage costs a further 4 min 40 s the first time (the colour and texture statistics are computed then cached) and 28 s on every rerun with warm caches. Peak memory is small (the 1024 px embeddings are 9 MB, the whole cache 66 MB). A forced re-embed of both scales reproduced every number in this document exactly.

The added set is embedded, cached, and then included in every probe: probe A's per-set score table and transfer matrix, probe C's confusion matrices and donor-slide-excluded control, and probe D's percentiles. Provenance is refreshed from the live manifest on every probe run, so a provenance fix does not force a re-embed. Set names are read from the records, so a Phase-2 run's arms appear as their own rows.

Files

What failed or was skipped

• Pass-1 normal and IM were never retrieved from the fullset run, so gen_pass1 covers only gastritis and mixed (10 fields). The pass-1 versus pass-2 comparison in Question 3 is therefore on those two categories.
• The lock run's records name release/, but the retrieved tree has release-lockonly/. The loader resolves both; noted so the next reader does not think files are missing.
• A first --add-set smoke test polluted summary.json with a duplicate set. The stray cache was deleted and the canonical run repeated; the numbers reproduced exactly.
• Two bugs were found and fixed during the run, both of which had changed numbers: a single label permutation gave a misleading control AUC of 0.80 (ten permutations give 0.498), and a provenance cache keyed on filename alone collided between pass1/ and pass2/ files of the same stem (now keyed on set and filename). The numbers in this document are from the final run after both fixes.
• No 4096 px or whole-field embedding was attempted; phikon takes 224 px input and tiling is the established path.
• Nothing was committed to git, per the brief.

Download public Markdown export