B1-ORGANISMS: looking for real H. pylori on the HiESD gastritis slides

Package root: operations/research/sgh-program-20260908/organisms/ Date: 2026-09-08. Compute: IOFlood workbench only (8 CPU, 8 GB RAM, no GPU). No GPU, no git commits.

Headline

The insight did not hold up. The premise was that the HiESD "chronic gastritis" slides are H. pylori gastritis, so the organisms are physically present at 0.25 um/px and can be found with classical image processing. After scanning 753 mucosal-surface windows across the 13 non-held-out slides and grading the top 60 candidates by eye:

• 0 of 60 candidates graded likely organisms. 5 graded ambiguous, 55 not organisms.
• The detector's top calls are collagen and muscle fibres, capillary walls and basement membrane, elongated endothelial/fibroblast/lymphocyte nuclei, surface fibrin, and stain precipitate — the hit galleries make this unmistakable.
• The negative arm scores like the positive arm. Normal-gland surface windows from the same slides reach a p90 score of 20.8 against 24.0 for gastritis surfaces; the single highest-scoring window in the first sweep was a negative. Best enrichment at any operating point is 2.7x, and it is 1.0-1.5x over most of the range.
• The decisive experiment: when the eligible region is tightened from "dim pale" (brightness

  = 200, which includes the interstitial space of loose lamina propria) to genuinely bright open luminal space (brightness >= 215 or >= 225 — actual mucus, pit and gland lumina), the signal collapses to zero on both arms (p90 score 1.5-1.8 at 215, 0.0 at 225). Pushing sensitivity back up in that region (threshold_abs down to 0.018, threshold_k down to 2.0) recovers hits but with enrichment 0.62-1.21 — the negatives score higher at two of the nine settings.

Read together: in the compartment where H. pylori lives, there is nothing detectable that distinguishes gastritis surfaces from normal surfaces on these slides. Everything the detector found was in the tissue texture next door.

This is not a pathologist's verification, and it is not a claim that the slides are H. pylori negative. It is an engineering result about what a classical detector can and cannot find on this H&E at this resolution. Yash must confirm anything used downstream.

What was used

13 non-held-out HiESD slides. Held out and never touched: 0d55cc29-…, 0dddc3e2-…, 1a203f16-… (matching scripts/extract_hiesd_pixcell_fields.py). Source MPP 0.2458 um/px on all 17 slides.

753 windows scanned, 0 read failures. Positive arm = HiESD Chronic Gastritis (0,0,255) or Chronic atrophic gastritis (70,130,180) within 120 um, alone (hpylori_gastritis, 56 of the top 60) or with intestinal metaplasia (mixed, 4 of the top 60). Negative arm = Normal Gland (138,43,226) within 120 um with no gastritis, metaplasia or lymphoid label nearby. Any tumour label (tub1/tub2/pap/others) keeps a 600 um exclusion zone.

How the mucosal surface was found

Two HiESD products do the geometry, so no whole slide is ever loaded:

• ESD_40X_Quality_Assessment_Mask/<uuid>.png — GrandQC segmentation at 0.9995 um/px (1 tissue, 2 folds, 3 dark spots, 4 pen, 5 bubbles/edge, 6 out of focus, 7 background), downsampled 4x to a 3.998 um/px grid before any distance transform.
• ESD_40X_annotation_downsample64/<uuid>.png — pathologist region labels at ~15.7 um/px.

The mucosal surface is the tissue boundary that faces glass and has annotated mucosa just inside it; the deep resection margin faces glass too but is submucosa, which carries no gland annotation. Window centres are sampled 25-150 um inside that edge with a 300 um minimum spacing and a 60 um keep-out around every GrandQC artefact class. A 1024 x 1024 window at 0.25 um/px is 256 um across, so each window straddles glass, surface mucus, surface epithelium and the upper pits.

Each window is one openslide.read_region of 1041 x 1041 at level 0, resampled Lanczos to 1024 x 1024 so every pixel measurement is in exact micrometres.

Detector

organisms/detector.py, version b1-organism-detector-v2. numpy + scipy + PIL only (no cv2, no skimage on the workbench). Deterministic; every number is a field of DetectorConfig and ships in the JSON beside any result. It runs identically on IOFlood and locally (verified: the library window 28f9b423_28751_46953.png scores 13.75 / 19 hits in both places).

1. Colour deconvolution — Ruifrok-Johnston H/E/residual, optical density base 10, the rgb_from_hed stain vectors. Gives separate haematoxylin and eosin planes.
2. Eligible region — large open pale components (pale_brightness 200 after a sigma-3 blur, min_pale_component_px 1500) that are 1-40 px from tissue (tissue_brightness 200), with a 12 px border margin. The large-component rule is load-bearing: without it, detector v1 scored oedematous lamina propria exactly like surface mucus.
3. Band-pass — difference of Gaussians on haematoxylin, sigma 1.1 and 4.0, sized for objects 2-4 um long and 0.5-1 um wide (8-16 px x 2-4 px at 0.25 um/px).
4. Threshold — max(0.045, median + 3.2 * 1.4826 * MAD) of the band-pass response inside the eligible region; 2x2 opening; 8-connected components.
5. Filters (rejection counts over all 753 windows, ~322k components examined): | filter | rule | rejected | |---|---|---| | area | 7-110 px | 153784 | | shape | length 6-22 px, mean width 1.0-4.6 px, width extent <= 6.5 px, aspect >= 2.5 | 128781 | | faintness | mean H OD 0.070-0.560, peak <= 0.780 (excludes nuclear fragments) | 6 | | context | H contrast >= 0.045 vs a 4 px annulus, surround brightness >= 186 | 0 | | stain | H contrast minus E contrast >= 0.015 — the contrast must be basophilic, so collagen, muscle, fibrin and basement membrane are rejected | 214 | | continuity | band-pass response must fall away beyond both ends along the object's own axis (probe 4-10 px, < 0.55 x threshold) — a fibre fragment continues, a free rod does not | 20314 | | distance | centroid <= 40 px from tissue | 0 |
6. Curvature — quadratic fit v = a u^2 + b u + c in the component's principal frame, curvature = |2a|; scored, not required (comma shapes preferred, straight rods kept).
7. Clustering — single-link BFS on the 30 px radius graph, minimum cluster 3.
8. Window score = 2 * max_cluster + n_hits_clustered + 0.25 * n_hits. Also recorded: hit_density_per_kpx, eligible_px, mucus_band_px, interface_px (pale space hugging nucleated epithelium — pit lumen and surface mucus, as opposed to interstitial space), tissue_fraction, nucleus_fraction, focus (99.9th percentile band-pass over tissue).

18683 hits survived across 753 windows; 890 CPU-seconds of detector time.

False-positive calibration on normal surfaces

Full table in organisms/logs/calibration.json. The negative arm is hard: it is normal-gland mucosal surface from the same 13 patients. If those stomachs are H. pylori positive, organisms can sit on histologically near-normal mucosa too, so the rates below are an upper bound on specificity loss — but that cuts both ways, and it does not explain a 1.0-1.5x enrichment.

Windows gated to nucleus_fraction >= 0.004 and interface_px >= 12000 (the gate used to pick the top 60): 230 positive, 62 negative.

Ungated (613 positive, 140 negative) the best point is score >= 30: positive rate 0.057, false-positive rate 0.021, enrichment 2.66. "At least one cluster of 3" fires in 24.1% of positives and 20.7% of negatives; "max cluster >= 5" fires in 1.8% of positives and 2.1% of negatives.

A real organism detector should be near-absent on normal surfaces. This one is not.

The eligible-region sweep (organisms/logs/sweep1-eligible-region.json)

Nine settings over the cached windows, ranked by the score distribution of each arm:

Every hit the detector produces lives in the 200-215 brightness band — the interstitial texture of loose stroma, not open mucus. Actual lumen and mucus are empty.

The sensitivity sweep in true luminal space (organisms/logs/sweep2-sensitivity.log)

Fixed at pale 215 / min component 4000, nine (threshold_abs, threshold_k) settings:

Cranking sensitivity in the luminal compartment finds noise, symmetrically on both arms.

Candidate grading

organisms/grades.json (per-candidate grade and substrate), organisms/candidates/index.json (detector numbers that produced the ranking), organisms/candidates/sheet-01.png .. sheet-05.png (12 per sheet, as briefed) and organisms/candidates/review/review-01.png .. review-15.png (2x2 at ~3x, which is what actually made grading possible — the 12-up sheets are too small for 2-4 um objects).

How reviewed. All 60 candidates viewed as 256 x 256 crops at 4x nearest-neighbour zoom with 2 um and 10 um scale bars. 54 on the 2x2 review sheets; 9 individually at full 1024 x 1024. Detector calls audited on the matching _zoomdet.png overlays and on three hit galleries (organisms/gallery/pos-cluster.png, neg-cluster.png, pos-curved.png — 64 x 64 boxes at 6x).

Substrate breakdown of the 55: fibre 17 (collagen, muscularis mucosae, basement membrane, capillary wall — eosinophilic and continuous), stroma 16 (loose/oedematous lamina propria; the detector fires on the fine strands crossing interstitial space), nuclei 12 (elongated or tangentially cut endothelial, fibroblast, lymphocyte and plasma-cell nuclei — too long, too dark, visible chromatin), fibrin 6 (surface fibrin and mucin strands), epithelium_no_organisms 4 (genuine foveolar/glandular epithelium with a lumen in which no rods are visible).

The negative-arm gallery is indistinguishable from the positive-arm gallery: the same thin pink-purple strands with a bluish edge, the same elongated nuclei, plus obvious muscle bundles and capillary cross-sections.

Again: this is a model's structured visual review, not a pathologist's verification. Nothing here settles the H. pylori status of any HiESD case. Yash must confirm.

Library

organisms/library/ — 12 windows, 1024 x 1024 RGB PNG at 0.25 um/px, from 8 distinct slides, with organisms/library/index.json (slide id, level-0 coordinates, raw read size, HiESD category, every detector score, grade, grade reason, anatomy note, sha256).

Because no candidate graded likely organisms, this is not an organism library and index.json says so at the top of the file. The brief asked for at least 8 windows even at lower confidence, marked; these are the 12 whose anatomy is the compartment H. pylori would occupy — the 5 ambiguous ones first, then 7 chosen for a real surface-mucus / pit-lumen / gland-lumen interface. Every entry carries organism_verified: false.

organisms/negatives/ — the comparison arm: 8 normal-gland surface windows from 5 slides, same format, with index.json and the same caveat that they are hard negatives from the same patients.

How this plugs into the generation test (GEN-CODE)

GEN-CODE expects organism-library/*.png as 1024 x 1024 RGB at 0.25 um/px and builds UNI2-h tokens from them (sixteen 256 crops resized to 224 per window). organisms/library/*.png meets that contract byte-for-byte; index.json is metadata GEN-CODE can ignore. To wire it up:

ln -s operations/research/sgh-program-20260908/organisms/library organism-library
# on the GPU VM, under the package root:
#   [local]/sgh-program-20260908<vm-slot>/organism-library/

For the B1 row of the Phase-2 matrix (gastritis 5 cartoons with organism donors, plus 10 native 1024 windows from organism donors), the 12 library windows give a donor rotation of 12 with per-window provenance for extra: {exp: "b1", pass: k, cartoon: ..., donor: "<file>", arm: ...}.

But the test it enables is weaker than the one the brief asked for. The intended experiment was "does PixCell render organisms when conditioned on windows that contain them?" That question cannot be asked, because no window was verified to contain them. What these 12 windows can test:

1. Whether conditioning on real gastritis surface tokens (mucus band, pit lumens, foveolar epithelium against glass) makes PixCell render surface architecture that the current donor rotation — drawn from 2048 x 1024 wide fields, mostly mid-mucosa — does not.
2. Whether the procedural organism stage (scripts/pixcell_hp_stage.py, rods 10-18 px, faint grey-blue, clustered in large lumina) survives a light img2img pass conditioned on these donors, i.e. whether PixCell preserves or erases a 2-4 um painted object. That is a genuine and answerable question, and this library is the right donor set for it.
3. As a negative control, organisms/negatives/*.png gives normal surfaces for the same test.

The honest framing for any downstream write-up: the organisms in SGH outputs remain procedural. Nothing in this package produced a verified real-organism reference.

Limitations

1. The premise is unverified. HiESD annotates "Chronic Gastritis" and "Chronic atrophic gastritis". It does not annotate H. pylori status. The work-package brief treated the gastritis slides as H. pylori gastritis; that is an assumption, not a dataset fact. HiESD is an ESD (endoscopic submucosal dissection) cohort for gastric neoplasia, where eradication therapy before or after resection is routine — post-eradication and atrophic/metaplastic mucosa both carry low or zero organism density. A null result here is entirely consistent with the slides simply not having many organisms.
2. H&E is the wrong stain. H. pylori on H&E is faint and unreliable even under a microscope; Giemsa, Warthin-Starry or immunohistochemistry is the standard. No permitted dataset here has those.
3. Effective optical resolution. Nominal 0.2458 um/px at 40x, and the best fields do resolve nuclear chromatin and individual red cells, so a 3 x 0.6 um rod is in principle resolvable. But sharpness varies a lot across and within slides, and the SVS JPEG compression softens fine detail. My focus proxy (99.9th percentile band-pass over tissue) did not separate the arms and did not reliably flag the visibly out-of-focus regions on 4fdce538 — it is recorded but should not be trusted as a quality gate.
4. The negatives are hard negatives from the same patients, as discussed above.
5. Surface targeting is imperfect. GrandQC gives the tissue/glass boundary, and the annotation mask distinguishes the mucosal edge from the deep margin, but many sampled windows still land in lamina propria or muscularis rather than on foveolar surface with mucus. A stronger targeting signal — an explicit surface-epithelium detector (palisaded columnar cells with basal nuclei and apical mucin against luminal space) — would raise the yield of anatomically correct windows and is the obvious next improvement if this line is continued.
6. Grading is by a model, not a pathologist. Stated once more because it matters: ambiguous here means "I cannot tell", not "probably organisms", and not organisms is my reading of the substrate, not a diagnosis.
7. Detector v1 is preserved only in this document. The first version (no stain-selectivity, no continuity filter, permissive eligible region) scored a normal window highest of all 753 and is the reason those two filters exist. The v1 scan file was overwritten by later passes.

What is where

operations/research/sgh-program-20260908/
  B1_RESULT.md                       this file
  code/organism_detector.py          programme-level re-export of the detector (PLAN's code/ convention)
  organisms/
    detector.py                      the detector; runs on IOFlood and locally, CLI + importable
    survey_surface.py                enumerates mucosal-surface windows from the HiESD masks
    scan_windows.py                  level-0 reads + detector, multiprocess
    export_candidates.py             ranks, re-reads winners, writes windows / zoom cards / sheets
    hit_gallery.py                   montage of individual hits at 6x, for auditing what fired
    sweep_config.py                  detector-setting sweeps over cached windows
    run_on_ioflood.sh                push / survey / scan / export / galleries / pull
    grades.json                      per-candidate grade and substrate, with the caveats
    candidates/                      60 windows, 60 zoom cards, 60 marked cards, sheet-01..05,
                                     review/review-01..15 (2x2 at ~3x), index.json
    library/                         12 windows + index.json  (NOT organism-verified)
    negatives/                       8 normal-surface windows + index.json
    gallery/                         pos-cluster.png, neg-cluster.png, pos-curved.png
    logs/                            windows.jsonl, windows-slides.json, scan.jsonl,
                                     calibration.json, sweep1-eligible-region.json,
                                     sweep2-sensitivity.log

On the workbench everything lives at [local]/b1-organisms (symlinked from [local]/b1-organisms), including out/windows-png/ — all 753 scanned windows cached as PNG, 1.3 GB, which is what makes a detector sweep a 2-minute job instead of a re-read of the slides.

Commands

cd research/sgh-synthetic-histopathology
B1=../../operations/research/sgh-program-20260908/organisms

bash $B1/run_on_ioflood.sh push        # ship the six python files to the workbench
bash $B1/run_on_ioflood.sh survey      # 13 slides -> out/windows.jsonl   (~2 min)
bash $B1/run_on_ioflood.sh scan        # 753 windows -> out/scan.jsonl    (~2.5 min, 7 workers)
bash $B1/run_on_ioflood.sh export      # top 60 candidates + 8 negatives + sheets
bash $B1/run_on_ioflood.sh galleries   # hit galleries for auditing
bash $B1/run_on_ioflood.sh pull        # fetch candidates/, negatives/, gallery/, logs

# detector-setting sweep over the cached windows (about 2 min per setting, 7 workers)
./ioflood exec 'cd [local]/b1-organisms && \
  [local]/python code/sweep_config.py \
    --windows out/scan.jsonl --png-dir out/windows-png --out out/sweep.json --grid out/grid.json'

# run the detector on any 1024x1024 window, anywhere
.venv/bin/python operations/research/sgh-program-20260908/organisms/detector.py \
  operations/research/sgh-program-20260908/organisms/library/*.png --json-out /tmp/lib.json --full

Gotcha: every ./ioflood exec opens a fresh SSH connection, and a burst of them gets reset by the host (kex_exchange_identification: read: Connection reset by peer). run_on_ioflood.sh now paces and retries; if you drive ioflood directly, pause between calls. All six shipped files were verified md5-identical between the repo and .artifacts/b1-organisms/code/ after the final push.

Recommendation

Do not present any of this as real organism evidence. Either

• keep the organisms procedural and say so plainly (the current pixcell_hp_stage.py path), using this library as the surface-anatomy donor set and running test (2) above — does PixCell preserve a painted 2-4 um object through img2img — which is a real, answerable question; or
• if verified organisms are genuinely needed, they have to come from a dataset with a special stain or an H. pylori annotation. Chasing them further on HiESD H&E with classical image processing is, on this evidence, not going to work.

Download public Markdown export