# B1-ORGANISMS: looking for real H. pylori on the HiESD gastritis slides Package root: `operations/research/sgh-program-20260908/organisms/` Date: 2026-09-08. Compute: IOFlood workbench only (8 CPU, 8 GB RAM, no GPU). No GPU, no git commits. ## Headline **The insight did not hold up.** The premise was that the HiESD "chronic gastritis" slides are H. pylori gastritis, so the organisms are physically present at 0.25 um/px and can be found with classical image processing. After scanning 753 mucosal-surface windows across the 13 non-held-out slides and grading the top 60 candidates by eye: - **0 of 60 candidates graded `likely organisms`.** 5 graded `ambiguous`, 55 `not organisms`. - The detector's top calls are **collagen and muscle fibres, capillary walls and basement membrane, elongated endothelial/fibroblast/lymphocyte nuclei, surface fibrin, and stain precipitate** — the hit galleries make this unmistakable. - **The negative arm scores like the positive arm.** Normal-gland surface windows from the same slides reach a p90 score of 20.8 against 24.0 for gastritis surfaces; the single highest-scoring window in the first sweep was a *negative*. Best enrichment at any operating point is 2.7x, and it is 1.0-1.5x over most of the range. - **The decisive experiment**: when the eligible region is tightened from "dim pale" (brightness >= 200, which includes the interstitial space of loose lamina propria) to genuinely bright open luminal space (brightness >= 215 or >= 225 — actual mucus, pit and gland lumina), the signal **collapses to zero on both arms** (p90 score 1.5-1.8 at 215, 0.0 at 225). Pushing sensitivity back up in that region (threshold_abs down to 0.018, threshold_k down to 2.0) recovers hits but with enrichment 0.62-1.21 — the negatives score *higher* at two of the nine settings. Read together: in the compartment where H. pylori lives, there is nothing detectable that distinguishes gastritis surfaces from normal surfaces on these slides. Everything the detector found was in the tissue texture next door. **This is not a pathologist's verification, and it is not a claim that the slides are H. pylori negative.** It is an engineering result about what a classical detector can and cannot find on this H&E at this resolution. Yash must confirm anything used downstream. ## What was used 13 non-held-out HiESD slides. Held out and never touched: `0d55cc29-…`, `0dddc3e2-…`, `1a203f16-…` (matching `scripts/extract_hiesd_pixcell_fields.py`). Source MPP 0.2458 um/px on all 17 slides. | slide | level-0 dims | tissue px (4 um grid) | surface band px | positive windows | negative windows | |---|---|---|---|---|---| | `0a682165` | 73761 x 34869 | 1208610 | 432712 | 48 | 0 | | `28f9b423` | 95670 x 61502 | 7377952 | 1451717 | 48 | 0 | | `324885ef` | 103658 x 61516 | 6436006 | 1259910 | 48 | 0 | | `40f3c0cf` | 59812 x 57386 | 3694321 | 1000620 | 48 | 24 | | `4769423f` | 97660 x 62532 | 9104873 | 1525352 | 48 | 0 | | `4fdce538` | 111613 x 66634 | 11321431 | 1581161 | 48 | 10 | | `6c6bcc80` | 59809 x 46123 | 2853103 | 512822 | 37 | 24 | | `9ded7e41` | 101645 x 78920 | 11463314 | 1991509 | 48 | 2 | | `badcd77d` | 103643 x 78923 | 11272407 | 1829681 | 48 | 1 | | `d123b43f` | 105631 x 72808 | 6917854 | 1325159 | 48 | 24 | | `da8667dc` | 103632 x 73819 | 11096234 | 1863343 | 48 | 8 | | `df8d49f3` | 99638 x 63539 | 4312049 | 940799 | 48 | 23 | | `e3367780` | 113593 x 59464 | 5273178 | 1257636 | 48 | 24 | | **13 slides** | | | | **613** | **140** | 753 windows scanned, **0 read failures**. Positive arm = HiESD `Chronic Gastritis` (0,0,255) or `Chronic atrophic gastritis` (70,130,180) within 120 um, alone (`hpylori_gastritis`, 56 of the top 60) or with intestinal metaplasia (`mixed`, 4 of the top 60). Negative arm = `Normal Gland` (138,43,226) within 120 um with no gastritis, metaplasia or lymphoid label nearby. Any tumour label (tub1/tub2/pap/others) keeps a 600 um exclusion zone. ### How the mucosal surface was found Two HiESD products do the geometry, so no whole slide is ever loaded: - `ESD_40X_Quality_Assessment_Mask/.png` — GrandQC segmentation at 0.9995 um/px (1 tissue, 2 folds, 3 dark spots, 4 pen, 5 bubbles/edge, 6 out of focus, 7 background), downsampled 4x to a 3.998 um/px grid before any distance transform. - `ESD_40X_annotation_downsample64/.png` — pathologist region labels at ~15.7 um/px. The mucosal surface is the tissue boundary that faces glass **and** has annotated mucosa just inside it; the deep resection margin faces glass too but is submucosa, which carries no gland annotation. Window centres are sampled 25-150 um inside that edge with a 300 um minimum spacing and a 60 um keep-out around every GrandQC artefact class. A 1024 x 1024 window at 0.25 um/px is 256 um across, so each window straddles glass, surface mucus, surface epithelium and the upper pits. Each window is one `openslide.read_region` of 1041 x 1041 at level 0, resampled Lanczos to 1024 x 1024 so every pixel measurement is in exact micrometres. ## Detector `organisms/detector.py`, version `b1-organism-detector-v2`. numpy + scipy + PIL only (no cv2, no skimage on the workbench). Deterministic; every number is a field of `DetectorConfig` and ships in the JSON beside any result. It runs identically on IOFlood and locally (verified: the library window `28f9b423_28751_46953.png` scores 13.75 / 19 hits in both places). 1. **Colour deconvolution** — Ruifrok-Johnston H/E/residual, optical density base 10, the `rgb_from_hed` stain vectors. Gives separate haematoxylin and eosin planes. 2. **Eligible region** — large open pale components (`pale_brightness` 200 after a sigma-3 blur, `min_pale_component_px` 1500) that are 1-40 px from tissue (`tissue_brightness` 200), with a 12 px border margin. The large-component rule is load-bearing: without it, detector v1 scored oedematous lamina propria exactly like surface mucus. 3. **Band-pass** — difference of Gaussians on haematoxylin, sigma 1.1 and 4.0, sized for objects 2-4 um long and 0.5-1 um wide (8-16 px x 2-4 px at 0.25 um/px). 4. **Threshold** — `max(0.045, median + 3.2 * 1.4826 * MAD)` of the band-pass response inside the eligible region; 2x2 opening; 8-connected components. 5. **Filters** (rejection counts over all 753 windows, ~322k components examined): | filter | rule | rejected | |---|---|---| | area | 7-110 px | 153784 | | shape | length 6-22 px, mean width 1.0-4.6 px, width extent <= 6.5 px, aspect >= 2.5 | 128781 | | faintness | mean H OD 0.070-0.560, peak <= 0.780 (excludes nuclear fragments) | 6 | | context | H contrast >= 0.045 vs a 4 px annulus, surround brightness >= 186 | 0 | | stain | H contrast minus E contrast >= 0.015 — the contrast must be basophilic, so collagen, muscle, fibrin and basement membrane are rejected | 214 | | continuity | band-pass response must fall away beyond **both** ends along the object's own axis (probe 4-10 px, < 0.55 x threshold) — a fibre fragment continues, a free rod does not | 20314 | | distance | centroid <= 40 px from tissue | 0 | 6. **Curvature** — quadratic fit v = a u^2 + b u + c in the component's principal frame, curvature = |2a|; scored, not required (comma shapes preferred, straight rods kept). 7. **Clustering** — single-link BFS on the 30 px radius graph, minimum cluster 3. 8. **Window score** = `2 * max_cluster + n_hits_clustered + 0.25 * n_hits`. Also recorded: `hit_density_per_kpx`, `eligible_px`, `mucus_band_px`, `interface_px` (pale space hugging *nucleated* epithelium — pit lumen and surface mucus, as opposed to interstitial space), `tissue_fraction`, `nucleus_fraction`, `focus` (99.9th percentile band-pass over tissue). 18683 hits survived across 753 windows; 890 CPU-seconds of detector time. ## False-positive calibration on normal surfaces Full table in `organisms/logs/calibration.json`. The negative arm is **hard**: it is normal-gland mucosal surface from the *same 13 patients*. If those stomachs are H. pylori positive, organisms can sit on histologically near-normal mucosa too, so the rates below are an upper bound on specificity loss — but that cuts both ways, and it does not explain a 1.0-1.5x enrichment. Windows gated to `nucleus_fraction >= 0.004` and `interface_px >= 12000` (the gate used to pick the top 60): 230 positive, 62 negative. | window score >= | positive rate | false-positive rate on normal surfaces | enrichment | |---|---|---|---| | 10 | 0.396 | 0.274 | 1.44 | | 15 | 0.283 | 0.161 | 1.75 | | 20 | 0.178 | 0.081 | 2.21 | | 25 | 0.113 | 0.065 | 1.75 | | 30 | 0.065 | 0.032 | 2.02 | | 40 | 0.017 | 0.016 | 1.08 | Ungated (613 positive, 140 negative) the best point is score >= 30: positive rate 0.057, false-positive rate 0.021, enrichment 2.66. "At least one cluster of 3" fires in 24.1% of positives and 20.7% of negatives; "max cluster >= 5" fires in 1.8% of positives and **2.1% of negatives**. **A real organism detector should be near-absent on normal surfaces. This one is not.** ### The eligible-region sweep (`organisms/logs/sweep1-eligible-region.json`) Nine settings over the cached windows, ranked by the score distribution of each arm: | pale brightness | min pale component px | median eligible px (pos) | pos p90 | neg p90 | enrichment | |---|---|---|---|---|---| | 200 | 1500 | 410313 | 24.0 | 20.8 | 1.63 | | 200 | 4000 | 401982 | 24.5 | 20.8 | 1.63 | | 200 | 10000 | 391900 | 24.8 | 21.2 | 1.66 | | **215** | 1500 | 198285 | **1.8** | 2.0 | 0.89 | | **215** | 4000 | 177633 | **1.5** | 1.8 | 1.03 | | **215** | 10000 | 156805 | **1.5** | 1.5 | 1.09 | | **225** | 1500 | 120676 | **0.0** | 0.0 | 0.63 | | **225** | 4000 | 100800 | **0.0** | 0.0 | 0.57 | | **225** | 10000 | 86688 | **0.0** | 0.0 | 0.62 | Every hit the detector produces lives in the 200-215 brightness band — the interstitial texture of loose stroma, not open mucus. Actual lumen and mucus are empty. ### The sensitivity sweep in true luminal space (`organisms/logs/sweep2-sensitivity.log`) Fixed at pale 215 / min component 4000, nine (threshold_abs, threshold_k) settings: | threshold_abs | threshold_k | pos p90 | neg p90 | enrichment | |---|---|---|---|---| | 0.018 | 2.0 | 22.2 | 22.2 | 1.21 | | 0.018 | 2.6 | 15.0 | 17.0 | 0.75 | | 0.018 | 3.2 | 7.5 | 11.5 | 0.62 | | 0.028 | 2.0 | 15.2 | 17.0 | 0.91 | | 0.028 | 2.6 | 10.0 | 11.8 | 0.88 | | 0.028 | 3.2 | 5.5 | 7.8 | 1.03 | | 0.045 | 2.0 | 1.5 | 1.8 | 1.09 | Cranking sensitivity in the luminal compartment finds noise, symmetrically on both arms. ## Candidate grading `organisms/grades.json` (per-candidate grade and substrate), `organisms/candidates/index.json` (detector numbers that produced the ranking), `organisms/candidates/sheet-01.png` .. `sheet-05.png` (12 per sheet, as briefed) and `organisms/candidates/review/review-01.png` .. `review-15.png` (2x2 at ~3x, which is what actually made grading possible — the 12-up sheets are too small for 2-4 um objects). **How reviewed.** All 60 candidates viewed as 256 x 256 crops at 4x nearest-neighbour zoom with 2 um and 10 um scale bars. 54 on the 2x2 review sheets; 9 individually at full 1024 x 1024. Detector calls audited on the matching `_zoomdet.png` overlays and on three hit galleries (`organisms/gallery/pos-cluster.png`, `neg-cluster.png`, `pos-curved.png` — 64 x 64 boxes at 6x). | grade | n | reasoning | |---|---|---| | `likely organisms` | **0** | Nothing in the 60 shows what the grade requires: a cluster of faint, uniform, gently curved rods 2-4 um long and 0.5-1 um wide lying free in surface mucus or a pit lumen against epithelium. | | `ambiguous` | 5 | Correct compartment and small dark objects of about the right size, but equally explicable as stain precipitate, pigment, small nuclei, debris or fibrin. `28f9b423_28751_46953` is the clearest case: a textbook foveolar pit lumen with apical mucin caps, carrying scattered **dark blue-black angular specks** that read as haematoxylin precipitate or pigment rather than uniform curved bacilli. Others: `da8667dc_88025_14436`, `40f3c0cf_14941_38104`, `6c6bcc80_25352_25986`, `4769423f_18600_58565`. | | `not organisms` | 55 | Identifiable as something else. | Substrate breakdown of the 55: **fibre** 17 (collagen, muscularis mucosae, basement membrane, capillary wall — eosinophilic and continuous), **stroma** 16 (loose/oedematous lamina propria; the detector fires on the fine strands crossing interstitial space), **nuclei** 12 (elongated or tangentially cut endothelial, fibroblast, lymphocyte and plasma-cell nuclei — too long, too dark, visible chromatin), **fibrin** 6 (surface fibrin and mucin strands), **epithelium_no_organisms** 4 (genuine foveolar/glandular epithelium with a lumen in which no rods are visible). The negative-arm gallery is indistinguishable from the positive-arm gallery: the same thin pink-purple strands with a bluish edge, the same elongated nuclei, plus obvious muscle bundles and capillary cross-sections. **Again: this is a model's structured visual review, not a pathologist's verification.** Nothing here settles the H. pylori status of any HiESD case. Yash must confirm. ## Library `organisms/library/` — 12 windows, 1024 x 1024 RGB PNG at 0.25 um/px, from **8 distinct slides**, with `organisms/library/index.json` (slide id, level-0 coordinates, raw read size, HiESD category, every detector score, grade, grade reason, anatomy note, sha256). Because **no candidate graded `likely organisms`**, this is **not an organism library** and `index.json` says so at the top of the file. The brief asked for at least 8 windows even at lower confidence, marked; these are the 12 whose **anatomy** is the compartment H. pylori would occupy — the 5 `ambiguous` ones first, then 7 chosen for a real surface-mucus / pit-lumen / gland-lumen interface. Every entry carries `organism_verified: false`. | file | slide | category | grade | anatomy | |---|---|---|---|---| | `28f9b423_28751_46953.png` | 28f9b423 | hpylori_gastritis | ambiguous | foveolar pit lumen with apical mucin caps | | `da8667dc_88025_14436.png` | da8667dc | hpylori_gastritis | ambiguous | gland lumina, interglandular cleft | | `4769423f_18600_58565.png` | 4769423f | hpylori_gastritis | ambiguous | surface mucinous exudate | | `6c6bcc80_25352_25986.png` | 6c6bcc80 | hpylori_gastritis | ambiguous | mucosal surface with mucus film | | `40f3c0cf_14941_38104.png` | 40f3c0cf | hpylori_gastritis | ambiguous | eroded surface, detached epithelium in mucus | | `e3367780_41454_39258.png` | e3367780 | hpylori_gastritis | not_organisms | foveolar/glandular epithelium, open lumina | | `9ded7e41_11101_69610.png` | 9ded7e41 | hpylori_gastritis | not_organisms | gastric glands with open lumina | | `28f9b423_88269_55606.png` | 28f9b423 | hpylori_gastritis | not_organisms | gland profile against inflamed lamina propria | | `da8667dc_48498_3880.png` | da8667dc | hpylori_gastritis | not_organisms | gland lumen with adjacent capillary | | `e3367780_53588_23496.png` | e3367780 | hpylori_gastritis | not_organisms | surface mucus and fibrin band | | `badcd77d_32833_8678.png` | badcd77d | mixed | not_organisms | mixed-category surface, glands + inflamed stroma | | `9ded7e41_60061_72554.png` | 9ded7e41 | mixed | not_organisms | mixed-category surface, muscularis and gland bases | `organisms/negatives/` — the comparison arm: 8 normal-gland surface windows from 5 slides, same format, with `index.json` and the same caveat that they are hard negatives from the same patients. ## How this plugs into the generation test (GEN-CODE) GEN-CODE expects `organism-library/*.png` as 1024 x 1024 RGB at 0.25 um/px and builds UNI2-h tokens from them (sixteen 256 crops resized to 224 per window). `organisms/library/*.png` meets that contract byte-for-byte; `index.json` is metadata GEN-CODE can ignore. To wire it up: ```bash ln -s operations/research/sgh-program-20260908/organisms/library organism-library # on the GPU VM, under the package root: # [local]/sgh-program-20260908/organism-library/ ``` For the B1 row of the Phase-2 matrix (gastritis 5 cartoons with organism donors, plus 10 native 1024 windows from organism donors), the 12 library windows give a donor rotation of 12 with per-window provenance for `extra: {exp: "b1", pass: k, cartoon: ..., donor: "", arm: ...}`. **But the test it enables is weaker than the one the brief asked for.** The intended experiment was "does PixCell render organisms when conditioned on windows that contain them?" That question cannot be asked, because no window was verified to contain them. What these 12 windows *can* test: 1. Whether conditioning on real gastritis **surface** tokens (mucus band, pit lumens, foveolar epithelium against glass) makes PixCell render surface architecture that the current donor rotation — drawn from 2048 x 1024 wide fields, mostly mid-mucosa — does not. 2. Whether the procedural organism stage (`scripts/pixcell_hp_stage.py`, rods 10-18 px, faint grey-blue, clustered in large lumina) survives a light img2img pass conditioned on these donors, i.e. whether PixCell preserves or erases a 2-4 um painted object. That is a genuine and answerable question, and this library is the right donor set for it. 3. As a **negative control**, `organisms/negatives/*.png` gives normal surfaces for the same test. The honest framing for any downstream write-up: the organisms in SGH outputs remain **procedural**. Nothing in this package produced a verified real-organism reference. ## Limitations 1. **The premise is unverified.** HiESD annotates "Chronic Gastritis" and "Chronic atrophic gastritis". It does **not** annotate H. pylori status. The work-package brief treated the gastritis slides as H. pylori gastritis; that is an assumption, not a dataset fact. HiESD is an ESD (endoscopic submucosal dissection) cohort for gastric neoplasia, where eradication therapy before or after resection is routine — post-eradication and atrophic/metaplastic mucosa both carry low or zero organism density. A null result here is entirely consistent with the slides simply not having many organisms. 2. **H&E is the wrong stain.** H. pylori on H&E is faint and unreliable even under a microscope; Giemsa, Warthin-Starry or immunohistochemistry is the standard. No permitted dataset here has those. 3. **Effective optical resolution.** Nominal 0.2458 um/px at 40x, and the best fields do resolve nuclear chromatin and individual red cells, so a 3 x 0.6 um rod is in principle resolvable. But sharpness varies a lot across and within slides, and the SVS JPEG compression softens fine detail. My `focus` proxy (99.9th percentile band-pass over tissue) did **not** separate the arms and did **not** reliably flag the visibly out-of-focus regions on `4fdce538` — it is recorded but should not be trusted as a quality gate. 4. **The negatives are hard negatives** from the same patients, as discussed above. 5. **Surface targeting is imperfect.** GrandQC gives the tissue/glass boundary, and the annotation mask distinguishes the mucosal edge from the deep margin, but many sampled windows still land in lamina propria or muscularis rather than on foveolar surface with mucus. A stronger targeting signal — an explicit surface-epithelium detector (palisaded columnar cells with basal nuclei and apical mucin against luminal space) — would raise the yield of anatomically correct windows and is the obvious next improvement if this line is continued. 6. **Grading is by a model, not a pathologist.** Stated once more because it matters: `ambiguous` here means "I cannot tell", not "probably organisms", and `not organisms` is my reading of the substrate, not a diagnosis. 7. **Detector v1 is preserved only in this document.** The first version (no stain-selectivity, no continuity filter, permissive eligible region) scored a *normal* window highest of all 753 and is the reason those two filters exist. The v1 scan file was overwritten by later passes. ## What is where ``` operations/research/sgh-program-20260908/ B1_RESULT.md this file code/organism_detector.py programme-level re-export of the detector (PLAN's code/ convention) organisms/ detector.py the detector; runs on IOFlood and locally, CLI + importable survey_surface.py enumerates mucosal-surface windows from the HiESD masks scan_windows.py level-0 reads + detector, multiprocess export_candidates.py ranks, re-reads winners, writes windows / zoom cards / sheets hit_gallery.py montage of individual hits at 6x, for auditing what fired sweep_config.py detector-setting sweeps over cached windows run_on_ioflood.sh push / survey / scan / export / galleries / pull grades.json per-candidate grade and substrate, with the caveats candidates/ 60 windows, 60 zoom cards, 60 marked cards, sheet-01..05, review/review-01..15 (2x2 at ~3x), index.json library/ 12 windows + index.json (NOT organism-verified) negatives/ 8 normal-surface windows + index.json gallery/ pos-cluster.png, neg-cluster.png, pos-curved.png logs/ windows.jsonl, windows-slides.json, scan.jsonl, calibration.json, sweep1-eligible-region.json, sweep2-sensitivity.log ``` On the workbench everything lives at `[local]/b1-organisms` (symlinked from `[local]/b1-organisms`), including `out/windows-png/` — all 753 scanned windows cached as PNG, 1.3 GB, which is what makes a detector sweep a 2-minute job instead of a re-read of the slides. ## Commands ```bash cd research/sgh-synthetic-histopathology B1=../../operations/research/sgh-program-20260908/organisms bash $B1/run_on_ioflood.sh push # ship the six python files to the workbench bash $B1/run_on_ioflood.sh survey # 13 slides -> out/windows.jsonl (~2 min) bash $B1/run_on_ioflood.sh scan # 753 windows -> out/scan.jsonl (~2.5 min, 7 workers) bash $B1/run_on_ioflood.sh export # top 60 candidates + 8 negatives + sheets bash $B1/run_on_ioflood.sh galleries # hit galleries for auditing bash $B1/run_on_ioflood.sh pull # fetch candidates/, negatives/, gallery/, logs # detector-setting sweep over the cached windows (about 2 min per setting, 7 workers) ./ioflood exec 'cd [local]/b1-organisms && \ [local]/python code/sweep_config.py \ --windows out/scan.jsonl --png-dir out/windows-png --out out/sweep.json --grid out/grid.json' # run the detector on any 1024x1024 window, anywhere .venv/bin/python operations/research/sgh-program-20260908/organisms/detector.py \ operations/research/sgh-program-20260908/organisms/library/*.png --json-out /tmp/lib.json --full ``` Gotcha: every `./ioflood exec` opens a fresh SSH connection, and a burst of them gets reset by the host (`kex_exchange_identification: read: Connection reset by peer`). `run_on_ioflood.sh` now paces and retries; if you drive `ioflood` directly, pause between calls. All six shipped files were verified md5-identical between the repo and `.artifacts/b1-organisms/code/` after the final push. ## Recommendation Do not present any of this as real organism evidence. Either - keep the organisms procedural and say so plainly (the current `pixcell_hp_stage.py` path), using this library as the **surface-anatomy donor set** and running test (2) above — does PixCell preserve a painted 2-4 um object through img2img — which is a real, answerable question; or - if verified organisms are genuinely needed, they have to come from a dataset with a special stain or an H. pylori annotation. Chasing them further on HiESD H&E with classical image processing is, on this evidence, not going to work.