D4diffCyto

Highlights

Computational cytology is a cell-level search problem: a whole-slide image may contain enormous numbers of cells, but only a tiny fraction may be abnormal.
Detailed cell-level annotation is difficult to scale, so D4diffCyto follows a normality-learning setting where the model is trained on normal cells and used to rank suspicious cells.
Unlike many histology patches, a centered cytology cell patch has no preferred global orientation: rotating or flipping the cell should not change its patch-level diagnostic label.
Standard diffusion anomaly detectors may nevertheless reconstruct rotated or flipped versions of the same cell differently, causing unstable anomaly scores and top-K rankings.
D4diffCyto addresses this by enforcing dihedral D4 symmetry, covering rotations and reflections of the square image grid.
We study two complementary routes: inference-time D4 enforcement using Frame Averaging and Equivariant Noise, and architectural D4 enforcement using a D4-equivariant U-Net.
On MLL, FA+EN improves top-400 abnormal-cell retrieval from 57 to 78 compared with vanilla AnoDDPM.
On AML-LMU, the D4-equivariant U-Net with attention achieves the best AUC and top-400 retrieval among the evaluated diffusion variants.

Why cytology?

Cytology studies disease at the level of individual cells and small cell clusters. In liquid-based cytology and other cytology workflows, cells are collected from the patient, placed on a slide, scanned as a whole-slide image, and then analyzed computationally. This makes cytology highly suitable for cell-level artificial intelligence, but also creates an enormous search problem.

A digital cytology slide may contain tens or hundreds of thousands of cells. In many screening settings, only a very small fraction of these cells may actually be malignant or atypical. The practical task is therefore not only to classify a slide, but to help the expert find the few suspicious cells hidden within a vast background of normal cells.

The central goal of D4diffCyto is to make diffusion-based anomaly detection useful for this “needle-in-a-haystack” setting: learn normality from normal cells, assign anomaly scores to test cells, and retrieve a compact top-K shortlist for expert review.

Cytology workflow from sample collection to whole-slide scanning and cell detection

A cytology workflow: sample collection, slide preparation, whole-slide scanning, and cell or nucleus detection for computational analysis.

The needle-in-a-haystack problem

Whole-slide cytology image with many cells

Cytology slides can contain a very large number of candidate cells. However, abnormal cells may be extremely rare, especially in early detection and screening settings. This creates severe class imbalance and makes exhaustive manual search difficult.

The problem is even harder because abnormal morphology can be heterogeneous and subtle. Cell-level annotation requires substantial expert time, and in many real datasets we may know the slide diagnosis but not the label of every individual cell.

For this reason, D4diffCyto is framed as normality learning: the model learns the appearance of normal cells and then ranks test cells by how strongly they deviate from that learned normality.

Whole-slide cytology contains a very large search space. The aim is to reduce this search space to a manageable list of suspicious cells.

Why D4 symmetry is especially natural in cytology

Histology WSI patches

Example histology WSI patch from the CAMELYON dataset.

A histology patch often preserves tissue architecture, neighborhood organization, glands, stroma, invasive fronts, or other contextual structures.
The label of a patch may depend not only on a single object, but also on how structures are arranged within the tissue context.
Rotations and flips may still be useful as augmentation, but the interpretation of a patch can be tied to larger tissue-level organization.

Cytology WSI patches

Detected cells in a cytology whole-slide image patch

In cytology, the signal is often cell-centric: a cropped patch is centered around an individual cell or a small cell cluster.
The diagnostic identity of a centered cell should not change if the patch is rotated by 90°, 180°, or 270°, or reflected.
This makes the patch-level label naturally invariant to the D4 group of square-image rotations and reflections.

D4diffCyto exploits a simple biological and imaging fact: in liquid-based and cell-centric cytology, a cell has no preferred orientation. A diffusion anomaly detector should therefore produce stable reconstructions and anomaly scores under rotations and flips of the same cell.

From annotation to normality learning

Exhaustively annotating all abnormal cells is difficult because a slide can contain an enormous number of candidate cells, expert time is limited, and abnormal morphology can vary between cases. Weak supervision through slide-level labels is practical, but when abnormal cells are extremely rare, the slide label becomes a weak signal for identifying the actual abnormal instances.

D4diffCyto follows the broader idea of learning from normality. Cells from negative slides can be treated as reliably normal. A model can therefore learn a normal-cell appearance prior without needing abnormal-cell annotations. At test time, cells that deviate strongly from this normality model receive high anomaly scores and can be ranked for expert inspection.

Normal slide-negative cells No cell-level abnormal labels Anomaly ranking Human-in-the-loop review

From AnoDDPM to D4diffCyto

The starting point is AnoDDPM-style anomaly detection. Instead of generating a sample from pure noise, the test image is noised only up to an intermediate diffusion step and then denoised back. Because the model is trained on normal data, the reverse process tends to reconstruct a pseudo-normal version of the query. The anomaly map is obtained from the reconstruction residual.

Input patch x0 → partial noising to xt* → reverse diffusion reconstruction x̂0
Anomaly map: |x0 − x̂0|
Anomaly score: mean squared reconstruction error

D4diffCyto keeps this partial-reconstruction principle but adapts it to cell-centric cytology. The key modification is that reconstruction should be consistent under the D4 group: rotations by 0°, 90°, 180°, 270° and their reflected counterparts. The goal is not only to obtain a good anomaly score, but to make the score stable under diagnostically irrelevant transformations.

What are FA and EN?

Frame Averaging (FA)

Frame Averaging applies the model to all transformed views of the same input and then maps the outputs back to the original orientation before averaging. For D4diffCyto, the frame is the D4 orbit of a cell patch: all rotations and flips of the same image.

Intuitively, FA asks the model: “What anomaly score would you give if the same cell appeared in every allowed orientation?” Averaging over these answers suppresses orientation-specific fluctuations and produces a more symmetry-consistent score.

FA score ≈ average over g in D4 of Score(g · x)

Equivariant Noise (EN)

Diffusion sampling is stochastic. Even if the denoiser is equivariant, independently sampled noise can make the trajectory for a rotated cell differ from the rotated trajectory of the original cell. EN fixes this by tying the random noise across transformed views.

For each D4 transform g, D4diffCyto uses the transformed version of the same base noise. Thus, the forward and reverse trajectories share the same underlying randomness up to the group action.

Base noise: ε
Noise for transformed view g · x: εg = g · ε

FA and EN play complementary roles. FA symmetrizes the outputs over the D4 orbit, while EN makes the stochastic diffusion trajectory itself consistent across transformed views. In D4diffCyto, the two are combined for inference-time D4 enforcement with a standard U-Net.

How FA+EN is adapted in D4diffCyto

In generic structure-preserving diffusion, FA and EN are used to make diffusion outputs respect a chosen transformation group. D4diffCyto adapts this idea to the specific operational problem of cytology screening: the system should rank abnormal cells consistently even when a cell appears rotated or flipped.

For inference-time D4 enforcement, a standard U-Net is first trained using the usual diffusion objective. No special group-equivariant architecture is required during training. At test time, the input patch is transformed by each element of D4. The same base noising variables are transformed accordingly, partial diffusion reconstruction is performed for each transformed view, residuals are mapped back to the original orientation, and scores are averaged.

Start with a query cell patch x0.
Generate the D4 orbit: rotated and flipped versions of x0.
Use equivariant noise so each transformed view receives the correspondingly transformed stochastic noise.
Run partial diffusion reconstruction for each view.
Map anomaly maps back to the original orientation.
Average residual maps and scalar scores over the orbit.
Use the final averaged score for top-K abnormal-cell ranking.

This gives a simple way to enforce symmetry at inference time while retaining the ordinary U-Net training pipeline. It is especially useful when one wants to add symmetry consistency to a trained diffusion anomaly detector without redesigning the entire model architecture.

Abstract

Computational cytology on whole-slide images is challenging because malignant cells are rare, heterogeneous, and annotated slides are scarce. Anomaly detection frameworks can be trained on normal slide-negative patches and then applied at test time to flag abnormal patches in held-out slides. Most unsupervised anomaly detection approaches, including generative ones, are tuned to organ-level imaging and require large curated datasets. In cytology, the signal is cell-centric: rotating or flipping a single-cell patch does not change its diagnostic class, yet standard diffusion models treat transformed views as distinct inputs, leading to transformation-dependent reconstructions and unstable anomaly scores.

We propose a D4-equivariant diffusion framework that enforces rotation and reflection symmetry both architecturally, via a D4-equivariant U-Net, and at inference, via equivariant noise coupling and frame averaging. This alignment with biological invariance yields transformation-consistent pseudo-healthy reconstructions and more stable anomaly ranking under symmetry. On two publicly available cytology datasets of bone marrow and peripheral blood smears, our D4-equivariant diffusion models achieve higher AUC and retrieve more abnormal cells in the top-K predictions than non-equivariant generative baselines, a deep one-class baseline, and a multiple instance learning based method, while substantially reducing score variance across rotations and flips.

Method Overview

1. Normal-only training

The diffusion model is trained only on normal cytology patches. This is well suited to cytology, where collecting exhaustive abnormal-cell annotations is difficult, but normal or slide-negative examples are more accessible.

2. Partial reconstruction

At test time, the query patch is noised to an intermediate timestep and denoised back. The reconstruction is interpreted as a pseudo-normal version of the input, and the residual is used as the anomaly signal.

3. Top-K retrieval

Instead of only asking whether a whole slide is positive, the method ranks individual cell patches. This is important for realistic cytology workflows where only a small number of candidate cells may be inspected.

Inference-time D4 route: FA+EN

A standard U-Net is trained as usual. During inference, D4 consistency is imposed by applying rotations and flips to the input, coupling their diffusion noise through EN, reconstructing each transformed view, and averaging the resulting scores and residual maps using FA.

Architectural D4 route: D4 U-Net

The denoiser itself is made D4-equivariant using group-tied convolutional filters. This reduces orientation-specific variability inside the model and encourages the denoising function to respect rotations and reflections by design.

D4 U-Net with attention

The attention-augmented variant extends the D4-equivariant backbone with an attention block, allowing the model to capture richer spatial interactions while still preserving the intended group-aware structure.

These two routes are complementary. FA+EN is an inference-time mechanism that can be applied to a standard denoiser, while the D4-equivariant U-Net builds the symmetry into the architecture itself. Both are designed to reduce nuisance orientation variance and improve the reliability of abnormal-cell ranking.

Main Quantitative Results

Method	MLL				AML-LMU
Method	AUC	TP400	TP200	TP100	AUC	TP400	TP200	TP100
Baselines
ItS2CLR	0.521107	27	20	19	0.360515	92	38	13
Deep SVDD	0.613331	49	35	21	0.514742	227	122	56
f-AnoGAN	0.594601	37	24	9	0.538857	273	131	57
THOR Gaussian	0.540851	24	14	7	0.582212	247	137	68
THOR Simplex	0.404264	8	3	1	0.544415	207	104	58
BerDiff	0.550799	44	32	26	0.708558	313	157	64
AnoDDPM	0.632846	57	30	18	0.694320	311	162	83
D4-equivariant diffusion variants
FA only	0.612543	46	32	19	0.680111	277	144	72
FA+EN	0.684121	78	53	34	0.678373	285	149	75
D4 conv.	0.675404	74	53	27	0.699635	315	159	86
D4 conv. + attn.	0.679251	67	44	33	0.720098	325	162	81

Patch-level anomaly detection results on MLL and AML-LMU. TP400, TP200, and TP100 denote the number of abnormal cells retrieved among the top-K most anomalous patches.

Qualitative Results

Qualitative comparison between AnoDDPM and D4-equivariant AnoDDPM

Qualitative comparison of vanilla AnoDDPM and D4-equivariant AnoDDPM on MLL bone marrow and AML-LMU peripheral blood cytology patches. Each panel shows inputs, reconstructions, and anomaly-map overlays with MSE scores.

BibTeX

@misc{chatterjee2026d4diffcyto,

title         = {Group Equivariant Diffusion for Anomaly Detection in Computational Cytology},
author        = {Chatterjee, Swarnadip and Sivakumar, Ssharvien Kumar and Mukhopadhyay, Anirban},
year          = {2026},
eprint        = {2607.25503},
archivePrefix = {arXiv},
primaryClass  = {cs.CV}
}

Group Equivariant Diffusion for Anomaly Detection in Computational Cytology

D4diffCyto enforces rotation and reflection symmetry in diffusion-based anomaly detection for computational cytology.