Group Equivariant Diffusion for Anomaly Detection in Computational Cytology

• Computational cytology is challenging because abnormal cells are rare, heterogeneous, and expensive to annotate.
• Diffusion anomaly detectors can be trained on normal slide-negative patches and then used to flag abnormal cells at test time.
• A key problem is that standard diffusion models may produce different reconstructions and anomaly scores for rotated or flipped versions of the same cell patch.
• D4diffCyto addresses this by enforcing dihedral symmetry, covering rotations and reflections of single-cell patches.
• We study two complementary routes to D4-consistent anomaly scoring: inference-time symmetry enforcement using Frame Averaging and Equivariant Noise, and architectural symmetry enforcement using a D4-equivariant U-Net.
• The method is evaluated on two public cytology datasets: MLL bone marrow cytomorphology and AML-LMU peripheral blood cytomorphology.
• On MLL, FA+EN improves top-400 abnormal-cell retrieval from 57 to 78 compared with vanilla AnoDDPM.
• On AML-LMU, the D4-equivariant U-Net with attention achieves the best AUC and top-400 retrieval among the evaluated diffusion variants.
• D4-aware diffusion reduces anomaly-score variation across rotations and flips, making top-K retrieval more stable for realistic cytology screening workflows.

Diffusion-based anomaly detection is attractive for cytology because it can be trained using only normal cells. A model learns a normal-cell appearance prior, and at test time an abnormal query patch is partially diffused and reconstructed toward the normal distribution. The residual between the input and the reconstruction becomes the anomaly signal.

However, single-cell cytology is different from organ-level medical imaging. In brain MRI, retinal OCT, or chest X-ray, anatomy often has a fixed orientation and lesions are spatially tied to that anatomy. In cytology, a centered cell patch can be rotated or flipped without changing its diagnostic meaning. A malignant or abnormal cell remains abnormal under the eight rotations and reflections of the square image grid.

This creates a structural mismatch: a standard diffusion model may treat rotated or flipped versions of the same cell as different inputs, leading to different pseudo-healthy reconstructions and different anomaly scores. Under extreme class imbalance, even small score changes can alter the top-K ranking and affect which cells are surfaced for expert review.

The starting point is AnoDDPM-style anomaly detection. Instead of generating a sample from pure noise, the test image is noised only up to an intermediate diffusion step and then denoised back. This partial diffusion reconstruction is useful because the model is trained on normal data, so the reverse process tends to reconstruct a pseudo-normal version of the query. The anomaly map is obtained from the reconstruction residual.

D4diffCyto keeps this partial-reconstruction principle but adapts it to cell-centric cytology. The key modification is that reconstruction should be consistent under the D4 group: rotations by 0°, 90°, 180°, 270° and their reflected counterparts. The goal is not only to obtain a good anomaly score, but to make the score stable under diagnostically irrelevant transformations.

In generic structure-preserving diffusion, FA and EN are used to make diffusion outputs respect a chosen transformation group. D4diffCyto adapts this idea to the specific operational problem of cytology screening: the system should rank abnormal cells consistently even when a cell appears rotated or flipped.

For inference-time D4 enforcement, a standard U-Net is first trained using the usual diffusion objective. No special group-equivariant architecture is required during training. At test time, the input patch is transformed by each element of D4. The same base noising variables are transformed accordingly, partial diffusion reconstruction is performed for each transformed view, residuals are mapped back to the original orientation, and scores are averaged.

1. Start with a query cell patch x0.
2. Generate the D4 orbit: rotated and flipped versions of x0.
3. Use equivariant noise so each transformed view receives the correspondingly transformed stochastic noise.
4. Run partial diffusion reconstruction for each view.
5. Map anomaly maps back to the original orientation.
6. Average residual maps and scalar scores over the orbit.
7. Use the final averaged score for top-K abnormal-cell ranking.

This gives a simple way to enforce symmetry at inference time while retaining the ordinary U-Net training pipeline. It is especially useful when one wants to add symmetry consistency to a trained diffusion anomaly detector without redesigning the entire model architecture.

Computational cytology on whole-slide images is challenging because malignant cells are rare, heterogeneous, and annotated slides are scarce. Anomaly detection frameworks can be trained on normal slide-negative patches and then applied at test time to flag abnormal patches in held-out slides. Most unsupervised anomaly detection approaches, including generative ones, are tuned to organ-level imaging and require large curated datasets. In cytology, the signal is cell-centric: rotating or flipping a single-cell patch does not change its diagnostic class, yet standard diffusion models treat transformed views as distinct inputs, leading to transformation-dependent reconstructions and unstable anomaly scores.

We propose a D4-equivariant diffusion framework that enforces rotation and reflection symmetry both architecturally, via a D4-equivariant U-Net, and at inference, via equivariant noise coupling and frame averaging. This alignment with biological invariance yields transformation-consistent pseudo-healthy reconstructions and more stable anomaly ranking under symmetry. On two publicly available cytology datasets of bone marrow and peripheral blood smears, our D4-equivariant diffusion models achieve higher AUC and retrieve more abnormal cells in the top-K predictions than non-equivariant generative baselines, a deep one-class baseline, and a multiple instance learning based method, while substantially reducing score variance across rotations and flips.