I kept running into the same uncomfortable question with single-cell perturbation benchmarks: what if the model result is clean, but the benchmark itself is not?
That sounds a little dramatic, but it is a real problem. A perturbation model can look useful because the split leaked perturbations across train and test, because the batch variable quietly predicts the condition, because controls are thin, or because the held-out task is not actually the biological claim being made.
PerturbGuard is my attempt to make that question less hand-wavy. Before asking whether a model generalizes, it asks whether the dataset and split design give the model a fair biological question in the first place.
The useful mental model is FastQC, but for perturbation benchmarks. You point it at an AnnData .h5ad file, give it a config or let it infer one, and it produces HTML, CSV, JSON, and Markdown outputs that explain what looks trustworthy and what needs a closer look.
The checks are deliberately boring in the best way. Does the object load? Are the cell and gene axes sane? Are controls present? Are perturbation labels consistent? Are targets mapped? Does the split actually hold out what the paper or benchmark says it holds out?
Once those pieces are in place, the tool can look for more interesting failure modes: train-test perturbation overlap, weak target support, metadata confounding, combination-split leakage, and claims about unseen targets or unseen perturbations that are not really supported by the held-out set.
The command-line surface is intentionally small:
perturbguard audit --data data/repaired.h5ad --config configs/inferred.yaml --out results/auditThe point of the output is not to give a magical pass or fail. I wanted the report to make the reasoning inspectable: what was checked, what was flagged, which tables support the warning, and which parts of the benchmark contract still need a human decision.
This is also where I would be careful about the claim. PerturbGuard cannot prove that a perturbation model is biologically right. It can only make it harder to accidentally trust a model for reasons that live in the dataset split rather than in the biology.
The next version I would want is a small benchmark suite of known-good and known-bad perturbation datasets. That would let the tool say not only “this pattern looks risky,” but also how often that pattern catches real benchmark failures.
References and artifacts: the project code and examples are in the PerturbGuard repository.