Two models for the same counts

Take one gene measured in several mice from the same group. Even with perfect sequencing, the gene’s true expression differs from mouse to mouse. The read count you see has two layers of variation: the biological differences between mice, and the sampling of reads from each library.

The Poisson distribution describes only the second layer. If a library contains a gene at an expected count of $\mu$, the count has variance $\mu$. That is a good model for technical replicates, the same library sequenced on several lanes (Marioni et al. 2008), but not for biological ones.

The negative binomial distribution adds the first layer. Its variance is $\mu + \phi \mu^2$, where $\phi$ is the dispersion: the squared coefficient of variation of the true expression between replicates. This is the model behind edgeR (Robinson, McCarthy and Smyth 2010; McCarthy, Chen and Smyth 2012) and DESeq2 (Love, Huber and Anders 2014). The ratio of the two variances is $1 + \phi \mu$, so the gap grows with expression. For a gene with a modest dispersion of 0.05 and a mean count of 150, the real variance is 8.5 times the Poisson variance.

The setup

Each gene gets a baseline mean count between a few reads and several thousand, and a dispersion that falls with expression and scatters between genes. There are 4 biological replicates per condition, and 10% of the 3,000 genes truly change by a factor of two, up or down. The rest are null. Sequencing depth differs between samples.

n_genes <- 3000
n_per <- 4                                        # biological replicates per condition
de_frac <- 0.10
group <- factor(rep(c("ctrl", "treat"), each = n_per))
is_de <- seq_len(n_genes) <= de_frac * n_genes

simulate_counts <- function() {
  mu0 <- exp(rnorm(n_genes, log(150), 1.2))       # baseline mean count per gene
  phi <- (0.05 + 0.5 / mu0) * exp(rnorm(n_genes, 0, 0.5))   # NB dispersion, gene to gene scatter
  lfc <- ifelse(is_de, sample(c(-1, 1), n_genes, replace = TRUE), 0)   # log2 fold change
  depth <- exp(rnorm(2 * n_per, 0, 0.2))          # sequencing depth per sample
  mu <- outer(mu0, depth) * 2^outer(lfc, as.numeric(group == "treat"))
  counts <- matrix(rnbinom(length(mu), mu = mu, size = 1 / phi), nrow = n_genes)
  storage.mode(counts) <- "integer"
  rownames(counts) <- paste0("gene", seq_len(n_genes))
  counts
}
set.seed(2026)
counts <- simulate_counts()

phi_example <- 0.05
mu_example <- 150
ratio_example <- 1 + phi_example * mu_example

All four analyses below start from the same filtered table: the edgeR recipe of filterByExpr() to drop genes with too few reads to test, then normLibSizes() for TMM normalisation. The Poisson, quasi-Poisson and edgeR fits use the same normalised library sizes as offsets (an offset is a per-sample adjustment for sequencing depth that the model takes as given), so between them only the variance differs. DESeq2 runs its standard workflow on the same genes and estimates its own size factors, and it tests with a Wald test rather than a likelihood ratio or F test.

design <- model.matrix(~ group)
y <- DGEList(counts, group = group)
keep <- filterByExpr(y, design)
y <- normLibSizes(y[keep, , keep.lib.sizes = FALSE])
is_de_k <- is_de[keep]
lib <- getNormLibSizes(y)                         # library size x TMM factor

# Poisson and quasi-Poisson GLM, one per gene (closed form, checked against glm() below)
pt <- pois_tests(y$counts, group, lib)

# edgeR quasi-likelihood pipeline: negative binomial GLM + empirical Bayes QL dispersion
fit <- glmQLFit(y, design, legacy = FALSE)
p_ql <- glmQLFTest(fit, coef = 2)$table$PValue

# DESeq2 standard workflow on the same genes
dds <- DESeqDataSetFromMatrix(y$counts, data.frame(group = group), ~ group)
p_deseq <- results(DESeq(dds, quiet = TRUE))$pvalue

s_pois <- score(pt$p_pois, is_de_k)
s_qp <- score(pt$p_qp, is_de_k)
s_ql <- score(p_ql, is_de_k)
s_deseq <- score(p_deseq, is_de_k)
n_kept <- sum(keep)
df_prior <- fit$df.prior
df_gene <- ncol(counts) - ncol(design)
crit_known <- qchisq(0.95, 1)                     # 5% cut-off with the variance known
crit_qp <- qf(0.95, 1, df_gene)                   # quasi-Poisson: variance from df_gene df
crit_ql <- qf(0.95, 1, df_gene + df_prior)        # edgeR QL: plus the prior df

A loop of glm() calls, one per gene, is how most people would write the Poisson test, and it is slow for thousands of genes. With two groups and a log link (the model works on the log of the expected count) the fitted values have a closed form, a direct formula with no iterative fitting, so the helper pois_tests() in the setup chunk computes the same likelihood ratio tests for all genes at once. The next chunk checks it against glm() and anova() on the first 200 genes.

n_check <- 200
off <- log(lib)
p_glm <- t(sapply(seq_len(n_check), function(i) {
  k <- y$counts[i, ]
  f_p <- glm(k ~ group + offset(off), family = poisson)
  f_q <- glm(k ~ group + offset(off), family = quasipoisson)
  c(anova(f_p, test = "Chisq")[2, "Pr(>Chi)"], anova(f_q, test = "F")[2, "Pr(>F)"])
}))
max_diff <- max(abs(p_glm - as.matrix(pt[seq_len(n_check), c("p_pois", "p_qp")])))
stopifnot(max_diff < 1e-4)

Across those genes the largest difference in p-value between the two routes is 6.6e-06.

What the Poisson test does to null genes

tab <- data.frame(
  model = c("Poisson GLM", "quasi-Poisson GLM", "edgeR QL (negative binomial)",
            "DESeq2 (negative binomial)"),
  `null genes with p < 0.05` = sapply(list(s_pois, s_qp, s_ql, s_deseq), function(s) pct(s$null_p05)),
  `calls at 5% FDR` = sapply(list(s_pois, s_qp, s_ql, s_deseq), function(s) num(s$n_call)),
  `false among calls` = sapply(list(s_pois, s_qp, s_ql, s_deseq), function(s) pct(s$fdp)),
  `changed genes found` = sapply(list(s_pois, s_qp, s_ql, s_deseq), function(s) pct(s$power)),
  check.names = FALSE)
knitr::kable(tab, align = "lrrrr")

The Poisson test calls 1,588 genes at 5% FDR. It finds nearly every changed gene, 98.7% of them, but 81.4% of its calls are false. The model assumes that each count varies only by the sampling of reads, so ordinary differences between mice in the same group look to it like a treatment effect. Both negative binomial tools keep the share of null genes below 0.05 close to the promised 5%. DESeq2’s share of false calls, 8.1%, is a little above 5%, and not only by chance: over 10 further simulated data sets it ranged from 6.5% to 10.0%, against 1.9% to 6.3% for edgeR. The scoring applies the Benjamini-Hochberg adjustment to every method’s raw p-values in the same way, which skips DESeq2’s own independent filtering; comparing the two negative binomial tools is not the subject of this post.

If you drew a histogram of the Poisson p-values, you would see the anti-conservative shape described in the post on p-value histograms: a floor that leans towards zero. Storey’s estimate of the null fraction from these p-values is 0.32, against a true value of 0.90.

norm <- t(t(y$counts) / (lib / mean(lib)))       # counts scaled to a common depth
gene_mean <- rowMeans(norm)
within_var <- (apply(norm[, group == "ctrl"], 1, var) + apply(norm[, group == "treat"], 1, var)) / 2
ratio_mv <- median(within_var / gene_mean)
phi_mom <- median((within_var - gene_mean) / gene_mean^2)   # one dispersion for all genes
pi0_pois <- mean(pt$p_pois > 0.5) / 0.5          # Storey's null fraction estimate, Poisson p-values
pi0_true <- mean(!is_de_k)

# Pearson goodness-of-fit: under a Poisson model the statistic follows chi-square on df_gene df
gof_fail <- mean(pchisq(pt$pearson, df_gene, lower.tail = FALSE) < 0.05)
disp_med <- median(pt$disp)
bins <- cut(gene_mean, quantile(gene_mean, 0:5 / 5), include.lowest = TRUE,
            labels = paste("bin", 1:5))
disp_bin <- tapply(pt$disp, bins, median)
mean_bin <- tapply(gene_mean, bins, median)

# false positive rate among null genes, by expression bin
fpr <- do.call(rbind, lapply(c("Poisson GLM", "quasi-Poisson GLM", "edgeR QL", "DESeq2"), function(m) {
  p <- switch(m, "Poisson GLM" = pt$p_pois, "quasi-Poisson GLM" = pt$p_qp,
              "edgeR QL" = p_ql, "DESeq2" = p_deseq)
  data.frame(model = m, bin = levels(bins), mean = as.numeric(mean_bin),
             fpr = as.numeric(tapply(p[!is_de_k] < 0.05, bins[!is_de_k], mean)))
}))
fpr$model <- factor(fpr$model, c("Poisson GLM", "quasi-Poisson GLM", "edgeR QL", "DESeq2"))
fpr_pois_low <- fpr$fpr[fpr$model == "Poisson GLM"][1]
fpr_pois_high <- fpr$fpr[fpr$model == "Poisson GLM"][5]

panels <- c("All four models", "Without Poisson, zoomed")
fpr_plot <- rbind(transform(fpr, panel = factor(panels[1], panels)),
                  transform(fpr[fpr$model != "Poisson GLM", ], panel = factor(panels[2], panels)))
ggplot(fpr_plot, aes(mean, fpr, colour = model, shape = model)) +
  geom_hline(yintercept = 0.05, linetype = "dashed", colour = "grey50") +
  geom_line(linewidth = 0.8) + geom_point(size = 2.4) +
  facet_wrap(~ panel, scales = "free_y") +
  scale_x_log10() +
  scale_y_continuous(labels = function(x) paste0(100 * x, "%")) +
  expand_limits(y = 0) +
  scale_colour_manual(values = c("Poisson GLM" = "#d14fa6", "quasi-Poisson GLM" = "#8a8799",
                                 "edgeR QL" = "#5a3fc0", "DESeq2" = "#2f2c3d"), name = NULL) +
  scale_shape_manual(values = c("Poisson GLM" = 16, "quasi-Poisson GLM" = 17,
                                "edgeR QL" = 15, "DESeq2" = 4), name = NULL) +
  labs(x = "median normalised count in bin (log scale)", y = "null genes with p < 0.05") +
  theme_pc + theme(strip.text = element_text(face = "bold"))

The damage grows with expression, as the variance ratio $1 + \phi \mu$ predicts. Among the fifth of genes with the lowest counts (median normalised count 34) the Poisson test already gives p < 0.05 for 30.6% of null genes; in the top fifth (median 720) it reaches 78.1%. Dropping weakly expressed genes therefore does nothing to rescue a Poisson analysis: the genes that remain are the ones where the assumption is furthest off. The other three models stay near the dashed line in every bin.

Diagnostic 1: the mean-variance plot

Variance larger than the mean is called overdispersion, and it is what both diagnostics below measure.

You cannot see the truth in real data, but you can see the variance. The first check needs no model. For each gene, scale the counts to a common sequencing depth, compute the mean across samples and the variance within each condition, and plot one against the other on log scales. The variance is taken within conditions and averaged, so that a real treatment effect does not inflate it.

Under a Poisson model the points would scatter around the diagonal where variance equals mean. Here the median ratio of variance to mean is 7.9, and the cloud pulls away from the diagonal as the mean rises. The violet curve is the negative binomial variance $\mu + \phi \mu^2$ with a single dispersion for all genes, $\phi$ = 0.047, taken as the median over genes of (variance minus mean) divided by the squared mean. It follows the cloud; the diagonal does not.

mv <- data.frame(mean = gene_mean, var = within_var)
grid_m <- 10^seq(log10(min(gene_mean)), log10(max(gene_mean)), length.out = 100)
nb_curve <- data.frame(mean = grid_m, var = grid_m + phi_mom * grid_m^2)
comma <- function(x) format(x, big.mark = ",", scientific = FALSE, trim = TRUE)
ggplot(mv, aes(mean, var)) +
  geom_point(size = 0.7, alpha = 0.35, colour = "#6b6980") +
  geom_abline(slope = 1, intercept = 0, linetype = "dashed", colour = "#d14fa6", linewidth = 0.8) +
  geom_line(data = nb_curve, colour = "#5a3fc0", linewidth = 0.9) +
  scale_x_log10(labels = comma) + scale_y_log10(labels = comma) +
  labs(x = "mean normalised count (log scale)", y = "within-group variance (log scale)") +
  theme_pc

Diagnostic 2: the Pearson dispersion statistic

The second check uses the Poisson fit itself. For each gene, sum the squared Pearson residuals, (observed minus fitted) squared divided by fitted, over the samples, and divide the sum by the residual degrees of freedom: 8 samples minus 2 fitted means leaves 6. This is the Pearson dispersion statistic. If the Poisson model were right, it would scatter around 1, and the sum would follow a chi-square distribution on 6 degrees of freedom.

Here the median statistic is 7.7, and 88.1% of genes fail the chi-square goodness-of-fit test at the 5% level, where about 5% would fail under a Poisson model. The statistic estimates the variance ratio $1 + \phi \mu$, so it also rises with the mean: its median is 2.7 in the lowest fifth of genes and 33.8 in the highest.

You do not need the closed-form helper to get this number for a single gene. It is what summary() of a glm() fit with family = quasipoisson prints as the dispersion parameter: base R’s summary.glm() estimates it as the sum of squared Pearson residuals divided by the residual degrees of freedom.

ggplot(data.frame(mean = gene_mean, disp = pt$disp), aes(mean, disp)) +
  geom_point(size = 0.7, alpha = 0.35, colour = "#6b6980") +
  geom_hline(yintercept = 1, linetype = "dashed", colour = "#d14fa6", linewidth = 0.8) +
  scale_x_log10(labels = comma) + scale_y_log10(labels = comma) +
  labs(x = "mean normalised count (log scale)", y = "Pearson dispersion (log scale)") +
  theme_pc

Quasi-Poisson: right error rate, wrong power

The quasi-Poisson model is the usual textbook answer to overdispersion when the software at hand only offers a Poisson model. It keeps the Poisson mean but lets the variance be a multiple of it, $\theta \mu$, with $\theta$ estimated for each gene by the Pearson statistic above. Tests divide by $\theta$ and compare against an F distribution; in R this is glm(family = quasipoisson) followed by anova(fit, test = "F"), which is what pois_tests() reproduces.

In the table, quasi-Poisson gets the error rate right: 5.3% of null genes below 0.05 and 2.0% false among its calls. But it calls only 98 genes and finds 32.1% of the changed ones, against 69.9% for edgeR.

The cost comes from estimating $\theta$ one gene at a time. With 6 residual degrees of freedom, the estimate is noisy, and the F test pays for that noise with a higher bar. The 5% cut-off for the test statistic is 3.84 when the variance is known and 5.99 on an F distribution with 6 denominator degrees of freedom. edgeR’s quasi-likelihood pipeline also estimates a quasi-dispersion for every gene, on top of the negative binomial variance, but it squeezes those estimates towards a trend fitted across all genes. That empirical Bayes step is worth 11.8 extra degrees of freedom here (prior degrees of freedom: the information borrowed from the other genes, counted as if it were extra replicates), and the cut-off drops to roughly 4.42.

To check that this is the reason, the next chunk applies the same kind of squeezing to the quasi-Poisson dispersions with limma’s squeezeVar(), using the log of the mean count as the trend covariate, and repeats the F test with the pooled degrees of freedom.

sq <- limma::squeezeVar(pt$disp, df = df_gene, covariate = log(gene_mean))
p_qp_shared <- pf(pt$lr / sq$var.post, 1, df_gene + sq$df.prior, lower.tail = FALSE)
s_qp_shared <- score(p_qp_shared, is_de_k)
df_prior_qp <- sq$df.prior

With the shared dispersion, quasi-Poisson gains 13.4 prior degrees of freedom, keeps 5.7% of null genes below 0.05, and finds 70.6% of the changed genes, close to edgeR. So the quasi-Poisson model is not badly calibrated; it is wasteful, because it treats each gene as if the other few thousand had nothing to say about its variance. It also assumes that the variance is proportional to the mean within a gene, which is off when the two conditions differ a lot in expression; in this simulation that mattered much less than the degrees of freedom.

One data set can flatter or punish the quasi-Poisson test, because with so few calls the Benjamini-Hochberg cut-off moves a lot. The next chunk repeats the whole simulation 10 more times with new seeds and keeps only the share of changed genes found.

n_rep <- 10
rep_runs <- do.call(rbind, lapply(seq_len(n_rep), function(seed) {
  set.seed(seed)
  cnt <- simulate_counts()
  yr <- DGEList(cnt, group = group)
  kr <- filterByExpr(yr, design)
  yr <- normLibSizes(yr[kr, , keep.lib.sizes = FALSE])
  de <- is_de[kr]
  libr <- getNormLibSizes(yr)
  ptr <- pois_tests(yr$counts, group, libr)
  gm <- rowMeans(t(t(yr$counts) / (libr / mean(libr))))
  sqr <- limma::squeezeVar(ptr$disp, df = df_gene, covariate = log(gm))
  p_sh <- pf(ptr$lr / sqr$var.post, 1, df_gene + sqr$df.prior, lower.tail = FALSE)
  p_q <- glmQLFTest(glmQLFit(yr, design, legacy = FALSE), coef = 2)$table$PValue
  ddr <- DESeqDataSetFromMatrix(yr$counts, data.frame(group = group), ~ group)
  p_d <- results(DESeq(ddr, quiet = TRUE))$pvalue
  data.frame(pow_qp = score(ptr$p_qp, de)$power, pow_shared = score(p_sh, de)$power,
             pow_ql = score(p_q, de)$power, fdp_ql = score(p_q, de)$fdp,
             fdp_deseq = score(p_d, de)$fdp)
}))
qp_share <- rep_runs$pow_qp / rep_runs$pow_ql     # quasi-Poisson power as a share of edgeR's

Across those runs, quasi-Poisson found 6.1% to 25.1% of the changed genes (mean 18.6%), edgeR 64.4% to 71.6% (mean 69.1%), and quasi-Poisson with the shared dispersion 64.4% to 73.1%. So the data set above is on the kind side for quasi-Poisson: on average it finds 27% of what edgeR finds, and at best 37% of it.

What to do in practice

For RNA-seq counts with biological replicates, use a negative binomial model that shares dispersion information across genes. That is what edgeR and DESeq2 are for, and in their standard workflows they need no tuning for this. The edgeR quasi-likelihood version used above:

design <- model.matrix(~ group)
y <- DGEList(counts, group = group)
keep <- filterByExpr(y, design)
y <- normLibSizes(y[keep, , keep.lib.sizes = FALSE])
fit <- glmQLFit(y, design, legacy = FALSE)
res <- glmQLFTest(fit, coef = 2)
topTags(res)

legacy = FALSE switches on the newer quasi-dispersion method introduced in edgeR 4.0. It estimates one common negative binomial dispersion itself, from the most highly expressed genes, so no separate estimateDisp() call is needed. The default in edgeR 4.0 is still legacy = TRUE, which stops with an error unless estimateDisp() has been run first; writing legacy = FALSE out gives the same method whichever edgeR version you run.

If you have inherited a per-gene Poisson analysis, from a general statistics package or a loop of glm() calls, run the Pearson check before trusting any of it. The helper pois_tests() has to be copied from the setup chunk, and its closed form handles one grouping factor only:

pt_check <- pois_tests(y$counts, y$samples$group, getNormLibSizes(y))
summary(pt_check$disp)

    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
  0.3133   3.5317   7.7152  18.1498  17.8413 584.7593 

For any other design, fit one gene at a time and read the same number from glm(), where k is the gene’s counts and lib the normalised library sizes:

summary(glm(k ~ group + offset(log(lib)), family = quasipoisson))$dispersion

Values scattered around 1 across the whole range of expression would mean the Poisson model is adequate, as it is for technical replicates of one library. Values well above 1 that rise with the mean, as here, mean the Poisson p-values are too small, and a gene list built on them will contain many false calls, as in the table above. If a general GLM tool is all you have, quasi-Poisson is the minimum repair: it controls the error rate, but with a handful of replicates it throws away much of the power that a shared dispersion estimate would give you.

References

• Marioni JC, Mason CE, Mane SM et al. 2008 Genome Research 18(9):1509-1517 (doi:10.1101/gr.079558.108)
• Robinson MD, McCarthy DJ, Smyth GK 2010 Bioinformatics 26(1):139-140 (doi:10.1093/bioinformatics/btp616)
• McCarthy DJ, Chen Y, Smyth GK 2012 Nucleic Acids Research 40(10):4288-4297 (doi:10.1093/nar/gks042)
• Love MI, Huber W, Anders S 2014 Genome Biology 15(12):550 (doi:10.1186/s13059-014-0550-8)

R version 4.5.3 (2026-03-11)
Bioconductor 3.22
Biobase 2.70.0, BiocGenerics 0.56.0, DESeq2 1.50.2, edgeR 4.8.2,
generics 0.1.4, GenomicRanges 1.62.1, ggplot2 4.0.3, IRanges 2.44.0,
limma 3.66.0, MatrixGenerics 1.22.0, matrixStats 1.5.0, S4Vectors
0.48.1, Seqinfo 1.0.0, SummarizedExperiment 1.40.0

Why proportions cannot be independent

In every sample the proportions add up to 1. A quantity that never varies has zero covariance with anything, so for any part $i$ the covariance of $p_i$ with the total is zero, and splitting the total into its parts gives

$$\sum_{j \neq i} \operatorname{cov}(p_i, p_j) = -\operatorname{var}(p_i).$$

Each part’s covariances with the others must add up to minus its own variance. That is an identity, not an estimate: if one part goes up, the others must, taken together, go down by the same amount. So the correlations of proportions cannot all be zero even when the underlying abundances have nothing to do with each other. Data with this property are called compositional, and dividing by the total is called closure.

Sequencing makes almost every abundance table compositional. The number of reads per sample is set by the sequencer, not by how many cells were in the tube, so only the relative amounts carry information. Gloor et al. (2017) make this case for microbiome data. The same applies to cell-type proportions from single-cell data, where the total is the number of cells captured.

The setup

clr <- function(x) log(x) - rowMeans(log(x))   # rows = samples, columns = parts
rho <- function(z) {                            # z: CLR-transformed matrix
  v <- apply(z, 2, var)
  2 * cov(z) / outer(v, v, "+")                 # = 1 - var(z_i - z_j) / (var z_i + var z_j)
}
lr_var <- function(x) {                         # variance of log(x_i / x_j), all pairs
  l <- log(x); v <- apply(l, 2, var)
  outer(v, v, "+") - 2 * cov(l)
}
pairs_of <- function(m) m[upper.tri(m)]         # one value per pair
cor_p <- function(r, n) {                       # two-sided p-value of the usual t test
  2 * pt(-abs(r * sqrt((n - 2) / (1 - r^2))), n - 2)
}

Each simulated data set has 50 samples and 30 taxa. The absolute abundance of every taxon in every sample is drawn independently from a log-normal distribution with a standard deviation of 1 on the natural log scale, so the true correlation between any two taxa is zero. The taxa differ in typical size along a steep curve (a Zipf curve with exponent 1.5, so the typical size of the $k$-th largest taxon falls as $k$ to the power -1.5), and on average the largest one holds 37% of each sample. Each sample is then sequenced: a number of reads between 5,000 and 50,000 is drawn from the true proportions. Only 0.2% of the resulting counts are zero.

n_samples <- 50; n_taxa <- 30; sd_log <- 1; zipf <- 1.5
depth_range <- c(5000, 50000); pc_main <- 0.5

zipf_shares <- function(D, a) { s <- seq_len(D)^-a; s / sum(s) }

simulate_community <- function(shares, n = n_samples, depth = depth_range) {
  D <- length(shares)
  # absolute abundances: independent log-normal, one column per taxon
  A <- exp(matrix(rnorm(n * D, mean = rep(log(shares), each = n), sd = sd_log), n, D))
  P <- A / rowSums(A)                                   # true proportions
  reads <- round(exp(runif(n, log(depth[1]), log(depth[2]))))
  Y <- t(vapply(seq_len(n), function(i) rmultinom(1, reads[i], P[i, ])[, 1], numeric(D)))
  list(A = A, P = P, Y = Y, reads = reads)
}

Three assumptions matter. Independence of the absolute abundances is the truth being tested, so any correlation that comes out is spurious. Every taxon has the same variance on the log scale; this makes the arithmetic further down exact, and real taxa vary in variability. And one taxon dominates, which the next sections vary on purpose.

The analysis is repeated on 200 data sets, giving 87,000 pairwise correlations. The $p$-values come from the same $t$ test that cor.test() uses; the code checks the two agree.

n_sets <- 200
shares_main <- zipf_shares(n_taxa, zipf)
is_dom <- matrix(FALSE, n_taxa, n_taxa); is_dom[1, ] <- is_dom[, 1] <- TRUE
set.seed(20260922)
main <- do.call(rbind, lapply(seq_len(n_sets), function(k) {
  s <- simulate_community(shares_main)
  props <- s$Y / rowSums(s$Y)                           # observed relative abundance
  z <- clr(s$Y + pc_main)
  data.frame(set = k, dominant = pairs_of(is_dom),
             absolute = pairs_of(cor(log(s$A))),
             proportion = pairs_of(cor(props)),
             log_proportion = pairs_of(cor(log((s$Y + pc_main) / rowSums(s$Y + pc_main)))),
             clr = pairs_of(cor(z)), rho = pairs_of(rho(z)),
             top_share = mean(props[, 1]), zero_frac = mean(s$Y == 0))
}))
n_pairs <- choose(n_taxa, 2)
p_of <- function(r) cor_p(r, n_samples)
fp_abs  <- mean(p_of(main$absolute) < 0.05)
fp_prop <- mean(p_of(main$proportion) < 0.05)
fp_logp <- mean(p_of(main$log_proportion) < 0.05)
fp_clr  <- mean(p_of(main$clr) < 0.05)
fp_prop_dom <- mean(p_of(main$proportion[main$dominant]) < 0.05)
fp_prop_oth <- mean(p_of(main$proportion[!main$dominant]) < 0.05)
med_dom_prop <- median(main$proportion[main$dominant])
med_oth_prop <- median(main$proportion[!main$dominant])
dom_neg <- mean(main$proportion[main$dominant] < 0)
oth_pos <- mean(main$proportion[!main$dominant] > 0)
med_oth_logp <- median(main$log_proportion[!main$dominant])
mean_abs <- mean(main$absolute); mean_clr <- mean(main$clr); mean_rho <- mean(main$rho)
med_dom_clr <- median(main$clr[main$dominant])
top_share_main <- mean(main$top_share); zero_main <- mean(main$zero_frac)
theory_main <- -1 / (n_taxa - 1)

set.seed(1)
x <- rlnorm(40); y <- 3 * x; w <- rlnorm(40); u <- rlnorm(40)
toy_abs <- cbind(x, y, w, u)                    # y is exactly proportional to x
toy_rel <- toy_abs / rowSums(toy_abs)           # the same data as proportions
z <- clr(toy_rel)
rho_def <- 1 - var(z[, "x"] - z[, "w"]) / (var(z[, "x"]) + var(z[, "w"]))
stopifnot(
  isTRUE(all.equal(rho(clr(toy_abs))["x", "y"], 1)),     # proportional pair: rho = 1 ...
  isTRUE(all.equal(rho(z)["x", "y"], 1)),                # ... before and after closure
  isTRUE(all.equal(rho(z)["x", "w"], rho_def)),          # matrix form = definition
  isTRUE(all.equal(lr_var(toy_abs), lr_var(toy_rel))),   # log-ratio variance ignores closure
  isTRUE(all.equal(cor_p(cor(toy_rel[, "x"], toy_rel[, "w"]), 40),
                   cor.test(toy_rel[, "x"], toy_rel[, "w"])$p.value)))

# a straight line on the log scale that is not proportionality: y = x^2, plus 9 other parts
set.seed(2)
x2 <- rlnorm(50)
toy_sq <- cbind(x = x2, y = x2^2, matrix(rlnorm(50 * 9), 50, 9))
z_sq <- clr(toy_sq)
cor_sq <- cor(z_sq[, "x"], z_sq[, "y"]); rho_sq <- rho(z_sq)["x", "y"]
stopifnot(rho_sq < cor_sq)

Correlations of proportions under independence

lab <- c(absolute = "log absolute abundance", proportion = "proportion", clr = "CLR")
hist_df <- do.call(rbind, lapply(names(lab), function(m)
  data.frame(method = factor(unname(lab[m]), levels = lab), r = main[[m]],
             pair = ifelse(main$dominant, "with the dominant taxon", "other pairs"))))
ggplot(hist_df, aes(r, after_stat(density), fill = pair)) +
  geom_histogram(binwidth = 0.05, boundary = 0, position = "identity", alpha = 0.6) +
  geom_vline(xintercept = 0, colour = grey) +
  facet_wrap(~ method) +
  scale_fill_manual(values = c("with the dominant taxon" = magenta, "other pairs" = violet),
                    name = NULL) +
  labs(x = "Pearson correlation across samples", y = "density") +
  theme_pc

The left panel is the truth: correlations of the log absolute abundances scatter around zero (mean -0.001), and 5.2% of pairs reach $p < 0.05$, as they should. The middle panel is what cor() on the proportions reports for the same samples. It is not one distribution but two. The pairs that include the dominant taxon are pushed down, to a median of -0.22, and 31% of them are “significant”. The other pairs are pushed slightly up, to a median of 0.05, and 9.7% of them pass $p < 0.05$. A network built from these correlations would put the dominant taxon at the centre of a star of negative edges, which looks like competition or exclusion and is purely arithmetic.

Spurious correlations of proportions are not always negative: here 63% of the pairs without the dominant taxon are positive. The identity only fixes the sum of each part’s covariances, and for a minor taxon the covariance with the dominant one more than covers it, leaving room for positive covariances with the other minor taxa.

A shared denominator

The positive side has an old explanation. The proportion of taxon $i$ is its abundance divided by the total, and the total is the same for every taxon in that sample. Two ratios with a common denominator are correlated even when their numerators are independent: when the denominator happens to be large, both ratios are small. Pearson (1897) described this “spurious correlation” for indices used in measuring organs, where one measurement is divided by another. When one taxon dominates, a large part of the total is that taxon, so every other proportion is roughly an independent abundance divided by the dominant one. They rise and fall together, and the dominant taxon moves against them.

Logs do not help, because $\log p_i = \log a_i - \log T$ still contains the shared total $T$. With log proportions (a pseudocount of 0.5 for the few zeros) the other pairs have a median correlation of 0.15 and 19.1% of all pairs pass $p < 0.05$.

zipf_grid <- c(0, 0.5, 1, 1.5, 2, 2.5)
n_sets_sweep <- 100
set.seed(20260923)
sweep_dom <- do.call(rbind, lapply(zipf_grid, function(a) {
  sh <- zipf_shares(n_taxa, a)
  r <- replicate(n_sets_sweep, {
    s <- simulate_community(sh)
    props <- s$Y / rowSums(s$Y); cp <- cor(props); cz <- cor(clr(s$Y + pc_main))
    c(top = mean(props[, 1]),
      prop_dom = mean(cp[1, -1]), prop_oth = mean(pairs_of(cp[-1, -1])),
      clr_dom = mean(cz[1, -1]), clr_oth = mean(pairs_of(cz[-1, -1])))
  })
  data.frame(zipf = a, t(rowMeans(r)))
}))
even <- sweep_dom[sweep_dom$zipf == 0, ]
steep <- sweep_dom[sweep_dom$zipf == max(zipf_grid), ]
clr_range <- range(unlist(sweep_dom[, c("clr_dom", "clr_oth")]))

The strength of the effect depends on how dominant the largest taxon is. The next simulation keeps 30 independent taxa and makes the abundance curve steeper step by step, with 100 data sets per step.

dom_long <- rbind(
  data.frame(top = sweep_dom$top, r = sweep_dom$prop_dom, method = "proportion", pair = "with the dominant taxon"),
  data.frame(top = sweep_dom$top, r = sweep_dom$prop_oth, method = "proportion", pair = "other pairs"),
  data.frame(top = sweep_dom$top, r = sweep_dom$clr_dom, method = "CLR", pair = "with the dominant taxon"),
  data.frame(top = sweep_dom$top, r = sweep_dom$clr_oth, method = "CLR", pair = "other pairs"))
dom_long$method <- factor(dom_long$method, levels = c("proportion", "CLR"))
dom_long$pair <- factor(dom_long$pair, levels = c("with the dominant taxon", "other pairs"))
ggplot(dom_long, aes(100 * top, r, colour = method, linetype = pair)) +
  geom_hline(yintercept = 0, colour = grey) +
  geom_line(linewidth = 0.8) + geom_point(size = 2) +
  scale_colour_manual(values = c(proportion = magenta, CLR = violet), name = NULL) +
  scale_linetype_manual(values = c("with the dominant taxon" = "solid", "other pairs" = "dashed"),
                        name = NULL) +
  labs(x = "Average share of the most abundant taxon (%)", y = "Mean correlation") +
  theme_pc + theme(legend.box = "vertical", legend.key.width = unit(2.5, "lines"))

With all taxa the same size (each about 3% of a sample), every pair of proportions has a mean correlation near -0.03. At the steepest curve the largest taxon holds 65%, its mean correlation with the others is -0.29, and the mean among the rest has risen to 0.17. The CLR lines barely move (-0.034 to -0.032), for the reason given next.

The centred log-ratio

The standard alternative comes from the log-ratio approach to compositional data (Aitchison 1982). The centred log-ratio (CLR) of a part is its log abundance minus the mean log abundance of all parts in that sample, which is the log of its ratio to the geometric mean. Two things follow. Dividing a sample by its total subtracts the same constant from every log, and the centring removes it, so the CLR values of proportions and of absolute abundances are identical. And every part contributes equally to the reference, whatever its size, so no single taxon can control it.

In the main simulation the CLR correlations have a mean of -0.034, the pairs with the dominant taxon have a median of -0.031, and 5.7% of all pairs reach $p < 0.05$. That is close to the 5.2% of the absolute abundances, and it is the right-hand panel of the first figure: the two groups of pairs now overlap.

The CLR needs the log of every count, and zeros have no log. The usual fix is to add a pseudocount, here 0.5, which has side effects of its own (see the post on pseudocounts and fold changes). In the main simulation zeros are rare, so the choice hardly matters. To see when it does, the next check uses a steeper abundance curve and shallower sequencing, and looks at the rarest taxa.

n_sets_zero <- 100; shallow <- c(500, 10000); pc_grid <- c(0.1, 0.5, 1)
rare <- 21:30
set.seed(20260924)
zero_runs <- replicate(n_sets_zero, {
  s <- simulate_community(zipf_shares(n_taxa, 2), depth = shallow)
  c(zero = mean(s$Y[, rare] == 0),
    truth = mean(pairs_of(cor(clr(s$A))[rare, rare])),
    sapply(pc_grid, function(pc) mean(pairs_of(cor(clr(s$Y + pc))[rare, rare]))),
    sapply(pc_grid, function(pc) mean(cor(log(s$reads), clr(s$Y + pc)[, rare]))))
})
zero_res <- rowMeans(zero_runs)
zero_rare <- zero_res[[1]]; zero_truth <- zero_res[[2]]
zero_pc <- zero_res[3:5]; zero_depth <- zero_res[6:8]   # per pseudocount

With 500 to 10,000 reads per sample, the 10 rarest taxa are zero in 30% of samples. On the absolute abundances, the CLR correlation among them averages -0.035. From the counts it averages -0.039 with a pseudocount of 0.1, -0.013 with 0.5 and 0.025 with 1. A pseudocount turns every zero into the same value, and zeros are more common in shallow samples, so the larger the pseudocount, the more the rare taxa share a dependence on sequencing depth. Their correlation with log depth averages -0.006, -0.125 and -0.204 for the three pseudocounts. The shift is small here, but the choice changes the answer.

What the CLR cannot fix

The CLR values of a sample also add up to zero, so the identity from the start applies to them too. With independent parts of equal log-scale variance it gives an exact result: the correlation between any two CLR values is $-1/(D - 1)$ for $D$ parts. With 30 taxa that is -0.034, which is the mean found above. With many parts this is close enough to zero to ignore; with few parts it is not.

parts_grid <- c(3, 4, 5, 6, 8, 12, 20, 30, 50)
set.seed(20260925)
sweep_parts <- do.call(rbind, lapply(parts_grid, function(D) {
  r <- replicate(n_sets_sweep, {
    s <- simulate_community(rep(1 / D, D))
    props <- s$Y / rowSums(s$Y); z <- clr(s$Y + pc_main)
    cz <- pairs_of(cor(z))
    c(proportion = mean(pairs_of(cor(props))), clr = mean(cz),
      rho = mean(pairs_of(rho(z))), fp_clr = mean(p_of(cz) < 0.05))
  })
  data.frame(parts = D, t(rowMeans(r)), theory = -1 / (D - 1))
}))
sp <- function(D) sweep_parts[sweep_parts$parts == D, ]

parts_long <- rbind(
  data.frame(parts = sweep_parts$parts, r = sweep_parts$proportion, method = "proportion"),
  data.frame(parts = sweep_parts$parts, r = sweep_parts$clr, method = "CLR"),
  data.frame(parts = sweep_parts$parts, r = sweep_parts$rho, method = "rho"))
parts_long$method <- factor(parts_long$method, levels = c("proportion", "CLR", "rho"))
ggplot(parts_long, aes(parts, r, colour = method, shape = method)) +
  geom_hline(yintercept = 0, colour = grey) +
  geom_line(data = sweep_parts, aes(parts, theory), inherit.aes = FALSE,
            linetype = "dashed", colour = "black") +
  geom_point(aes(size = method), stroke = 1) +
  scale_x_log10(breaks = parts_grid) +
  scale_size_manual(values = c(proportion = 4.5, CLR = 3, rho = 1.4), name = NULL) +
  scale_colour_manual(values = c(proportion = magenta, CLR = violet, rho = grey), name = NULL) +
  scale_shape_manual(values = c(proportion = 1, CLR = 2, rho = 15), name = NULL) +
  labs(x = "Number of parts (log scale)", y = "Mean correlation") +
  theme_pc

The simulation uses evenly sized parts, so there is no dominant one, and all three measures land on the curve. A single-cell study that tracks 5 cell types gets a mean CLR correlation of -0.25 between cell types that are independent, and 43% of pairs pass $p < 0.05$ with 50 samples. With 8 parts it is still 16%; with 3 parts, 96%. Only with a few dozen parts does the rate come down to about 5% (5.5% at 30). These rates depend on the number of samples: with fewer donors fewer pairs reach significance, but the mean correlation stays where it is.

No transformation of the proportions removes this, because the information is not in the data. If five cell types always make up the whole, an increase in one must be matched by a decrease somewhere else, and nothing computed from the shares can say whether that was a real loss of cells or only a smaller slice of the same total.

Proportionality

The CLR fixes the dominant-part problem but still gives a correlation, and correlation is the wrong question for relative data. Lovell et al. (2015) propose asking instead whether two parts are proportional: whether their ratio stays constant across samples. The natural measure is the variance of the log-ratio, $\operatorname{var}(\log(x/y))$. Because $\log(p_x / p_y) = \log(a_x / a_y)$, the total cancels, and this variance is the same whether you compute it from absolute abundances, from proportions, or from any subset of the parts.

A log-ratio variance on its own has no fixed scale, so it is usually rescaled. Lovell et al. divide it by the variance of one of the two parts and call the result phi. The symmetric version used here, $\rho$ (rho), is the one implemented in the propr R package, computed on CLR values by default (Quinn et al. 2017). It divides the log-ratio variance by the sum of the two CLR variances and subtracts the result from 1:

$$\rho(x, y) = 1 - \operatorname{var}(\operatorname{clr} x - \operatorname{clr} y) \,/\, [\operatorname{var}(\operatorname{clr} x) + \operatorname{var}(\operatorname{clr} y)]$$

which is the same as

$$\rho(x, y) = 2 \operatorname{cov}(\operatorname{clr} x, \operatorname{clr} y) \,/\, [\operatorname{var}(\operatorname{clr} x) + \operatorname{var}(\operatorname{clr} y)].$$

It is 1 when the two parts are exactly proportional, 0 when the log-ratio varies as much as the two parts together, and it is Lin’s concordance correlation coefficient without the term for a difference in means, so a constant ratio other than 1 still scores 1. The rho() function in the setup code implements it; the check after the main simulation confirms that a pair with $y = 3x$ gets exactly 1 before and after closure and that the matrix form matches the definition.

Under independence rho behaves like the CLR correlation: a mean of -0.034 in the main simulation, and the same $-1/(D - 1)$ curve in the last figure. So rho is not a test of independence, and it does not rescue a five-part composition. What it adds over a CLR correlation is a narrower question. A correlation is 1 for any straight-line relation between two CLR columns; rho is 1 only when the slope is 1, which on the log scale means the two parts are proportional. In a second toy check, one part is the square of another ($y = x^2$) among 9 other parts: their CLR correlation is 0.98 and their rho is 0.72. Two taxa that rise together, one much more steeply than the other, get a high correlation and a lower rho.

Changing the set of parts

A correlation of proportions also depends on which other parts were measured. The next check scores the same pairs of the rarest half of the taxa twice: once in the full community, and once after the abundant taxa are dropped and the rest re-closed to 100%, as happens when a study reports only one phylum or excludes a contaminant.

keep <- 16:30                           # the 15 least abundant taxa
n_sets_sub <- 50
set.seed(20260926)
sub <- do.call(rbind, lapply(seq_len(n_sets_sub), function(k) {
  s <- simulate_community(shares_main)
  full <- s$Y + pc_main
  pf <- full / rowSums(full)                              # full community
  pp <- full[, keep] / rowSums(full[, keep])              # subcomposition, re-closed
  data.frame(set = k,
             cor_full = pairs_of(cor(pf)[keep, keep]), cor_sub = pairs_of(cor(pp)),
             clr_full = pairs_of(cor(clr(pf))[keep, keep]), clr_sub = pairs_of(cor(clr(pp))),
             rho_full = pairs_of(rho(clr(pf))[keep, keep]), rho_sub = pairs_of(rho(clr(pp))),
             lrv_diff = max(abs(pairs_of(lr_var(pf)[keep, keep]) - pairs_of(lr_var(pp)))))
}))
cor_shift <- median(abs(sub$cor_full - sub$cor_sub))
clr_shift <- median(abs(sub$clr_full - sub$clr_sub))
rho_shift <- median(abs(sub$rho_full - sub$rho_sub))
lrv_shift <- max(sub$lrv_diff)
mean_cor_full <- mean(sub$cor_full); mean_cor_sub <- mean(sub$cor_sub)
mean_clr_full <- mean(sub$clr_full); mean_clr_sub <- mean(sub$clr_sub)
mean_rho_full <- mean(sub$rho_full); mean_rho_sub <- mean(sub$rho_sub)
theory_sub <- -1 / (length(keep) - 1)

sub_long <- rbind(
  data.frame(full = sub$cor_full, part = sub$cor_sub, score = "Pearson on proportions"),
  data.frame(full = sub$clr_full, part = sub$clr_sub, score = "Pearson on CLR values"),
  data.frame(full = sub$rho_full, part = sub$rho_sub, score = "rho on CLR values"))
sub_long$score <- factor(sub_long$score, levels = unique(sub_long$score))
ggplot(sub_long, aes(full, part)) +
  geom_abline(slope = 1, intercept = 0, colour = grey) +
  geom_point(size = 0.6, alpha = 0.3, colour = violet) +
  facet_wrap(~ score) + coord_equal() +
  labs(x = "score in the full community", y = "score in the subcomposition") +
  theme_pc

The correlation of proportions depends heavily on what else was measured. For the same pairs of taxa it averages 0.09 in the full community and -0.06 in the subcomposition, and the typical pair moves by 0.15 (median absolute change). Moving to log-ratios removes most of that. The CLR correlation moves by 0.04 and rho by 0.04; their averages go from -0.034 to -0.071 and from -0.033 to -0.070, the $-1/(D - 1)$ of 15 parts being -0.071. The two move the same small amount because both are scaled by CLR variances, and a CLR value depends on the geometric mean of whatever parts are present. The log-ratio variance itself does not move at all: the largest difference is 3e-15, which is floating-point rounding. If a result has to survive a change in which taxa were measured, the log-ratio variance of the pair is the quantity to report.

What to do in practice

Do not run cor() on proportions, percentages or their logs, and do not build co-occurrence networks from them. One dominant part against everything and the rest mildly positive looks like biology and is arithmetic.

With many parts, compute on CLR values. A pseudocount is needed only because of the zeros; say which one you used, and check that your conclusions hold with a second one. With CLR correlations or rho, report which parts went into the reference, because the numbers depend on it. For finding parts that move together, rho answers a question that relative data can actually support. It has no built-in test, and the propr paper picks its highly proportional pairs with an arbitrary cutoff. A cutoff can instead be calibrated by shuffling each taxon’s counts across samples, which destroys every pairing, and recording how high rho gets by chance:

x <- counts + 0.5                            # samples in rows, taxa in columns
z <- log(x) - rowMeans(log(x))               # centred log-ratio
v <- apply(z, 2, var)
rho <- 2 * cov(z) / outer(v, v, "+")         # proportionality
lr_var <- outer(v, v, "+") - 2 * cov(z)      # var(log(x_i / x_j)) for every pair

# permutation null for rho: shuffle each taxon independently, recompute
null_rho <- replicate(100, {
  xp <- apply(x, 2, sample)
  zp <- log(xp) - rowMeans(log(xp)); vp <- apply(zp, 2, var)
  r <- 2 * cov(zp) / outer(vp, vp, "+"); r[upper.tri(r)]
})
quantile(null_rho, 0.99)                     # rho this high arises by chance in 1% of pairs

With few parts, such as a handful of cell types, accept that the proportions are dependent and ask a question in ratio form instead: does the ratio of B cells to T cells change with treatment? That ratio is a single number per sample and can be tested like any other. If the question is really about absolute numbers (did this cell type expand, or did the others shrink?), it needs a measurement of the total: cells per microlitre from a counter or flow cytometry, a spike-in, or total bacterial load by qPCR. RNA-seq has the same problem when most genes change (see the post on CPM and TMM), and the same answer: the reference has to come from outside the shares.

References

• Pearson K 1897 Proceedings of the Royal Society of London 60:489-498 (doi:10.1098/rspl.1896.0076)
• Aitchison J 1982 Journal of the Royal Statistical Society Series B 44(2):139-160 (doi:10.1111/j.2517-6161.1982.tb01195.x)
• Lovell D, Pawlowsky-Glahn V, Egozcue JJ et al. 2015 PLOS Computational Biology 11(3):e1004075 (doi:10.1371/journal.pcbi.1004075)
• Gloor GB, Macklaim JM, Pawlowsky-Glahn V et al. 2017 Frontiers in Microbiology 8:2224 (doi:10.3389/fmicb.2017.02224)
• Quinn TP, Richardson MF, Lovell D et al. 2017 Scientific Reports 7:16252 (doi:10.1038/s41598-017-16520-0)

R version 4.5.3 (2026-03-11)
Bioconductor 3.22
ggplot2 4.0.3

The setup

simulate_cells <- function(n_cells = 2000, n_genes = 1000, phi = 0, sf_sd = 0.3,
                           n_sub = 0, n_marker = 0, fold = 1) {
  mu <- exp(rnorm(n_genes, 0, 1))                   # mean UMI per cell, per gene
  sf <- exp(rnorm(n_cells, 0, sf_sd))               # capture efficiency, per cell
  m <- outer(mu, sf)
  sub <- seq_len(n_cells) <= n_sub                  # optional real subpopulation
  m[seq_len(n_marker), sub] <- fold * m[seq_len(n_marker), sub]
  x <- if (phi == 0) rpois(length(m), m) else rnbinom(length(m), mu = m, size = 1 / phi)
  matrix(x, n_genes, dimnames = list(sprintf("g%04d", seq_len(n_genes)),
                                     sprintf("c%04d", seq_len(n_cells))))
}

set.seed(1)
counts <- simulate_cells()
n_genes <- nrow(counts); n_cells <- ncol(counts)
sf_sd <- formals(simulate_cells)$sf_sd
share_low <- mean(rowMeans(counts) < 1)

The simulated data set has 2,000 cells and 1,000 genes. Each gene has a baseline level (48% of genes average less than one UMI per cell; a UMI, or unique molecular identifier, is one captured mRNA molecule). Each cell has a capture efficiency that scales all its counts up or down, with a standard deviation of 0.3 on the natural log scale; this is what makes cells differ in sequencing depth. Counts are Poisson around the product of the two. Every cell comes from the same distribution, so any gene called a marker is a false positive. The Poisson noise is the assumption that matters most, and a later section replaces it with negative binomial noise.

The common way: cluster, then test the clusters

as_sce <- function(counts) logNormCounts(SingleCellExperiment(list(counts = counts)))

cluster_pipeline <- function(counts, n_hvg = 500, n_pc = 10) {
  sce <- as_sce(counts)
  hvg <- getTopHVGs(modelGeneVar(sce), n = n_hvg)
  sce <- fixedPCA(sce, rank = n_pc, subset.row = hvg)
  metadata(sce)$hvg <- hvg
  sce$cluster <- clusterCells(sce, use.dimred = "PCA")   # SNN graph + walktrap
  sce
}

# each cluster against all other cells: a Wilcoxon rank-sum p-value per gene
markers_vs_rest <- function(sce, cluster) {
  sapply(levels(cluster), function(k) {
    in_k <- factor(ifelse(cluster == k, "in", "out"))
    findMarkers(sce, groups = in_k, test.type = "wilcox")$`in`[rownames(sce), "p.value"]
  })
}

# per cluster: how many genes pass a Benjamini-Hochberg FDR of alpha
n_markers <- function(P) colSums(apply(P, 2, p.adjust, method = "BH") < alpha)

sce <- cluster_pipeline(counts)
p_naive <- markers_vs_rest(sce, sce$cluster)

p_nn <- NNGraphParam()                               # what clusterCells() uses by default
c(class = class(p_nn)[1], k = p_nn@k, type = p_nn@type, cluster.fun = p_nn@cluster.fun)

          class               k            type     cluster.fun 
"SNNGraphParam"            "10"          "rank"      "walktrap" 

# the default test: findMarkers() without test.type gives the same p-values as test.type = "t"
g2 <- factor(ifelse(sce$cluster == levels(sce$cluster)[1], "in", "out"))
same_as_t <- identical(findMarkers(sce[1:20, ], groups = g2)$`in`$p.value,
                       findMarkers(sce[1:20, ], groups = g2, test.type = "t")$`in`$p.value)
c(default_is_t = same_as_t)

default_is_t 
        TRUE 

The pipeline is the standard Bioconductor one: log-normalise with logNormCounts(), keep the most variable genes, reduce them to 10 principal components, and cluster with clusterCells(), whose default builds a shared nearest-neighbour graph from the 10 nearest neighbours of each cell and splits it with the walktrap algorithm (output above); Seurat and scanpy do the same with Louvain or Leiden. For markers, findMarkers() compares each cluster with all other cells using test.type = "wilcox", a two-sided Wilcoxon rank-sum test (its default is a t-test, as the check shows; Seurat’s FindMarkers() defaults to the Wilcoxon test), with Benjamini-Hochberg adjustment within each cluster.

The clustering found 10 clusters in data that has none. Figure 1 shows them on the first two principal components: one continuous cloud of cells, cut into pieces.

pcs <- reducedDim(sce, "PCA")
dpc <- data.frame(PC1 = pcs[, 1], PC2 = pcs[, 2], cluster = sce$cluster)
ggplot(dpc, aes(PC1, PC2, colour = cluster)) +
  geom_point(size = 0.7, alpha = 0.7) +
  scale_colour_manual(values = hcl.colors(nlevels(dpc$cluster), "Dark 3")) +
  guides(colour = guide_legend(override.aes = list(size = 3, alpha = 1))) +
  labs(colour = "cluster")

All 10 clusters got marker genes at an FDR of 5%, between 6 and 49 per cluster, with a smallest p-value of 1e-19. Across every gene and every cluster, 12% of the raw p-values fell below 0.05, where a valid test on null data gives 5%.

Why it happens

A clustering algorithm has one job: find groups of cells that differ in expression. In data without structure, noise still makes some cells a little higher in some genes and a little lower in others, and the algorithm puts its boundaries where those chance differences line up best. The marker test then asks whether the groups differ. They do, because that is how the groups were chosen.

The p-value of a Wilcoxon test (or a t-test) assumes the groups were fixed before anyone looked at the data. Here they were chosen from the same counts that are then tested: the data are used once to pick the hypothesis and again to test it. Zhang, Kamath and Tse (2019) describe this for single-cell clustering: because clustering forces the groups apart, reusing the data gives artificially low p-values. They also propose a corrected test.

The inflation sits where the clustering looked. Among the 352 variable genes that went into the principal components (getTopHVGs() was asked for up to 500 but by default keeps only genes above the fitted mean-variance trend), 21% of the p-values fell below 0.05; among the other genes, 7%.

The clustering also picked up something real, if uninteresting: sequencing depth. The first principal component correlates with the log of each cell’s total count (correlation 0.68), and the clusters account for 33% of the variance in log total count. Log-normalisation does not remove depth completely (a cell with fewer counts has more zeros), so clusters that differ in depth also differ a little in the normalised values of every gene, which is the likely reason the genes the clustering never used came out at 7% rather than 5%. This matters again below.

The tempting fix: split the cells

# assign held-out cells to the nearest cluster centre in the training PCA space
assign_heldout <- function(train_sce, test_sce) {
  hvg <- metadata(train_sce)$hvg
  rot <- attr(reducedDim(train_sce, "PCA"), "rotation")[hvg, ]
  centre <- rowMeans(as.matrix(logcounts(train_sce)[hvg, ]))
  z <- t(as.matrix(logcounts(test_sce)[hvg, ]) - centre) %*% rot
  cent <- rowsum(reducedDim(train_sce, "PCA"), train_sce$cluster) /
    as.vector(table(train_sce$cluster))
  d2 <- outer(rowSums(z^2), rowSums(cent^2), "+") - 2 * z %*% t(cent)
  factor(rownames(cent)[max.col(-d2)], levels = rownames(cent))
}

set.seed(2)
half <- sample(n_cells, n_cells / 2)
sce_a <- cluster_pipeline(counts[, half])            # cluster on half A
sce_b <- as_sce(counts[, -half])                     # test on half B
cl_b <- droplevels(assign_heldout(sce_a, sce_b))
p_cells <- markers_vs_rest(sce_b, cl_b)

The textbook answer is to use different data for the two steps: cluster half of the cells, test in the other half. But the held-out cells have no cluster labels, and the only way to give them one is from their own expression (above: project each onto the training principal components and pick the nearest cluster centre, the idea behind label transfer).

That puts the circularity back. Half A gave 7 clusters; in half B, 7 of them got markers, 6.7 per cluster on average, with a smallest p-value of 2e-09, and 9% of the raw p-values were below 0.05. Half as many cells means less power, but the share of small p-values stays well above 5%, because each held-out cell was sorted into a cluster by the genes that are then tested. Neufeld et al. (2024) make the same point: sample splitting, the usual remedy elsewhere, does not apply here.

Count splitting, and the depth both halves share

count_split <- function(counts, eps = 0.5, phi = NULL) {
  p <- eps                                           # Poisson: binomial thinning
  if (!is.null(phi)) {                               # negative binomial: beta-binomial
    b <- rep(1 / pmax(phi, 1e-8), ncol(counts))
    p <- rbeta(length(counts), eps * b, (1 - eps) * b)
  }
  train <- matrix(rbinom(length(counts), counts, p), nrow(counts),
                  dimnames = dimnames(counts))
  list(train = train, test = counts - train)
}

# negative binomial GLM in edgeR; the offset is each cell's log total count (no TMM)
nb_vs_rest <- function(counts, cluster, clusters = levels(cluster)) {
  design <- model.matrix(~ 0 + cluster)
  fit <- glmQLFit(DGEList(counts), design, legacy = FALSE)
  K <- nlevels(cluster)
  sapply(clusters, function(k) {                     # cluster k against the mean of the others
    con <- ifelse(levels(cluster) == k, 1, -1 / (K - 1))
    glmQLFTest(fit, contrast = con)$table$PValue
  })
}

set.seed(3)
cs <- count_split(counts)
sce_train <- cluster_pipeline(cs$train)              # cluster on the training counts
sce_test <- as_sce(cs$test)                          # test on the held-out counts
p_split <- markers_vs_rest(sce_test, sce_train$cluster)
p_split_nb <- nb_vs_rest(cs$test, sce_train$cluster)

Count splitting (Neufeld et al. 2024) splits the counts instead of the cells. Every UMI is sent to a training copy or a test copy by a coin flip: for a count $x$, the training count is binomial with $x$ trials and probability 0.5, and the rest goes to the test copy. If the original count is Poisson, the two copies are independent Poisson counts. Both copies contain every cell, so the clusters found on the training copy label the same cells in the test copy, and the test copy had no say in where the boundaries went. The whole pipeline runs on the training counts; only the marker test uses the test counts.

Independence covers the noise, not what the copies share: both copies of a cell inherit its capture efficiency, so clusters that follow depth in the training copy differ in depth in the test copy too, and a Wilcoxon test on log-normalised values can see that. A test that models the counts with each cell’s total count as an offset does not: nb_vs_rest() above fits edgeR’s negative binomial quasi-likelihood model (with legacy = FALSE, glmQLFit() estimates the negative binomial dispersion itself, so no estimateDisp() step is needed) and compares each cluster with the average of the others.

clusterings <- list("walktrap, k = 10" = NNGraphParam(k = 10),
                    "Louvain, k = 20"  = NNGraphParam(k = 20, cluster.fun = "louvain"))
wide_sd <- 0.5                                       # a wider spread of sequencing depth
r2 <- function(y, cl) summary(lm(y ~ cl))$r.squared
set.seed(6)
runs <- list(); per_cluster <- list()
for (job in list(c(sf_sd, 1), c(sf_sd, 2), c(wide_sd, 1))) {
  d <- job[1]; s <- job[2]
  if (d == sf_sd && s == 1) { cs_s <- cs; tr <- sce_train } else {
    cs_s <- count_split(simulate_cells(sf_sd = d))
    tr <- cluster_pipeline(cs_s$train)
  }
  te <- as_sce(cs_s$test)
  for (j in seq_along(clusterings)) {
    cl <- if (j == 1) tr$cluster else
      clusterCells(tr, use.dimred = "PCA", BLUSPARAM = clusterings[[j]])
    main <- d == sf_sd && s == 1 && j == 1
    p_w <- if (main) p_split else markers_vs_rest(te, cl)
    run_nb <- j == 1 && s == 1                       # edgeR is slow: first clustering only
    p_e <- if (main) p_split_nb else if (run_nb) nb_vs_rest(cs_s$test, cl) else NULL
    runs[[length(runs) + 1]] <- data.frame(
      depth_sd = d, data_set = s, clustering = names(clusterings)[j], clusters = nlevels(cl),
      depth_R2 = round(r2(log(colSums(cs_s$test)), cl), 2),
      wilcox_markers = sum(n_markers(p_w)), wilcox_p_below = round(mean(p_w < alpha), 3),
      edgeR_markers = if (run_nb) sum(n_markers(p_e)) else -1,
      edgeR_p_below = if (run_nb) round(mean(p_e < alpha), 3) else -1)
    per_cluster[[length(per_cluster) + 1]] <- rbind(
      data.frame(depth_sd = d, test = "Wilcoxon", n = n_markers(p_w)),
      if (run_nb) data.frame(depth_sd = d, test = "edgeR", n = n_markers(p_e)))
  }
}
runs <- do.call(rbind, runs); per_cluster <- do.call(rbind, per_cluster)
lo <- runs[runs$depth_sd == sf_sd, ]; hi <- runs[runs$depth_sd == wide_sd, ]
nb_runs <- runs[runs$edgeR_markers >= 0, ]

shown <- runs
shown$edgeR_markers <- ifelse(runs$edgeR_markers < 0, "not run", runs$edgeR_markers)
shown$edgeR_p_below <- ifelse(runs$edgeR_p_below < 0, "not run", runs$edgeR_p_below)
knitr::kable(shown)

The table repeats the count split on 2 data sets with the depth spread used so far (0.3) and one with a wider spread (0.5), each training copy clustered in 2 ways; depth_R2 is the share of variance in the test copy’s log total count that the clusters account for. At the narrower spread the clusters accounted for 20% to 22% of it, and the Wilcoxon test gave 11 to 39 marker genes per clustering, with 8.2% to 9.7% of p-values below 0.05. Small counts like these move around with the clustering, the seed and the package versions. At the wider spread they accounted for 51% to 53%, and the Wilcoxon test gave 1,688 to 1,703 marker genes per clustering, with 59% to 62% of p-values below 0.05. The copies share nothing else, so those markers come from depth, not from double dipping. The edgeR model, run on the first clustering of the first data set at each spread, gave 0 marker genes per clustering in the runs shown, with 5.0% to 5.1% of p-values below 0.05. Over other clusterings it will sometimes report a few markers, about as many as Benjamini-Hochberg lets through by chance when nothing is there (a false marker in up to about 5% of clusters), at either depth spread.

lab_p <- c("same cells,\nsame counts", "split cells", "split counts,\nWilcoxon",
           "split counts,\nedgeR")
dp <- rbind(data.frame(approach = lab_p[1], p = as.vector(p_naive)),
            data.frame(approach = lab_p[2], p = as.vector(p_cells)),
            data.frame(approach = lab_p[3], p = as.vector(p_split)),
            data.frame(approach = lab_p[4], p = as.vector(p_split_nb)))
dp$approach <- factor(dp$approach, levels = lab_p)
ggplot(dp, aes(p)) +
  geom_histogram(aes(y = after_stat(density)), breaks = seq(0, 1, 0.05),
                 fill = violet, colour = "white", linewidth = 0.2) +
  geom_hline(yintercept = 1, linetype = "dashed", colour = grey) +
  facet_wrap(~ approach, nrow = 1) +
  scale_x_continuous(breaks = c(0, 0.5, 1)) +
  labs(x = "p-value (every gene, every cluster against the rest)", y = "density")

Count splitting does not stop the algorithm from finding clusters in noise (the training copy was still cut into 8 pieces); it makes the tested noise different from the noise that drew the boundaries. In real data, batch, cell cycle and anything else that differs between cells sits in both copies, like depth here, and belongs in the test’s model too.

The assumption that matters: Poisson noise

# method-of-moments negative binomial overdispersion, one value per gene
nb_dispersion <- function(counts) {
  sf <- colSums(counts) / mean(colSums(counts))
  m <- outer(rowSums(counts) / sum(sf), sf)
  pmax(rowSums((counts - m)^2 - m) / rowSums(m^2), 0)
}

phi_nb <- 0.5
set.seed(4)
counts_nb <- simulate_cells(phi = phi_nb)
phi_hat <- nb_dispersion(counts_nb)

cs_pois <- count_split(counts_nb)                     # assumes Poisson
tr_pois <- cluster_pipeline(cs_pois$train)
p_nb_pois <- markers_vs_rest(as_sce(cs_pois$test), tr_pois$cluster)

cs_nb <- count_split(counts_nb, phi = phi_hat)        # uses the estimated overdispersion
tr_nb <- cluster_pipeline(cs_nb$train)
p_nb_nb <- markers_vs_rest(as_sce(cs_nb$test), tr_nb$cluster)

Independence of the two copies holds for Poisson counts. Real UMI counts are often more variable (overdispersed, usually modelled as negative binomial). Then a cell with a burst of one gene has a high count in both copies, the copies are positively correlated, and double dipping comes back through those genes. The next simulation gives every gene negative binomial noise with overdispersion 0.5 (variance = mean + 0.5 x mean squared) and still no clusters. After a Poisson split, 7 of the 8 training clusters got markers in the test copy, 22 in total, with a smallest p-value of 1e-31 (against 1e-19 for the same-data analysis on Poisson data). Only 6.4% of all raw p-values were below 0.05, so the damage sits in a few genes with extreme p-values that a histogram can hide.

The fix is a split that matches the noise. Draw the thinning probability for each count from a beta distribution set by the overdispersion, then split binomially with it. Seen as a Poisson count whose mean varies from cell to cell, this divides the varying mean itself into two independent parts, and the copies are independent negative binomial counts again (the phi branch of count_split()). With per-gene overdispersion estimated by the method of moments (median 0.49, true value 0.5), the test copy gave no marker genes across 9 clusters. Both analyses here use the Wilcoxon test to keep the run short; the narrower-spread rows of the table show what depth alone adds at this spread. The estimate must be right in both directions: too small leaves the copies positively correlated, too large makes them negatively correlated.

lab_m <- c("same cells, same counts", "split cells",
           paste("split counts, Wilcoxon, depth sd", fmt_num(sf_sd)),
           paste("split counts, Wilcoxon, depth sd", fmt_num(wide_sd)),
           "split counts, edgeR, both spreads",
           "overdispersed: Poisson split", "overdispersed: NB split")
pc <- function(test, d = c(sf_sd, wide_sd))
  per_cluster$n[per_cluster$test == test & per_cluster$depth_sd %in% d]
dm <- rbind(data.frame(approach = lab_m[1], n = n_markers(p_naive)),
            data.frame(approach = lab_m[2], n = n_markers(p_cells)),
            data.frame(approach = lab_m[3], n = pc("Wilcoxon", sf_sd)),
            data.frame(approach = lab_m[4], n = pc("Wilcoxon", wide_sd)),
            data.frame(approach = lab_m[5], n = pc("edgeR")),
            data.frame(approach = lab_m[6], n = n_markers(p_nb_pois)),
            data.frame(approach = lab_m[7], n = n_markers(p_nb_nb)))
dm$approach <- factor(dm$approach, levels = rev(lab_m))
ggplot(dm, aes(n, approach)) +
  geom_point(position = position_jitter(width = 0, height = 0.15, seed = 1),
             size = 2, alpha = 0.6, colour = violet) +
  scale_x_continuous(trans = "log1p", breaks = c(0, 1, 3, 10, 30, 100, 300, 1000)) +
  labs(x = paste("marker genes per cluster at FDR <", alpha, "(log scale)"), y = NULL)

k_naive <- ncol(p_naive); fpr_naive <- mean(p_naive < alpha)
mk_naive <- mean(n_markers(p_naive)); any_naive <- sum(n_markers(p_naive) > 0)
minp_naive <- min(p_naive); mk_naive_range <- range(n_markers(p_naive))
k_cells <- ncol(p_cells); fpr_cells <- mean(p_cells < alpha); minp_cells <- min(p_cells)
mk_cells <- mean(n_markers(p_cells)); any_cells <- sum(n_markers(p_cells) > 0)
k_split <- ncol(p_split)
k_nbp <- ncol(p_nb_pois); any_nbp <- sum(n_markers(p_nb_pois) > 0)
tot_nbp <- sum(n_markers(p_nb_pois)); minp_nbp <- min(p_nb_pois)
fpr_nbp <- mean(p_nb_pois < alpha)
k_nbnb <- ncol(p_nb_nb); tot_nbnb <- sum(n_markers(p_nb_nb))
phi_hat_med <- median(phi_hat); phi_hat_pois_med <- median(nb_dispersion(counts))
hvg_naive <- rownames(sce) %in% metadata(sce)$hvg; n_hvg_used <- sum(hvg_naive)
fpr_hvg <- mean(p_naive[hvg_naive, ] < alpha); fpr_other <- mean(p_naive[!hvg_naive, ] < alpha)
n_hvg <- formals(cluster_pipeline)$n_hvg; n_pc <- formals(cluster_pipeline)$n_pc
cor_pc1_lib <- abs(cor(reducedDim(sce, "PCA")[, 1], log(colSums(counts))))
r2_lib_naive <- r2(log(colSums(counts)), sce$cluster)

Does count splitting still find real clusters?

n_sub <- 200; n_true <- 30; fold_true <- 3
set.seed(5)
counts_pos <- simulate_cells(n_sub = n_sub, n_marker = n_true, fold = fold_true)
is_sub <- seq_len(n_cells) <= n_sub
is_true <- seq_len(n_genes) <= n_true
cs_pos <- count_split(counts_pos)
tr_pos <- cluster_pipeline(cs_pos$train)
cl_pos <- tr_pos$cluster
best <- names(which.max(tapply(is_sub, cl_pos, mean)))   # cluster holding the subpopulation
q_pos <- p.adjust(nb_vs_rest(cs_pos$test, cl_pos, clusters = best)[, 1], method = "BH")
lower <- rowSums(cs_pos$test[, cl_pos == best]) / sum(cs_pos$test[, cl_pos == best]) <
  rowSums(cs_pos$test[, cl_pos != best]) / sum(cs_pos$test[, cl_pos != best])
pos <- c(cells_found = mean(cl_pos[is_sub] == best), purity = mean(is_sub[cl_pos == best]),
         true_found = sum(q_pos[is_true] < alpha), false_found = sum(q_pos[!is_true] < alpha),
         false_lower = sum(q_pos[!is_true] < alpha & lower[!is_true]))

Halving the counts costs information, so does a real cluster survive? The last simulation adds one: 200 of the 2,000 cells express 30 genes at 3 times the baseline level. The training copy put 100.0% of them in one cluster (which was 100% subpopulation cells), and the edgeR test on the test copy found 30 of the 30 markers. A clear subpopulation stays clear with half the counts; a subtle one may not, and that is the price of an honest p-value.

The test also flagged 11 other genes for that cluster, 11 of them lower there than elsewhere. That is the known weakness of a total-count offset (composition bias): the subpopulation’s extra RNA from its 30 marker genes raises its total count, so every other gene’s share of it falls. Many strongly up-regulated genes in one cluster will do this with any total-count normalisation.

The same check in Python

py <- read.csv("double_dipping_check-results.csv", stringsAsFactors = FALSE)
pyv <- setNames(py$value, py$name)
py_num <- function(k) as.numeric(pyv[[k]])
py_k_naive <- py_num("naive_n_clusters"); py_mk_naive <- py_num("naive_markers_per_cluster")
py_any_naive <- py_num("naive_clusters_with_marker"); py_fpr_naive <- py_num("naive_fpr")
py_k_split <- py_num("split_n_clusters"); py_mk_split <- py_num("split_markers_per_cluster")
py_any_split <- py_num("split_clusters_with_marker"); py_fpr_split <- py_num("split_fpr")
py_scanpy <- pyv[["scanpy_version"]]

The script below runs one null simulation through the usual scanpy steps, ending in rank_genes_groups with the Wilcoxon test against the rest and the tie correction switched on (its default is tie_correct = False) to match the R test. The numbers come from scanpy 1.11.5 and a different random generator. Leiden found 16 clusters, 16 of them with markers (16.7 per cluster on average, 12% of raw p-values below 0.05). After a Poisson count split, 3 of 16 clusters had a marker (0.44 per cluster) and 6.2% of raw p-values were below 0.05; with a Wilcoxon test on log-normalised values, the depth effect described above is the likely source of those.

# Double dipping in scanpy: cluster cells from ONE population, then test markers between clusters.
# Same design as the R code in the post: 2,000 cells, 1,000 genes, Poisson counts, no clusters.
# Writes double_dipping_check-results.csv (columns name,value).
from importlib.metadata import version

import anndata as ad
import numpy as np
import pandas as pd
import scanpy as sc

rng = np.random.default_rng(2026)
n_cells, n_genes = 2000, 1000

mu = np.exp(rng.normal(0.0, 1.0, n_genes))          # gene means, UMI per cell
sf = np.exp(rng.normal(0.0, 0.3, n_cells))          # capture efficiency per cell
counts = rng.poisson(sf[:, None] * mu[None, :])     # cells x genes, one population
genes = [f"g{i + 1:04d}" for i in range(n_genes)]
cells = [f"c{i + 1:04d}" for i in range(n_cells)]


def make(x):
    a = ad.AnnData(X=x.astype(np.float32), obs=pd.DataFrame(index=cells),
                   var=pd.DataFrame(index=genes))
    sc.pp.normalize_total(a)                # default: scale to the median total count
    sc.pp.log1p(a)
    return a


def cluster(a):
    sc.pp.highly_variable_genes(a, n_top_genes=500)
    sc.pp.pca(a, n_comps=10, mask_var="highly_variable")
    sc.pp.neighbors(a, random_state=0)
    sc.tl.leiden(a, flavor="igraph", n_iterations=2, directed=False, random_state=0)
    return a.obs["leiden"]


def markers(a, labels):
    a.obs["cl"] = labels.values
    # each cluster vs rest; tie_correct=True to match the R test (scanpy's default is False)
    sc.tl.rank_genes_groups(a, "cl", method="wilcoxon", tie_correct=True)
    df = sc.get.rank_genes_groups_df(a, group=None)
    per = df.groupby("group", observed=True)["pvals_adj"].apply(lambda q: np.sum(q < 0.05))
    return {"n_clusters": int(labels.nunique()),
            "fpr": float(np.mean(df["pvals"] < 0.05)),
            "markers_per_cluster": float(per.mean()),
            "clusters_with_marker": int(np.sum(per > 0))}


# 1. the common way: cluster and test on the same cells and counts
adata = make(counts)
naive = markers(adata, cluster(adata))

# 2. count splitting: Poisson thinning, cluster on train, test on test
train = rng.binomial(counts, 0.5)
test = counts - train
labels = cluster(make(train))
split = markers(make(test), labels)

rows = {f"naive_{k}": v for k, v in naive.items()}
rows.update({f"split_{k}": v for k, v in split.items()})
rows.update({"n_cells": n_cells, "n_genes": n_genes,
             "scanpy_version": version("scanpy"), "anndata_version": version("anndata"),
             "leidenalg_version": version("leidenalg"), "igraph_version": version("igraph")})
pd.DataFrame({"name": list(rows), "value": list(rows.values())}).to_csv(
    "double_dipping_check-results.csv", index=False)
print(pd.Series(rows))

What to do in practice

Use marker p-values after clustering to rank genes within a cluster, not as evidence. A small p-value says the genes differ between groups that were built to differ; it cannot tell you whether the cluster is a cell type or a cut through a continuum. Effect sizes (a log-fold change, or the AUC that findMarkers() reports with its Wilcoxon p-values) do that job without looking like evidence.

When you need a p-value that means something, for example to claim a new subpopulation, split the raw UMI counts (not normalised or integrated values) before anything else, run the whole pipeline on the training copy, and test the test copy with a count model that includes each cell’s total count:

# counts: raw UMI matrix, genes x cells
cs <- count_split(counts)                  # Poisson split; see below for overdispersed counts
train <- cluster_pipeline(cs$train)        # normalise, HVGs, PCA, cluster: training copy only
p <- nb_vs_rest(cs$test, train$cluster)    # edgeR, total count as offset; slow on many cells

Add batch and other known cell-level factors to that model’s design. Check the noise model before splitting: nb_dispersion() gave a median of 0.00 on the Poisson data and 0.49 on the overdispersed data. It assumes one population, so on real data estimate it within cells you believe to be one population, or from a model that includes the clusters. The authors’ countsplit package on CRAN (install.packages("countsplit")) does both Poisson and negative binomial splitting in one function, countsplit(); see its documentation.

All of this is about comparing clusters within one data set. Comparing the same cell type between conditions is a different question, with its own trap: cells are not replicates there either.

References

• Zhang JM, Kamath GM, Tse DN 2019 Cell Systems 9(4):383-392.e6 (doi:10.1016/j.cels.2019.07.012)
• Neufeld A, Gao LL, Popp J, et al. 2024 Biostatistics 25(1):270-287 (doi:10.1093/biostatistics/kxac047)

R version 4.5.3 (2026-03-11)
Bioconductor 3.22
Biobase 2.70.0, BiocGenerics 0.56.0, bluster 1.20.0, edgeR 4.8.2,
generics 0.1.4, GenomicRanges 1.62.1, ggplot2 4.0.3, IRanges 2.44.0,
limma 3.66.0, MatrixGenerics 1.22.0, matrixStats 1.5.0, S4Vectors
0.48.1, scran 1.38.1, scuttle 1.20.0, Seqinfo 1.0.0,
SingleCellExperiment 1.32.0, SummarizedExperiment 1.40.0
Python part: scanpy 1.11.5, anndata 0.12.19, leidenalg 0.12.0, igraph 1.0.0

What the interaction coefficient is

The experiment has two factors with two levels each: genotype A or B, control or treated. That gives four groups of samples. DESeq2 fits a negative binomial generalised linear model to each gene; its coefficients are on the log2 scale, so each one other than the intercept is a log2 fold change (Love et al. 2014). With the interaction in the design formula there are four coefficients for the four group means:

cd <- data.frame(genotype  = factor(rep(c("A", "B"), each = 6)),
                 treatment = factor(rep(rep(c("ctrl", "trt"), each = 3), 2)))
table(cd$genotype, cd$treatment)

   
    ctrl trt
  A    3   3
  B    3   3

colnames(model.matrix(~ genotype + treatment + genotype:treatment, cd))

[1] "(Intercept)"            "genotypeB"              "treatmenttrt"          
[4] "genotypeB:treatmenttrt"

Read them with genotype A and control as the reference levels (the first level of each factor). The intercept is the log2 mean of untreated genotype A. genotypeB is the difference between the genotypes in untreated samples. treatmenttrt is the treatment effect in genotype A only, not an average over genotypes; the DESeq2 vignette calls this the key point to remember about designs with interaction terms. The interaction, genotypeB:treatmenttrt, is the treatment effect in genotype B minus the treatment effect in genotype A. A gene that goes up fourfold with treatment in A (log2 fold change 2) and twofold in B (log2 fold change 1) has an interaction of minus 1. The treatment effect in B is the sum of the treatment coefficient and the interaction.

So the interaction coefficient answers exactly the question “does the treatment response differ between genotypes?”, and its test is the test of that question. The figure shows the three patterns used below.

ex <- expand.grid(treatment = c("ctrl", "trt"), genotype = c("A", "B"),
                  type = c("equal effect", "A only", "opposite"))
slope <- c("equal effect" = 1, "A only" = 0, "opposite" = -1)   # effect in B for an effect of 1 in A
ex$log2mu <- 7 + 0.5 * (ex$genotype == "B") + (ex$treatment == "trt") *
  ifelse(ex$genotype == "A", 1, slope[as.character(ex$type)])
lab <- data.frame(type = factor(names(slope), levels = levels(ex$type)),
                  txt = paste("interaction =", slope - 1))
ggplot(ex, aes(treatment, log2mu, colour = genotype, group = genotype)) +
  geom_line(linewidth = 1) + geom_point(size = 2.5) +
  geom_text(data = lab, aes(x = 1.5, y = 9, label = txt), inherit.aes = FALSE,
            colour = "grey30", size = 3.5) +
  facet_wrap(~ type) + coord_cartesian(ylim = c(6.3, 9.2)) +
  scale_colour_manual(values = c(A = violet, B = magenta)) +
  labs(x = NULL, y = "true log2 mean") + theme_pc

The simulation

Each simulated experiment has twelve samples, three per group, and 3,000 genes. Counts are negative binomial, the usual model for RNA-seq read counts, with more scatter for weakly expressed genes. Every gene gets its own baseline level and a random baseline difference between the genotypes. The genes fall into four types:

• 1,800 genes do not respond to the treatment in either genotype;
• 400 respond with the same log2 fold change in both genotypes;
• 400 respond in genotype A and not at all in B;
• 400 respond in A and by the same amount in the opposite direction in B.

For the responding genes the size of the log2 fold change is drawn between 0.6 and 1.4, with a random sign. Only the last two types have a true interaction.

G <- 3000
n_cls <- c(null = 1800, equal = 400, A_only = 400, opposite = 400)
cls <- factor(rep(names(n_cls), n_cls), levels = names(n_cls))
lfc_range <- c(0.6, 1.4)                               # |log2 fold change| of responding genes
sim_genes <- function() {
  mu0 <- exp(rnorm(G, log(150), 1.3))                  # baseline mean count
  d <- runif(G, lfc_range[1], lfc_range[2]) * sample(c(-1, 1), G, TRUE)
  bA <- ifelse(cls == "null", 0, d)                    # treatment effect in genotype A
  bB <- ifelse(cls %in% c("null", "A_only"), 0,        # treatment effect in genotype B
               ifelse(cls == "equal", d, -d))
  list(mu0 = mu0, phi = 0.03 + 0.3 / sqrt(mu0),        # NB dispersion, higher for low counts
       bA = bA, bB = bB, gx = rnorm(G, 0, 0.5))        # gx: genotype B baseline shift
}
sim_counts <- function(g) {
  isB <- cd$genotype == "B"; isT <- cd$treatment == "trt"
  lf <- outer(g$gx, isB) + outer(g$bA, isT & !isB) + outer(g$bB, isT & isB)
  mu <- g$mu0 * 2^lf * rep(runif(12, 0.8, 1.25), each = G)
  m <- matrix(rnbinom(G * 12, mu = mu, size = 1 / g$phi), G, 12)
  storage.mode(m) <- "integer"
  m
}

The size of the problem shown below depends on power, the chance that a truly responding gene reaches significance. If power were close to 1 or close to 0, the two-list approach would rarely split a gene. It does the most damage in between, which is where this three-replicate experiment with moderate effects sits. The other assumptions (balanced groups, effects that add on the log scale, a correct count model) are shared by both analyses.

Each dataset is analysed in two ways, both with the DESeq2 standard workflow (DESeq(), then results() at a 5% false discovery rate, FDR, with its alpha argument set to that cut-off as the vignette asks whenever the cut-off differs from the default). The first fits one model to all twelve samples and tests the interaction coefficient. The second splits the samples by genotype, fits ~ treatment to each half, and compares the two lists of significant genes, which is what a Venn diagram of the two lists does.

alpha <- 0.05                                           # FDR cut-off
sig <- function(r) !is.na(r$padj) & r$padj < alpha
analyse <- function(cts) {
  # one model with the interaction
  dds <- DESeq(DESeqDataSetFromMatrix(cts, cd, ~ genotype + treatment + genotype:treatment),
               quiet = TRUE)
  inter <- results(dds, name = "genotypeB.treatmenttrt", alpha = alpha)
  inA   <- results(dds, name = "treatment_trt_vs_ctrl", alpha = alpha)
  inB   <- results(dds, contrast = list(c("treatment_trt_vs_ctrl", "genotypeB.treatmenttrt")),
                   alpha = alpha)
  # two separate analyses, one per genotype
  sep <- lapply(c(A = "A", B = "B"), function(k) {
    s <- cd$genotype == k
    d <- DESeq(DESeqDataSetFromMatrix(cts[, s], cd[s, ], ~ treatment), quiet = TRUE)
    results(d, name = "treatment_trt_vs_ctrl", alpha = alpha)
  })
  list(dds = dds, inter = inter, inA = inA, inB = inB, sepA = sep$A, sepB = sep$B)
}

The loop repeats the whole experiment 5 times with new genes and new counts. For every repeat it records, per gene type, the share of genes each approach calls genotype-dependent.

n_rep <- 5
set.seed(2026)
rates <- list()
for (r in seq_len(n_rep)) {
  g <- sim_genes()
  cts <- sim_counts(g)
  a <- analyse(cts)
  if (r == 1) { g1 <- g; cts1 <- cts; a1 <- a }
  sA <- sig(a$sepA); sB <- sig(a$sepB)
  venn  <- xor(sA, sB)                         # significant in one genotype only
  venn1 <- xor(sig(a$inA), sig(a$inB))         # the same, from the one-model contrasts
  inter <- sig(a$inter)
  diff_true <- g$bA != g$bB
  rates[[r]] <- data.frame(rep = r, class = levels(cls),
    venn  = tapply(venn, cls, mean), venn1 = tapply(venn1, cls, mean),
    inter = tapply(inter, cls, mean), both = tapply(sA & sB, cls, mean),
    sigA = tapply(sA, cls, mean), sigB = tapply(sB, cls, mean),
    fdp_venn = mean(!diff_true[venn]), fdp_venn1 = mean(!diff_true[venn1]),
    fdp_inter = mean(!diff_true[inter]), n_venn = sum(venn), n_inter = sum(inter),
    se_ratio = median(a$inter$lfcSE / a$inA$lfcSE, na.rm = TRUE))
}
rates <- do.call(rbind, rates)

Two lists and a Venn diagram

avg <- aggregate(. ~ class, rates[, setdiff(names(rates), "rep")], mean)
v <- function(cl, col) avg[avg$class == cl, col]
venn_eq <- v("equal", "venn"); venn1_eq <- v("equal", "venn1"); inter_eq <- v("equal", "inter")
sigA_eq <- v("equal", "sigA"); sigB_eq <- v("equal", "sigB")
venn_null <- v("null", "venn"); inter_null <- v("null", "inter")
venn_Aonly <- v("A_only", "venn"); inter_Aonly <- v("A_only", "inter")
both_opp <- v("opposite", "both"); venn_opp <- v("opposite", "venn"); inter_opp <- v("opposite", "inter")
fdp_venn <- v("null", "fdp_venn"); fdp_venn1 <- v("null", "fdp_venn1"); fdp_inter <- v("null", "fdp_inter")
n_venn <- v("null", "n_venn"); n_inter <- v("null", "n_inter")
se_ratio <- v("null", "se_ratio")
venn_eq_rng <- range(rates$venn[rates$class == "equal"])
fdp_venn_rng <- range(rates$fdp_venn[rates$class == "null"])
fdp_inter_rng <- range(rates$fdp_inter[rates$class == "null"])
# design factor: variance of the interaction vs a within-genotype treatment effect
X <- model.matrix(~ genotype + treatment + genotype:treatment, cd)
XtXi <- solve(crossprod(X))
se_design <- sqrt(XtXi["genotypeB:treatmenttrt", "genotypeB:treatmenttrt"] /
                  XtXi["treatmenttrt", "treatmenttrt"])
# first simulated experiment: equal-effect genes significant in one genotype only
eq <- cls == "equal"
sA1 <- sig(a1$sepA); sB1 <- sig(a1$sepB)
lA1 <- a1$sepA$log2FoldChange; lB1 <- a1$sepB$log2FoldChange
one <- eq & xor(sA1, sB1)
n_one <- sum(one)
n_one_inter <- sum(sig(a1$inter)[one])                 # of these, flagged by the interaction test
side_share <- mean(ifelse(sA1, abs(lA1) > abs(lB1), abs(lB1) > abs(lA1))[one])
gap_one  <- median(abs(lA1 - lB1)[one])
gap_both <- median(abs(lA1 - lB1)[eq & sA1 & sB1])
avg

     class       venn      venn1      inter         both       sigA   sigB
1   A_only 0.69700000 0.71200000 0.42150000 0.0080000000 0.70350000 0.0095
2    equal 0.29000000 0.27400000 0.00750000 0.5335000000 0.70250000 0.6545
3     null 0.02588889 0.02988889 0.01355556 0.0002222222 0.01633333 0.0100
4 opposite 0.28250000 0.26750000 0.86200000 0.5350000000 0.69900000 0.6535
   fdp_venn fdp_venn1  fdp_inter n_venn n_inter se_ratio
1 0.2927728 0.2941584 0.05077606  554.4   540.8 1.415617
2 0.2927728 0.2941584 0.05077606  554.4   540.8 1.415617
3 0.2927728 0.2941584 0.05077606  554.4   540.8 1.415617
4 0.2927728 0.2941584 0.05077606  554.4   540.8 1.415617

aggregate(cbind(venn, inter) ~ class, rates, range)   # min and max over repeats

     class     venn.1     venn.2    inter.1    inter.2
1   A_only 0.66750000 0.74750000 0.38750000 0.45000000
2    equal 0.26000000 0.31000000 0.00250000 0.01000000
3     null 0.01555556 0.03333333 0.01166667 0.01555556
4 opposite 0.26750000 0.31250000 0.83750000 0.87750000

c(se_design = se_design, se_ratio = se_ratio, fdp_venn_rng = fdp_venn_rng, fdp_inter_rng = fdp_inter_rng)

     se_design       se_ratio  fdp_venn_rng1  fdp_venn_rng2 fdp_inter_rng1 
    1.41421356     1.41561660     0.26728972     0.30598291     0.04332130 
fdp_inter_rng2 
    0.05950096 

c(n_one = n_one, n_one_inter = n_one_inter, side_share = side_share,
  gap_one = gap_one, gap_both = gap_both)

      n_one n_one_inter  side_share     gap_one    gap_both 
104.0000000   1.0000000   0.9807692   0.4673976   0.2340765 

In the separate analyses, 70% of the equal-effect genes are significant in genotype A and 65% in genotype B. Those are two imperfect detectors pointed at the same signal, and they often disagree: 29% of the equal-effect genes are significant in exactly one genotype (between 26% and 31% across the repeats), and a Venn diagram puts every one of them in a genotype-specific segment. The next figure shows these genes for the first simulated experiment.

lab_v <- ifelse(sA1 & sB1, "both", ifelse(sA1, "A only", ifelse(sB1, "B only", "neither")))
sc <- data.frame(lA = a1$sepA$log2FoldChange, lB = a1$sepB$log2FoldChange,
                 venn = factor(lab_v, levels = c("both", "A only", "B only", "neither")))[eq, ]
ggplot(sc, aes(lA, lB, colour = venn)) +
  geom_abline(slope = 1, intercept = 0, colour = "grey60", linetype = "dashed") +
  geom_point(size = 1.6, alpha = 0.8) +
  scale_colour_manual(values = c(both = grey, "A only" = violet, "B only" = magenta,
                                 neither = "grey85"), name = "significant in") +
  coord_equal() +
  labs(x = "estimated log2 fold change, genotype A", y = "estimated log2 fold change, genotype B") +
  theme_pc

The coloured points are not spread at random. Genes called in genotype A only sit on the A side of the diagonal, with a larger estimated effect in A than in B, and genes called in B only sit on the B side; in this experiment that holds for 98% of them. They form the edges of the same clouds as the grey genes. That is selection, not biology. A gene lands in the A-only segment when noise pushes its estimate in A over the significance threshold and its estimate in B below it, so the genes in that segment are the ones whose two estimates happened to drift apart. The median gap between the two estimated log2 fold changes is 0.47 for the genes significant in one genotype only, against 0.23 for the genes significant in both.

A reader looking at the figure could take that wider gap as evidence of a different response. It is not: the true effects are identical, and the gap is no larger than the noise in two separate estimates produces. The interaction test, which asks exactly whether the two effects differ, flags 1 of the 104 genes in this experiment that the Venn diagram puts in a genotype-specific segment. Gelman and Stern (2006) put the general point in their title: the difference between “significant” and “not significant” is not itself statistically significant. A non-significant result in genotype B means the data could not show an effect there, not that there is none.

The error is not caused by fitting the two genotypes separately. Taking the two per-genotype effects from the single interaction model (shown further down) and comparing their significance calls in the same way labels 27% of the equal-effect genes as genotype-specific, and 29% of those calls are false. The problem is the comparison of two significance calls, wherever they come from.

The two-list approach also mislabels many of the most striking genes. Of the genes that respond in opposite directions, 54% are significant in both analyses. In a Venn diagram of gene identifiers they sit in the shared segment, as “responds in both genotypes”, although the response is reversed. Only 28% of them are called genotype-specific. Splitting the lists by direction of change catches these genes, but not the equal-effect problem above.

The interaction test

long <- rbind(data.frame(rates[, c("rep", "class")], method = "Venn of two analyses", value = rates$venn),
              data.frame(rates[, c("rep", "class")], method = "interaction test", value = rates$inter))
long$class <- factor(long$class, levels = rev(levels(cls)),
                     labels = rev(c("null", "equal effect", "A only", "opposite")))
long$y <- as.numeric(long$class) + ifelse(long$method == "interaction test", 0.15, -0.15)
ggplot(long, aes(value, y, colour = method)) +
  geom_point(size = 2.4, alpha = 0.8) +
  scale_colour_manual(values = c("Venn of two analyses" = magenta, "interaction test" = violet),
                      name = NULL) +
  scale_y_continuous(breaks = seq_along(levels(long$class)), labels = levels(long$class)) +
  coord_cartesian(xlim = c(0, 1)) +
  labs(x = "share of genes called genotype-dependent", y = NULL) + theme_pc

The interaction test calls 0.8% of the equal-effect genes and 1.4% of the non-responding genes genotype-dependent. Among all genes it calls, 5.1% have no true interaction on average (between 4.3% and 6.0% across the repeats), about the 5% target it was run at (a little above it in some repeats). For the two-list approach the same share is 29.3% on average. Both lists are about the same length, 541 genes for the interaction test and 554 for the Venn approach; they differ in what is on them. The interaction test also finds 86% of the opposite-direction genes.

There is a cost, and the figure shows it. For genes that respond in genotype A only, the interaction test finds 42%, while the two-list approach labels 70% of them genotype-specific. That is not extra power in the two-list approach. It applies a lower bar (significant in one genotype, not in the other), and the same lower bar is what mislabels the equal-effect genes; a list that cannot tell you which 29.3% of its entries are wrong is not a better list.

The interaction really is harder to estimate than a single treatment effect. It is a difference between two differences, each estimated from six samples, so its variance is the sum of theirs. In this balanced design the standard error of the interaction coefficient is 1.41 times that of the treatment effect in one genotype (the square root of 2, from the design matrix alone); the median ratio of the standard errors DESeq2 reports is 1.42. To detect a given difference in response with the same power as a response of the same size within one genotype, the sample size has to grow by a factor of about 2, and by more if the difference in response you care about is smaller than the response itself. If a genotype-dependent response is the main question of the experiment, that belongs in the sample size calculation.

The treatment effect in each genotype

Often you want the per-genotype effects as well, for example to report a fold change for each genotype next to the interaction test. All of them come from the one model. The names DESeq2 gives the coefficients differ from the model matrix column names (the main effects are renamed and the colon becomes a dot), so read them off resultsNames() rather than typing them from memory:

resultsNames(a1$dds)

[1] "Intercept"              "genotype_B_vs_A"        "treatment_trt_vs_ctrl" 
[4] "genotypeB.treatmenttrt"

# argument names used below
c("name", "contrast", "alpha") %in% names(formals(results))

[1] TRUE TRUE TRUE

formals(results)$alpha                                   # default FDR target of results()

[1] 0.1

c("test", "reduced") %in% names(formals(DESeq))

[1] TRUE TRUE

The treatment effect in genotype A is the coefficient treatment_trt_vs_ctrl, extracted with results(dds, name = "treatment_trt_vs_ctrl"). The effect in genotype B is that coefficient plus the interaction. results() accepts a list for its contrast argument; according to its help page the first element names the coefficients that are added up for the numerator, and a list of length one is completed with an empty denominator. This is the route shown in Examples 2 and 3 of the results() help page:

results(dds, contrast = list(c("treatment_trt_vs_ctrl", "genotypeB.treatmenttrt")))

The chunk below checks it three ways on the first simulated dataset: against the sum of the two fitted coefficients, against a refit with genotype B as the reference level, and against the combined-factor design (~ group, with levels such as Bctrl and Btrt) that the DESeq2 vignette suggests when only per-group comparisons are wanted.

b <- coef(a1$dds)                                        # MLE coefficients, log2 scale
sum_coef <- b[, "treatment_trt_vs_ctrl"] + b[, "genotypeB.treatmenttrt"]
diff_sum <- max(abs(a1$inB$log2FoldChange - sum_coef), na.rm = TRUE)
# the same effect with genotype B as the reference level
dds_B <- a1$dds
dds_B$genotype <- relevel(dds_B$genotype, "B")
dds_B <- DESeq(dds_B, quiet = TRUE)
rB_relevel <- results(dds_B, name = "treatment_trt_vs_ctrl")
diff_relevel <- max(abs(a1$inB$log2FoldChange - rB_relevel$log2FoldChange), na.rm = TRUE)
diff_relevel_p <- max(abs(a1$inB$pvalue - rB_relevel$pvalue), na.rm = TRUE)
# the combined-factor route from the vignette
dds_g <- a1$dds
dds_g$group <- factor(paste0(dds_g$genotype, dds_g$treatment))
design(dds_g) <- ~ group
dds_g <- DESeq(dds_g, quiet = TRUE)
rB_group <- results(dds_g, contrast = c("group", "Btrt", "Bctrl"))
diff_group <- max(abs(a1$inB$log2FoldChange - rB_group$log2FoldChange), na.rm = TRUE)
# the interaction as a single contrast in the ~ group design
resultsNames(dds_g)

[1] "Intercept"            "group_Atrt_vs_Actrl"  "group_Bctrl_vs_Actrl"
[4] "group_Btrt_vs_Actrl" 

int_group <- results(dds_g, contrast = list("group_Btrt_vs_Actrl",
                                            c("group_Bctrl_vs_Actrl", "group_Atrt_vs_Actrl")))
diff_int_group <- max(abs(a1$inter$log2FoldChange - int_group$log2FoldChange), na.rm = TRUE)
diff_int_group_p <- max(abs(a1$inter$pvalue - int_group$pvalue), na.rm = TRUE)
c(diff_sum = diff_sum, diff_relevel = diff_relevel, diff_relevel_p = diff_relevel_p,
  diff_group = diff_group, diff_int_group = diff_int_group, diff_int_group_p = diff_int_group_p)

        diff_sum     diff_relevel   diff_relevel_p       diff_group 
    4.440892e-16     4.758795e-06     1.483779e-06     7.719630e-06 
  diff_int_group diff_int_group_p 
    1.710804e-05     9.779561e-07 

The list contrast equals the sum of the coefficients to machine precision (largest difference 4e-16). The refit with genotype B as reference and the combined-factor design give the same log2 fold changes up to the tolerance of the fitting algorithm (largest differences 5e-06 and 8e-06), and the relevelled fit gives the same p-values to within 1e-06.

The ~ group design does not take the interaction test away, although its separate per-group tables invite the Venn comparison. The interaction is the treated-minus-control difference in B minus the same difference in A, which is one list contrast of the group coefficients:

results(dds_g, contrast = list("group_Btrt_vs_Actrl",
                               c("group_Bctrl_vs_Actrl", "group_Atrt_vs_Actrl")))

In the chunk above it reproduces the interaction coefficient of the first model (largest difference in log2 fold change 2e-05, in p-value 1e-06).

What to do in practice

When the question is whether a response differs between groups, put that question into the design formula and test the coefficient that answers it. This is the same rule as for batch effects (see the post on confounded batches): the comparison belongs in the model, not in a comparison of two results tables after the fact.

dds <- DESeqDataSetFromMatrix(counts, samples,
                              design = ~ genotype + treatment + genotype:treatment)
dds$genotype  <- relevel(dds$genotype, "A")      # set reference levels before DESeq()
dds$treatment <- relevel(dds$treatment, "ctrl")
dds <- DESeq(dds)
resultsNames(dds)

res_diff <- results(dds, name = "genotypeB.treatmenttrt", alpha = 0.05)   # does the response differ?
res_A    <- results(dds, name = "treatment_trt_vs_ctrl", alpha = 0.05)    # response in genotype A
res_B    <- results(dds, contrast = list(c("treatment_trt_vs_ctrl",
                                           "genotypeB.treatmenttrt")),
                    alpha = 0.05)                                        # response in genotype B

Report genes as genotype-dependent only when res_diff calls them, and show the two per-genotype fold changes side by side, with their standard errors, rather than as a Venn diagram. For plotting or ranking those fold changes, the DESeq2 vignette recommends shrunken estimates from lfcShrink(). Its default apeglm method needs a coefficient name (coef) rather than a contrast, so the effect in genotype B comes from a fit with B as the reference level:

shr_A <- lfcShrink(dds, coef = "treatment_trt_vs_ctrl")          # apeglm package required
dds_B <- dds
dds_B$genotype <- relevel(dds_B$genotype, "B")
dds_B <- DESeq(dds_B)
shr_B <- lfcShrink(dds_B, coef = "treatment_trt_vs_ctrl")

With more than two genotypes, a likelihood ratio test asks whether the response differs anywhere: DESeq(dds, test = "LRT", reduced = ~ genotype + treatment). A gene significant in one genotype and not in the other has not been shown to be genotype-specific; it may only have been measured with less certainty in one of them.

References

Gelman A, Stern H 2006 The American Statistician 60(4):328-331 (doi:10.1198/000313006X152649)

Love MI, Huber W, Anders S 2014 Genome Biology 15(12):550 (doi:10.1186/s13059-014-0550-8)

R version 4.5.3 (2026-03-11)
Bioconductor 3.22
Biobase 2.70.0, BiocGenerics 0.56.0, DESeq2 1.50.2, generics 0.1.4,
GenomicRanges 1.62.1, ggplot2 4.0.3, IRanges 2.44.0, MatrixGenerics
1.22.0, matrixStats 1.5.0, S4Vectors 0.48.1, Seqinfo 1.0.0,
SummarizedExperiment 1.40.0

Why removing genes can add power

The usual FDR correction is the Benjamini-Hochberg procedure (Benjamini and Hochberg 1995). It sorts the p-values of all tested genes and compares the k-th smallest with k/m times the target FDR, where m is the number of genes tested. The larger m is, the higher the bar every gene has to clear. In RNA-seq a large share of genes have so few reads that they could never produce a small p-value, whatever their true change, yet each of them adds to m. Removing them before the correction lowers the bar for the rest. (The same genes bend the p-value histogram towards 1, as shown in the post on p-value histograms; here the question is what removing them does to the FDR.)

The catch is in how you choose which genes to remove. The procedure’s guarantee rests on the p-values of null genes, genes with no true change, being spread evenly between 0 and 1. Bourgon, Gentleman and Huber (2010) showed that filtering keeps this property when, for null genes, the filter statistic is independent of the test statistic. The overall mean count, computed across all samples without looking at the group labels, is the standard example. A statistic that looks at the difference between the groups is not: a null gene whose groups differ by chance has both a large difference and a small p-value, so a filter on the difference hands the correction a set of null genes with too many small p-values. The simulation below measures both.

The setup

Each simulated experiment has 3,000 genes measured in 4 control and 4 treated samples, with counts drawn from a negative binomial distribution (the standard model for RNA-seq counts, in which the variance grows faster than the mean). Mean counts are spread over several orders of magnitude, so that 49% of genes average fewer than 10 reads per sample, much as in a real annotation with many barely expressed genes. The dispersion, the biological part of the variance, is larger for weakly expressed genes. In each experiment 15% of genes truly change, with a log2 fold change between 0.5 and 2 in either direction; the rest are null.

I simulate 16 experiments and report averages over them, because the FDR is itself an average over experiments: in a single experiment the share of false calls can land above the target without anything being wrong.

Two assumptions deserve a flag. Genes are simulated independently of each other, while real genes are correlated. And the share of changed genes and the size of their changes are my choices; the last section shows that the damage done by a bad filter depends on the first.

n_per <- 4                                   # samples per condition
n_genes <- 3000                              # genes per experiment
n_exp <- 16                                  # simulated experiments
de_frac <- 0.15                              # share of genes that truly change
lfc_min <- 0.5; lfc_max <- 2                 # size of true changes, |log2 fold change|
group <- factor(rep(c("ctrl", "treat"), each = n_per))
design <- model.matrix(~ group)

simulate <- function(n_rows, de_frac, n_block) {
  mu0 <- exp(rnorm(n_rows, log(10), 2.5))                  # mean count, most genes low
  phi <- (0.05 + 0.2 / sqrt(mu0 + 1)) * exp(rnorm(n_rows, 0, 0.3))   # dispersion
  is_de <- rep(seq_len(n_block) <= de_frac * n_block, length.out = n_rows)
  lfc <- ifelse(is_de, sample(c(-1, 1), n_rows, TRUE) * runif(n_rows, lfc_min, lfc_max), 0)
  size_factor <- exp(rnorm(2 * n_per, 0, 0.15))
  mu <- outer(mu0, size_factor) * 2^outer(lfc, as.numeric(group == "treat"))
  counts <- matrix(as.integer(rnbinom(length(mu), mu = mu, size = 1 / phi)), nrow = n_rows)
  list(counts = counts, is_de = is_de, lfc = lfc)
}

set.seed(2026)
sim <- simulate(n_genes * n_exp, de_frac, n_genes)
experiment <- rep(seq_len(n_exp), each = n_genes)

DESeq2 runs its standard workflow, DESeq() then results() (Love, Huber and Anders 2014). To save time it is run once on all experiments stacked on top of each other; they share the 8 samples’ sequencing depths and the fitted dispersion trend, and every experiment is filtered, corrected and scored on its own. edgeR runs per experiment with the quasi-likelihood recipe of Chen, Lun and Smyth (2016), updated with the current filterByExpr() and normLibSizes(): filter, normalise, then glmQLFit() and glmQLFTest(). The fit uses edgeR’s newer quasi-likelihood method, legacy = FALSE, set explicitly so that older and newer edgeR releases run the same method; this method estimates its own dispersion, so estimateDisp() is not needed.

# one DESeq2 fit for all experiments (to save time); each experiment is scored on its own below
dds <- DESeqDataSetFromMatrix(sim$counts, data.frame(group = group), ~ group)
dds <- DESeq(dds, quiet = TRUE)
res_all <- results(dds, independentFiltering = FALSE)
low_count <- 10
share_low <- mean(res_all$baseMean < low_count)

# the edgeR quasi-likelihood recipe, filterByExpr first
run_edger <- function(counts, fc_cut) {
  y <- DGEList(counts, group = group)
  keep <- filterByExpr(y, group = group)
  y <- normLibSizes(y[keep, , keep.lib.sizes = FALSE])
  fit <- glmQLFit(y, design, robust = TRUE, legacy = FALSE)
  tab <- glmQLFTest(fit, coef = 2)$table
  treat <- glmTreat(fit, coef = 2, lfc = fc_cut)$table     # used in the last section
  p <- lfc <- p_treat <- rep(NA_real_, nrow(counts))
  p[keep] <- tab$PValue; lfc[keep] <- tab$logFC; p_treat[keep] <- treat$PValue
  list(p = p, lfc = lfc, keep = keep, p_treat = p_treat)
}

The next chunk scores every analysis in this post on each experiment: DESeq2 with and without its automatic filter, the edgeR recipe, both with a fold-change pre-filter added, and the two threshold tests from the last section.

fc_cut <- 0.5                                # |log2 fold change| cut-off of the pre-filter
thresholds <- c(0, 0.25, 0.5, 0.75, 1, 1.5)  # cut-offs for the sweep further down

per_exp <- lapply(seq_len(n_exp), function(e) {
  i <- which(experiment == e)
  de <- sim$is_de[i]
  p <- res_all$pvalue[i]
  fc <- abs(res_all$log2FoldChange[i])
  auto <- results(dds[i, ], alpha = fdr_level)             # automatic independent filtering
  auto_default <- results(dds[i, ])                        # same, alpha left at its default
  treat <- results(dds[i, ], lfcThreshold = fc_cut, alpha = fdr_level)
  ed <- run_edger(sim$counts[i, ], fc_cut)
  fc_keep <- !is.na(fc) & fc > fc_cut
  below_cut <- res_all$baseMean[i] < metadata(auto)$filterThreshold
  ed_fc_keep <- ed$keep & abs(ed$lfc) > fc_cut
  ed_padj <- bh_on(ed$p, ed$keep)                          # correction first ...
  post_cut <- function(cut) ifelse(abs(ed$lfc) > cut, ed_padj, NA_real_)   # ... then a fold-change cut
  list(
    rows = rbind(
      deseq_none   = score(bh_on(p, TRUE), de),
      deseq_auto   = score(auto$padj, de),
      deseq_fc     = score(bh_on(p, fc_keep), de),
      edger        = score(bh_on(ed$p, ed$keep), de),
      edger_fc     = score(bh_on(ed$p, ed_fc_keep), de),
      deseq_treat  = score(treat$padj, de),
      edger_treat  = score(bh_on(ed$p_treat, ed$keep), de)),
    auto_default = score(auto_default$padj, de),
    post = rbind(small = score(post_cut(fc_cut), de), large = score(post_cut(1), de)),
    null_share_fc = mean(!de[ed_fc_keep]),
    filtered = mean(is.na(auto$padj)),
    sweep = sapply(thresholds, function(t) c(
      deseq = score(bh_on(p, !is.na(fc) & fc > t), de)[["fdp"]],
      edger = score(bh_on(ed$p, ed$keep & abs(ed$lfc) > t), de)[["fdp"]])),
    # null p-values that survive each filter, for the histogram
    null_p = rbind(
      data.frame(p = ed$p[ed$keep & !de], filter = "recipe"),
      data.frame(p = ed$p[ed_fc_keep & !de], filter = "fold"),
      data.frame(p = p[fc_keep & !de],
                 filter = ifelse(below_cut[fc_keep & !de], "deseq_fold_low", "deseq_fold_high"))),
    auto = if (e == 1) auto else NULL)
})

tab <- Reduce(`+`, lapply(per_exp, `[[`, "rows")) / n_exp
auto_default <- colMeans(do.call(rbind, lapply(per_exp, `[[`, "auto_default")))
filtered_share <- mean(sapply(per_exp, `[[`, "filtered"))
gain <- tab["deseq_auto", "n_call"] / tab["deseq_none", "n_call"] - 1
fdp_fc_edger <- sapply(per_exp, function(x) x$rows["edger_fc", "fdp"])
fdp_recipe <- sapply(per_exp, function(x) x$rows["edger", "fdp"])
n_fc_worse <- sum(fdp_fc_edger > fdp_recipe)
null_p <- do.call(rbind, lapply(per_exp, `[[`, "null_p"))
null_sig <- tapply(null_p$p < fdr_level, null_p$filter, mean)
deseq_null <- null_p[null_p$filter %in% c("deseq_fold_low", "deseq_fold_high"), ]
deseq_null_sig <- mean(deseq_null$p < fdr_level)
deseq_fc_null_low <- mean(deseq_null$filter == "deseq_fold_low")
sweep <- Reduce(`+`, lapply(per_exp, `[[`, "sweep")) / n_exp
null_share_fc <- mean(sapply(per_exp, `[[`, "null_share_fc"))
post <- Reduce(`+`, lapply(per_exp, `[[`, "post")) / n_exp
auto1 <- per_exp[[1]]$auto

Filtering on expression

DESeq2 filters for you. With the default independentFiltering = TRUE, results() filters on the mean of normalised counts across all samples. It tries a series of cut-offs, counts the genes called at each, and keeps the lowest cut-off whose count comes within one residual standard deviation of the peak of a curve fitted through those counts (this is described in the DESeq2 vignette). Genes below the cut-off keep their p-value, but their adjusted p-value is left empty. The search for the first experiment looks like this:

nr <- metadata(auto1)$filterNumRej
lo <- as.data.frame(metadata(auto1)$lo.fit)
ggplot(nr, aes(theta, numRej)) +
  geom_line(data = lo, aes(x, y), colour = "#8e8c9c", linewidth = 0.7) +
  geom_point(colour = "#5a3fc0", size = 1.8) +
  geom_vline(xintercept = metadata(auto1)$filterTheta, linetype = "dashed",
             colour = "#d14fa6", linewidth = 0.7) +
  labs(x = "share of genes removed (quantile of the mean normalised count)",
       y = "genes called") +
  theme_minimal(base_size = 12) +
  theme(panel.grid.minor = element_blank(),
        plot.background = element_rect(fill = "white", colour = "white"))

In this experiment the chosen cut-off removes the 20% of genes whose mean normalised count is below 1.26. Across all experiments it removed 26% of genes on average. DESeq2 then called 172 genes per experiment against 163 without the filter, and found 36.5% of the truly changed genes instead of 34.9%. The realised FDR was 4.7% with the filter and 3.5% without it, both below the nominal level. The gain is modest in this simulation; how large it is in your data depends on how many genes in the table have too few reads to ever be significant.

One detail matters here. The cut-off is chosen to maximise the calls at the significance level alpha, whose default is 0.1. If you call genes at an adjusted p-value of 0.05, the vignette says to set alpha to that value too. Leaving it at the default made little difference in this simulation (171 against 172 genes called), but it costs nothing to set.

edgeR filters before the model is fitted. filterByExpr() keeps genes with at least min.count reads (default 10, converted to counts per million with the median library size) in at least as many samples as the smallest group, and at least min.total.count reads in total (default 15). It uses the group sizes only to decide how many samples a gene must be expressed in; it counts those samples across all columns, so the result does not depend on which sample sits in which group. With filterByExpr() first, the edgeR recipe had a realised FDR of 4.7% and found 35.5% of the changed genes.

Filtering on a fold change

Now the tempting shortcut: keep only genes whose groups differ by at least 1.4-fold (an absolute log2 fold change above 0.5), taken from the same results table, and run the correction on those. Added to the edgeR recipe, this raised the calls from 167 to 223 per experiment and the realised FDR from 4.7% to 16.4%. The false discovery proportion was higher with the fold-change filter in every one of the 16 experiments, ranging from 12.0% to 21.1%.

The reason is visible in the null genes that survive the filter.

lv <- c(recipe = "filterByExpr only", fold = "filterByExpr, then |logFC| > cut-off")
d <- null_p[null_p$filter %in% names(lv), ]
d$panel <- factor(lv[d$filter], levels = lv)
ggplot(d, aes(p)) +
  geom_histogram(aes(y = after_stat(density)), breaks = seq(0, 1, 0.05),
                 fill = "#5a3fc0", colour = "white", linewidth = 0.2) +
  geom_hline(yintercept = 1, linetype = "dashed", colour = "#d14fa6", linewidth = 0.7) +
  facet_wrap(~ panel) +
  labs(x = "p-value of null genes that pass the filter", y = "density") +
  theme_minimal(base_size = 12) +
  theme(panel.grid.minor = element_blank(), strip.text = element_text(face = "bold"),
        plot.background = element_rect(fill = "white", colour = "white"))

Among null genes that pass filterByExpr(), 5.1% have a p-value below 0.05, which is what a p-value promises. Among null genes that also pass the fold-change filter, 35.5% do. The correction receives a set of genes with far more small p-values than its null model allows for, and treats the excess as signal.

The same filter on DESeq2’s p-values, applied to all genes, raised the realised FDR from 3.5% to 7.1%. That is a smaller rise, and the reason says something about the mechanism. DESeq2’s fold change for a gene with a handful of reads is very noisy, so many null genes with almost no reads pass a fold-change filter: 48% of the null genes that passed had a mean count below the automatic filter’s cut-off. Only 0.8% of those had a p-value below 0.05, against 16.5% of the better expressed null genes that passed, so the low-count genes dilute the problem without removing it. Overall, 8.9% of the null genes passing this filter had a p-value below 0.05, still more than a p-value promises.

The damage depends on what you cannot see

Is a larger cut-off safer? The next figure repeats the fold-change filter at several cut-offs and adds a second scenario, simulated the same way except that only 5% of genes truly change.

de_frac_low <- 0.05                          # a second scenario: fewer genes truly change
set.seed(2027)
sim_low <- simulate(n_genes * n_exp, de_frac_low, n_genes)
null_share_fc_low <- numeric(n_exp)
fdp_low <- sapply(seq_len(n_exp), function(e) {
  i <- which(experiment == e)
  ed <- run_edger(sim_low$counts[i, ], fc_cut)
  null_share_fc_low[e] <<- mean(!sim_low$is_de[i][ed$keep & abs(ed$lfc) > fc_cut])
  sapply(thresholds, function(t)
    score(bh_on(ed$p, ed$keep & abs(ed$lfc) > t), sim_low$is_de[i])[["fdp"]])
})
sweep_low <- rowMeans(fdp_low)
se_low <- apply(fdp_low, 1, sd) / sqrt(n_exp)   # Monte Carlo standard error

lv <- c(sprintf("DESeq2, %s of genes changed", pct(de_frac, 0)),
        sprintf("edgeR, %s of genes changed", pct(de_frac, 0)),
        sprintf("edgeR, %s of genes changed", pct(de_frac_low, 0)))
sw <- data.frame(threshold = rep(thresholds, 3),
                 fdr = c(sweep["deseq", ], sweep["edger", ], sweep_low),
                 analysis = factor(rep(lv, each = length(thresholds)), levels = lv))
ggplot(sw, aes(threshold, fdr, colour = analysis)) +
  geom_hline(yintercept = fdr_level, linetype = "dashed", colour = "#c9c7d4") +
  geom_line(linewidth = 0.8) +
  geom_point(size = 2) +
  scale_colour_manual(values = setNames(c("#5f5c70", "#5a3fc0", "#d14fa6"), lv), name = NULL) +
  scale_y_continuous(labels = function(x) paste0(100 * x, "%")) +
  labs(x = "|log2 fold change| cut-off applied before the correction",
       y = "realised FDR") +
  theme_minimal(base_size = 12) +
  theme(legend.position = "bottom", legend.direction = "vertical",
        panel.grid.minor = element_blank(),
        plot.background = element_rect(fill = "white", colour = "white"))

With no cut-off the analyses sit near the nominal level. The edgeR run in the second scenario is slightly above it, at 6.6% with a Monte Carlo standard error of 0.7% (the uncertainty that comes from simulating only 16 experiments). Any cut-off in the middle of the range pushes the FDR up, and the push is larger when fewer genes truly change, because a larger share of what passes the filter is then null: at a cut-off of 0.5, 73% of the genes passing were null in the second scenario against 46% in the first. In the second scenario the realised FDR peaked at 36.5% at a cut-off of 0.75. At the popular two-fold cut-off (log2 fold change 1), edgeR’s realised FDR was 6.2% in the first scenario and 20.5% in the second. At the largest cut-off, 1.5, it falls to 1.9%.

So the harm from a fold-change pre-filter is not a fixed penalty you could correct for. It depends on how many genes truly change (and, by the same logic, on how large the changes are), which is exactly what the experiment is trying to find out. A cut-off that looks harmless on one data set carries no guarantee on the next.