Interactions, Delta-Deltas, and How We Measure Uncertainty

On delta-deltas, interaction terms, and the bootstrap

A short essay on delta-delta means, regression interaction terms, ANOVA interactions, and why bootstrap confidence intervals are empirically different from model-based uncertainty.

Same Estimate, Different Uncertainty

TL;DR: The interaction can be written as a regression coefficient. That is not the disagreement. The difference is whether uncertainty is presented as a model-based standard error or as a resampled distribution of the effect size.

“Isn’t a delta-delta just an interaction term in disguise?”

People ask this a lot, and the answer is mostly yes, but the uncertainty tells a different story. This post is not an argument against regression. In fact, part of the point is that the regression people are right: in a simple 2×2 ordinary least-squares model, a delta-delta can be written as an interaction coefficient. ANOVA, t-tests, ANCOVA, and regression all live inside the same general linear model family when the predictors and contrasts are set up appropriately. But that equivalence mainly settles the point estimate. It does not settle how we should communicate uncertainty around that estimate.

You calculate a mean. Then you wonder: how precise is this estimate? For a simple mean difference, this question is already important. For a delta-delta, it becomes even more important. A delta-delta asks whether one change is larger than another change: for example, whether disease burden is larger in a treatment group than in a placebo group. The mean estimate itself is just arithmetic. The disagreement, or at least the unfamiliarity, begins with the interval.

In a simple 2×2 design with dummy (genotype) coding, the delta-delta mean is exactly the treatment × group interaction coefficient in a saturated linear model. This equivalence depends on the coding scheme: with effect (sum-to-zero) coding, the interaction coefficient is scaled differently: in a 2×2 design it is one quarter of the delta-delta. The same basic interaction question is also asked by a two-way ANOVA: does the effect of one factor depend on the other? Regression and ANOVA can express the same general linear model structure, but the meaning of coefficients depends on the coding scheme used for categorical variables and interactions.¹

The uncertainty is where the methods part ways. A classical confidence interval usually starts with a mathematical story about the estimate. For a simple mean, that story often relies on the sampling distribution of the mean being approximately normal. For regression coefficients, the usual interval relies on the fitted model: independent observations, a correctly specified linear structure, residual variance, degrees of freedom, and a theoretical reference distribution. These assumptions may be reasonable, but they are still assumptions. A bootstrap confidence interval takes a different route. DABEST resamples the observed data with replacement, recomputes the effect size on each resample, and uses the resulting resampled effect-size distribution to form the confidence interval (with BCa correction applied).² The bootstrap was introduced by Efron as a computational approach to estimating uncertainty from resampling.³

That is why the bootstrap CI feels empirical. It asks: if I repeatedly resample data like the data I actually observed, how much does my estimate wobble? That said, the bootstrap is not assumption-free. BCa correction can be unreliable when samples are very small (n < 10 per group) or when outcomes are highly discrete, because the resampling distribution itself becomes unstable.

This point is important in practice because real biological data often make the usual textbook picture uncomfortable: small sample sizes, skewed distributions, unequal variances, batches, and the occasional outlier that pulls the mean around. The bootstrap does not make these problems disappear, but it changes the question. Instead of deriving uncertainty only from a theoretical standard-error formula, it directly recomputes the statistic on resampled versions of the observed dataset. For well-behaved data with reasonable group sizes (say, n ≥ 20 per cell), bootstrap and model-based CIs are often numerically similar — width ratios near 1.0. The gap opens up meaningfully when data are skewed, variances are unequal across groups, or outliers are present. Those are precisely the conditions where the bootstrap’s empirical approach is most valuable.

The underappreciated part is not that regression cannot do this. It can. You can bootstrap a linear regression coefficient by repeatedly resampling the data, refitting the model, and collecting the interaction beta each time. In a simple 2×2 design, that would give a resampled uncertainty interval for the same contrast. But this is not what most people mean when they say “the regression result.” The default regression table usually gives a coefficient, a standard error, a t statistic, and a model-based confidence interval or P value. DABEST changes what is made central: it computes the effect size directly, resamples the observed data, and shows the bootstrap distribution and confidence interval around that effect size.

So the disagreement is not about whether the interaction can be written as a regression coefficient. It can. The difference is whether uncertainty is presented as a model-based standard error or as a resampled distribution of the effect size.

This fits the broader estimation-statistics argument: report effect sizes and confidence intervals, not only whether a P value crossed a threshold. Gardner and Altman made this argument in “Confidence intervals rather than P values,” and Cumming’s “new statistics” similarly emphasises estimation based on effect sizes, confidence intervals, and meta-analysis.⁴⁵

Same arithmetic in a simple 2×2 design

Method	What you calculate	With dummy (treatment) coding
Delta-delta mean	Difference between two mean differences	The raw interaction estimate
Regression interaction beta	treatment × group coefficient	Exactly equal to the delta-delta
Regression interaction beta	treatment × group coefficient (effect/sum coding)	Delta-delta ÷ 4 — scaled, not equal
ANOVA interaction	Non-additive effect of two factors	Same interaction question, no estimate of size

Different uncertainty machinery

Method	Uncertainty output	How uncertainty is built
Bootstrap delta-delta CI	Empirical/resampling CI	Resample observed data and recompute the full delta-delta
Regression beta CI	Model-based CI	Use model assumptions, a standard error formula, and usually a t distribution
ANOVA interaction P value	Model-based test	Use an F statistic under a no-interaction null

A numerical illustration

The cells below generate a synthetic 2×2 dataset with the following target means:

Group	Placebo	Drug	Difference
WT	10	12	+2
KO	18	10	−8

The delta-delta is (10 − 18) − (12 − 10) = −10.

The DABEST estimation plot, the regression interaction coefficient, and the two-way ANOVA all recover the same point estimate. The confidence intervals and the P value come from different machinery.

A companion Colab notebook extends this with a larger dataset, a messier unbalanced variant, and a side-by-side plot of the bootstrap distribution against the model-based CI:

import numpy as np
import pandas as pd
import dabest
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
import statsmodels.stats.anova as sm_anova

rng = np.random.default_rng(7)
n = 25

df = pd.DataFrame({
    'measurement': np.concatenate([
        rng.normal(10, 2, n),   # WT Placebo
        rng.normal(12, 2, n),   # WT Drug
        rng.normal(18,  2, n),  # KO Placebo
        rng.normal(10, 2, n),   # KO Drug
    ]),
    'genotype':  ['WT']*(n*2) + ['KO']*(n*2),
    'treatment': (['Placebo']*n + ['Drug']*n) * 2,
})

DABEST delta-delta estimation plot. The upper panel shows the raw data and group means. The lower panel shows the two mean differences (WT Drug − WT Placebo; KO Drug − KO Placebo) and their delta — the delta-delta — with a 95% bootstrap BCa confidence interval.

analysis = dabest.load(
    df,
    x=['genotype', 'treatment'],
    y='measurement',
    delta2=True,
    experiment='treatment',
    x1_level=['WT', 'KO'],
    experiment_label=['Placebo', 'Drug'],
)
fig = analysis.mean_diff.plot(
    raw_marker_size=3,
    bar_width=0.25,
    raw_desat=1,
    contrast_desat=1,
    custom_palette={'Placebo': 'steelblue', 'Drug': 'orangered'},
    raw_bars_kwargs={'alpha': 0.1},
    contrast_bars_kwargs={'alpha': 0.1},
    contrast_errorbar_kwargs={'color': 'black', 'lw': 2, 'linestyle': '-', 'zorder': 1},
)
plt.tight_layout()

Regression: interaction coefficient and model-based CI. With dummy (treatment) coding — the default in statsmodels — the interaction coefficient is directly the delta-delta: it estimates how much the Drug effect differs between WT and KO. The CI comes from the model’s standard error and a t distribution.

model = smf.ols('measurement ~ C(genotype) * C(treatment)', data=df).fit()

coef_table = model.summary2().tables[1][['Coef.', '[0.025', '0.975]', 'P>|t|']]
print(coef_table.to_string())

                                               Coef.     [0.025     0.975]         P>|t|
Intercept                                   9.846924   9.145814  10.548035  8.120196e-48
C(genotype)[T.WT]                           1.722020   0.730500   2.713540  8.416897e-04
C(treatment)[T.Placebo]                     8.093328   7.101808   9.084848  3.221129e-29
C(genotype)[T.WT]:C(treatment)[T.Placebo] -10.401834 -11.804055  -8.999613  2.302268e-26

Two-way ANOVA: interaction F test.

anova_table = sm_anova.anova_lm(model, typ=2)
print(anova_table.to_string())

                              sum_sq    df           F        PR(>F)
C(genotype)               302.568140   1.0   97.011765  3.131617e-16
C(treatment)              209.151118   1.0   67.059668  1.129454e-12
C(genotype):C(treatment)  676.238404   1.0  216.820849  2.302268e-26
Residual                  299.412566  96.0         NaN           NaN

The interaction row of the ANOVA table and the interaction coefficient in the regression table both point at the same effect. The bootstrap CI in the DABEST plot is empirically derived from the data; the regression CI is derived from the model’s residual variance and degrees of freedom. In clean, balanced, normally distributed data these will be similar. In practice they can differ — and knowing why they can differ is reason enough to look at both.

When the three stop matching neatly

The equivalence is cleanest for a simple 2×2 ordinary least-squares model with matching contrasts. The answers can diverge when the design is unbalanced, when Type I/II/III sums of squares are chosen differently, when contrast coding changes the coefficient, when covariates are added, when observations are paired or clustered, when mixed effects are needed, when robust standard errors are used, when outcomes are modeled with logistic or Poisson regression, or when effects are estimated on a transformed scale. Statsmodels explicitly exposes Type I/II/III ANOVA choices and robust covariance options, which is a reminder that “the interaction P value” is not a single assumption-free object.⁶

The takeaway is simple: the mean is the estimate; the interval is the story of how much we trust it.

This work was done in collaboration with the DABEST Team. The DABEST 2.0 paper is published in Nature Methods (2026).

Footnotes

1. UCLA Statistical Methods and Data Analytics. “Regression with SPSS Chapter 6: More on Interactions of Categorical Variables.” https://stats.oarc.ucla.edu/spss/webbooks/reg/chapter6/regressionwith-spsschapter-6-more-on-interactions-of-categorical-variablesdraft-version/

2. DABEST. “Bootstrap Confidence Intervals.” https://acclab.github.io/DABEST-python/blog/posts/bootstraps/bootstraps.html

3. Efron, B. (1979). “Bootstrap Methods: Another Look at the Jackknife.” The Annals of Statistics, 7(1), 1–26. https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods-Another-Look-at-the-Jackknife/10.1214/aos/1176344552.short

4. Gardner, M. J., & Altman, D. G. (1986). “Confidence intervals rather than P values: estimation rather than hypothesis testing.” British Medical Journal, 292, 746–750. https://pmc.ncbi.nlm.nih.gov/articles/PMC1339793/

5. Cumming, G. (2014). “The new statistics: why and how.” Psychological Science, 25(1), 7–29. https://pubmed.ncbi.nlm.nih.gov/24220629/

6. Statsmodels. statsmodels.stats.anova.anova_lm documentation. https://www.statsmodels.org/stable/generated/statsmodels.stats.anova.anova_lm.html