we-solve (beta)

Econ-159 · economics · Economics

Multiplicity and power in adaptive platform trials

LLM status Open unverified

Statement

CI-05 |  Econometric theory and causal inference

Research agenda formalization

Definitions and assumptions

Use a filtered platform experiment with bounded outcomes and known, positive, predictable allocation probabilities. Arm $j$ enters at a stopping time $\tau_j$; its hypothesis and target are fixed using $\mathcal F_{\tau_j}$ before any observations used to test that hypothesis. In one explicit model class, the conditional mean difference between arm $j$ and its contemporaneous comparator is a constant $\theta_j$ on every eligible post-entry history, although the comparator mean can drift. The null is $H_j:\theta_j\leq0$.

Let $R_t$ be the number of rejected hypotheses by time $t$ and $V_t$ the number of rejected true nulls, counting each hypothesis once. Arbitrarily many arms may enter, and statistics may be dependent through shared comparators.

Formal question

Characterize the achievable power--information tradeoff among procedures satisfying strong, time-uniform familywise error control \[ \sup_{P\in\mathcal M}\Pr_P\{\exists t\geq1:V_t\geq1\} \leq\alpha. \] For specified alternatives $\theta_j\geq\delta_j>0$, require rejection probabilities at least $1-\beta_j$ after an explicitly quantified amount of eligible information; otherwise the rule rejecting nothing would satisfy the problem vacuously. Determine matching lower bounds and the gains obtainable by exploiting shared-comparator dependence. A separate variant replaces the displayed guarantee by $\sup_t\E_P[V_t/(R_t\vee1)]\leq\alpha$; this is a different error criterion.

Scope and status

Generic error spending already supplies conservative familywise-valid procedures. The agenda refinement concerns efficiency and assumptions permitting stronger methods. Data-dependent hypothesis entry does not license recycling the selection outcomes without conditional validity or selection correction. The source's multiple-comparison challenge is broader than the concrete constant-effect subclass formulated here.

References for the source of the problem

Carlos Cinelli, Avi Feller, Guido Imbens, Edward Kennedy, Sara Magliacane, and Jose Zubizarreta, Challenges in Statistics: A Dozen Challenges in Causality and Causal Inference, arXiv:2508.17099v1, 23 August 2025, Section 3.1.3, paragraph Platform trials. https://arxiv.org/html/2508.17099v1.

Common conventions: Source mathematical conventions

Well-defined objects. All probability spaces and measurable variables are specified or inherited from a local statistical model. Expectations used as finite targets require the stated integrability. A decision rule or estimator is measurable with respect to its available information; randomized rules may use an auxiliary independent random variable. Norms without a subscript are Euclidean in finite-dimensional spaces. An optimizer is requested only when attainment is assured; otherwise the question concerns the optimal value and $\varepsilon$-optimal admissible rules for every $\varepsilon>0$.

Identification. A structural model $m\in\mathcal M$ implies an observable law $P_m^{\rm obs}$ and target $\theta(m)$. The target is point identified on $\mathcal M$ if \[ P_{m_1}^{\rm obs}=P_{m_2}^{\rm obs} \quad\Longrightarrow\quad\theta(m_1)=\theta(m_2) \qquad(m_1,m_2\in\mathcal M). \] When this fails, the sharp identified set is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M, P_m^{\rm obs}=P\}. \] Specifying an intervention or estimand does not establish identification. Positivity, exclusion, causal sufficiency, measurement restrictions, and transport assumptions are substantive local inputs, never automatic consequences of notation.

Minimax risk and nontrivial inference. For a statistical experiment with data law $P^{(n)}$, target $\theta(P)$, and nonnegative loss $\ell$, define \[ \mathcal R_n(\mathcal P,\ell) =\inf_{\widehat\theta_n}\sup_{P\in\mathcal P} \E_{P^{(n)}}\ell(\widehat\theta_n,\theta(P)). \] The infimum is over measurable estimators. A sharp rate is a sequence $r_n$ for which $0<c\leq C<\infty$, independent of $n$, satisfies $c r_n\leq\mathcal R_n\leq C r_n$ for all sufficiently large $n$. Squared-error parametric risk is of order $n^{-1}$; the corresponding root-mean-squared error is of order $n^{-1/2}$. These different conventions are distinguished locally. A uniform asymptotic confidence region $C_n$ must satisfy \[ \liminf_{n\to\infty}\inf_{P\in\mathcal P} \Pr_{P^{(n)}}\{\theta(P)\in C_n\}\geq 1-\alpha, \qquad 0<\alpha<1, \] together with an informative diameter or expected-width requirement. Coverage alone permits the vacuous whole parameter space. Testing questions similarly impose both size control and power against a declared separated alternative class.

Binary causal baseline. When an entry explicitly invokes this baseline, one observes independent copies of $O=(X,W,Y)$, with $W\in\{0,1\}$ and potential outcomes $Y(0),Y(1)$. Assume consistency $Y=Y(W)$, no interference, conditional exchangeability $(Y(0),Y(1))\perp W\mid X$, and overlap $\epsilon\leq e(X)\leq1-\epsilon$ almost surely for a fixed $0<\epsilon<1/2$, where \[ e(x)=\Pr(W=1\mid X=x),\qquad m_w(x)=\E[Y\mid W=w,X=x],\qquad \tau(x)=m_1(x)-m_0(x). \] These assumptions identify conditional mean effects almost everywhere under the covariate law. Pointwise or uniform effects require appropriate regularity and a specified support. Continuous treatment, longitudinal data, adaptive experiments, and network interference require their own assumptions in place of this baseline. Bounded outcomes, densities, and smoothness are additional assumptions only where stated.

Function classes. On $[0,1]^d$, $\mathcal H_d^s(L)$ is the isotropic Hölder ball of radius $L>0$: put $k=\lceil s\rceil-1$ and $\gamma=s-k\in(0,1]$, bound derivatives through order $k$, and require the order-$k$ derivatives to be $\gamma$-Hölder with constant at most $L$. Thus integer orders use a Lipschitz final derivative convention. The class $\operatorname{BV}_d(L)$ consists of functions $f\in L^1((0,1)^d)$ whose distributional derivative is a finite vector measure and for which $\|f\|_{L^1}+|Df|((0,1)^d)\leq L$. An $L^\infty$ bound or a particular pointwise version is imposed separately when needed. Sparse and neural-network classes require a declared dictionary or architecture and coefficient bounds.

An anisotropic Hölder class with declared block smoothness indices means coordinatewise Hölder regularity in each block, uniformly in the other blocks, with all corresponding derivative and Hölder bounds at most the stated radius. Derivative orders and the integer-order convention in each block are those just given. Mixed-derivative bounds are additional restrictions only when explicitly stated.

Decision and computational questions. Policy regret compares admissible feasible policies under the same law and constraints. In computational entries, finite data and thresholds have rational encodings; running time is measured in their total bit length. Real-valued equilibrium search uses the explicitly declared algebraic or FIXP encoding. An approximation ratio for maximization is written $\operatorname{OPT}/\operatorname{ALG}$ and for minimization as $\operatorname{ALG}/\operatorname{OPT}$, with zero optima and infeasible instances handled locally. Historical approximation bounds are not automatically current bounds.

Statement licence: CC-BY-4.0 · Source file

Contributions 1

Econ-159-A1 · Exploration · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-159

Multiplicity and power in adaptive platform trials Conditional entry, eligible information, and dependence-sensitive benchmarks

Problem ID: C12-156. Status: Unresolved; rigorous restricted results.

Short recap and precise scope

The problem asks for strong familywise error control at every time, together with nonvacuous power after a quantified amount of information, even when arms enter adaptively and share controls. We establish conditional validity and an explicit bounded-score power bound, a multiplicity lower bound in a separate independent-stream submodel, and a bounded shared-control family where one simultaneous boundary replaces a multiplicity penalty. These benchmarks do not characterize the general efficiency frontier.

Cinelli et al. [challenge], Section 3.1.3, identify multiple comparisons as one of several platform-trial challenges. Our proofs below concern the catalogue's constant conditional-effect formulation. Exponential supermartingale testing and error spending are standard tools; we supply all arguments needed to handle adaptive entry rather than claim these tools themselves as new.

The filtration and an eligible observation clock

Let $(\mathcal F_t)_{t\geq0}$ contain every platform observation and randomization revealed by time $t$. At a stopping time $\tau_j$, the platform fixes arm $j$, a contemporaneous comparator, a target effect, and a testing protocol. Index the countably many entrants by $j\geq1$, with deterministic tie-breaking if several enter together. Eligibility at a later time is decided before that time's allocation and outcome. Within an eligible observation, treatment label $A$ has known predictable probabilities $p_j,p_0>0$ for the arm and comparator. Other allocations are allowed. Outcomes satisfy $0\leq Y\leq1$ and \[ {\mathbb{E}}[Y\mid\mathcal F_{t-1},A=j] -{\mathbb{E}}[Y\mid\mathcal F_{t-1},A=0]=\theta_j. \tag{1} \] The comparator mean may depend arbitrarily on the past. Conditional on the entry history, $\theta_j$ is fixed; the truth event $\{\theta_j\leq0\}$ is entry-measurable in this formulation. For each eligible observation define \[ X_{j,s}=\frac{{\mathbf 1}\{A=j\}Y}{p_j} -\frac{{\mathbf 1}\{A=0\}Y}{p_0}. \] Here $s$ is arm $j$'s eligible-observation index and the filtration at that observation includes the entire intervening platform history. Assume an entry-known constant $c_j<\infty$ bounds $p_j^{-1}+p_0^{-1}\leq c_j$ on every eligible history. This is an additional quantitative overlap bound. Positivity alone supplies no uniform finite information guarantee. Equation (1) gives conditional mean $\theta_j$ and a conditional support interval of length at most $c_j$ for $X_{j,s}$.
Lemma (Conditional exponential bound): If a random variable $X$ has conditional mean $\mu$ and conditional support in an interval of length $c$, then for real $\lambda$, \[ {\mathbb{E}}[\exp\{\lambda(X-\mu)\}\mid\mathcal G] \leq\exp(\lambda^2c^2/8). \] Consequently, for an entry-chosen $\lambda_j>0$, \[ E_{j,s}=\exp\left\{\lambda_j\sum_{v=1}^s X_{j,v} -s\lambda_j^2c_j^2/8\right\},\quad E_{j,0}=1, \tag{2} \] is a nonnegative supermartingale under $\theta_j\leq0$, conditional on that arm's entry history. Hold it constant between eligible observations.
Proof: For a fixed conditional law let $g(\lambda)=\log{\mathbb{E}} e^{\lambda(X-\mu)}$. Boundedness permits differentiation and gives $g(0)=g'(0)=0$. The second derivative is the variance under exponential tilting, still supported on an interval of length $c$. For any variable $V$ in $[a,a+c]$, its variance is at most ${\mathbb{E}}(V-a-c/2)^2\leq c^2/4$. Thus $g''\leq c^2/4$, and integration twice proves the inequality for either sign of $\lambda$. Apply it to each eligible increment, whose conditional mean is $\theta_j\leq0$; the conditional expected multiplier in (2) is at most $e^{\lambda_j\theta_j}\leq1$. This uses the global pre-observation history, not merely the history of that arm. Holding a process constant at other times preserves the supermartingale property. ∎

Strong error control and a power guarantee

Choose an entry-measurable error allocation $a_j\in(0,1)$ such that $\sum_j a_j\leq\alpha$ almost surely. Such a rule may spend the remaining budget adaptively but never exceed it. Reject arm $j$ on its first crossing $E_{j,s}\geq1/a_j$, and count the hypothesis only once.
Theorem (Conditional entry spending): This procedure satisfies strong, time-uniform familywise error control: \[ \Pr\{\text{some true null is ever rejected}\}\leq\alpha. \] Suppose arm $j$ has entry-specified alternative $\theta_j\geq\delta_j>0$ and its first $m_j$ eligible observations occur almost surely. Put $A_j=\log(1/a_j)$, $B_j=\log(1/\beta_j)$ and choose at entry \[ \lambda_j=\sqrt{\frac{8A_j}{m_jc_j^2}},\qquad m_j\geq\frac{c_j^2}{2\delta_j^2} (\sqrt{A_j}+\sqrt{B_j})^2. \tag{3} \] Then its conditional probability of rejection by that eligible count is at least $1-\beta_j$. The rule remains valid at all earlier and later times.
Proof: Condition on an entry history under its null. For a finite horizon $T$, stop (2) at its first crossing or at $T$. Iterated conditional expectations for this bounded stopping time give expectation at most one. Its value on a crossing is at least $1/a_j$, hence crossing probability is at most $a_j$. Increasing $T$ proves the same for any crossing. This argument also works on a null-truth event measurable at entry; multiplication by its indicator and conditional expectation give \[ \Pr\{H_j\text{ true and rejected}\} \leq{\mathbb{E}}[{\mathbf 1}\{H_j\text{ true}\}a_j]. \] Countable union and monotone convergence bound the probability of any false rejection by ${\mathbb{E}}\sum_j a_j\leq\alpha$. Arbitrary shared-control dependence is allowed throughout this calculation.

For power, apply the lemma to $-(X_{j,s}-\theta_j)$ and iterate to obtain \[ \Pr\left\{\sum_{s=1}^mX_{j,s}<m\theta_j-u \mid\mathcal F_{\tau_j}\right\} \leq e^{-2u^2/(mc_j^2)}. \] The first $m_j$ eligible observations are well-defined almost surely by assumption. With probability at least $1-\beta_j$, their sum is at least $m_j\delta_j-c_j\sqrt{m_jB_j/2}$. The crossing threshold at that count is \[ A_j/\lambda_j+m_j\lambda_jc_j^2/8 =c_j\sqrt{m_jA_j/2}. \] Inequality (3) makes the lower bound on the sum at least this threshold. Crossing at that count entails rejection by that count. All the calculations are conditional on the entry choices, proving the stated power assertion. ∎

This theorem gives a prespecified-horizon design with monitoring at every time, not an effect-size-adaptive optimal boundary. The information proxy $m_j/c_j^2$ can be conservative when allocation probabilities are small. Also, a count of eligible observations is not calendar time: an arm that stops receiving eligible observations need never attain the power horizon. Testing outcomes must occur strictly after selection of the hypothesis; the proof supplies no license to reuse the observations that selected it.

A multiplicity cost that a universal rule cannot avoid

For a lower bound take $K$ independent experimental streams. In each stream there are $m$ independent Bernoulli observations, with an exactly known comparator mean $1/2$. Under $P_0$ all means equal $1/2$; under $P_j$ only stream $j$ has mean $1/2+\delta$, with $0<\delta\leq1/4$. This can also be embedded in a trial with extra independent comparator observations whose distribution is identical under every law. These extra observations supply no likelihood information in the comparison below.
Theorem (Uniform-power multiplicity lower bound): Let an arbitrary, possibly randomized procedure based on these observations have familywise error at most $\alpha$ under $P_0,P_1,\ldots,P_K$ and reject the active stream with probability at least $1-\beta$ under each $P_j$. If $q=1-\beta-\alpha>0$, then \[ 4m\delta^2\geq q\log(K/\alpha)-\log2. \tag{4} \] The conclusion also applies to any time-uniform procedure evaluated at this fixed information horizon.
Proof: Let $A_j$ be the event that the rejection set is exactly $\{j\}$. Under $P_j$, failure to reject $j$ has probability at most $\beta$ and rejection of any other stream has probability at most $\alpha$. Thus $P_j(A_j)\geq q$. Under $P_0$ these events are disjoint and all imply some false rejection, so $\sum_jP_0(A_j)\leq\alpha$. At least one $j$ has $P_0(A_j)\leq\alpha/K$.

For any event with probabilities $v$ and $u$ under $P,Q$, conditional Jensen on the event and its complement gives ${\operatorname{KL}}(P,Q)\geq\operatorname{kl}(v,u)$, with the usual endpoint limits. Moreover \[ \operatorname{kl}(v,u) =v\log(1/u)+(1-v)\log(1/(1-u)) +v\log v+(1-v)\log(1-v) \geq v\log(1/u)-\log2. \] For the chosen event this is at least $q\log(K/\alpha)-\log2$. Only $m$ observations change law between $P_j$ and $P_0$. Their one-draw KL divergence is bounded by their chi-square divergence $4\delta^2$: $\log x\leq x-1$ proves ${\operatorname{KL}}(P,Q)\leq\chi^2(P,Q)$ directly. Independence adds KL divergences, giving ${\operatorname{KL}}(P_j,P_0)\leq4m\delta^2$. Independent algorithmic randomization changes neither bound. This proves (4). ∎

Thus simultaneous power over unrestricted alternatives has a logarithmic multiplicity obstruction even if the comparator mean is known perfectly. Equation (4) matches the logarithmic dependence of (3) under equal spending, up to constants and the score information proxy. It does not prove optimality of inverse-probability scores, and it is not a lower bound asserting that all strongly dependent scientific models must pay $\log K$.

A shared-comparator family with no multiplicity factor

To exhibit why additional structure matters, suppose treatment-arm means $m_j\in[0,1]$ are known constants, possibly distinct, while every arm is compared with the same sequence of independent outcomes $C_s\in[0,1]$ with common unknown mean $\mu$. Arm effects are $\theta_j=m_j-\mu$. Known means are a strong restriction, for example a model with deterministic calibrated treatment responses. All arms in this finite family are declared before the comparator sequence is used.
Proposition (One boundary for an ordered null family): Fix $\lambda>0$ and reject $j$ once \[ \exp\{\lambda\textstyle\sum_{s=1}^t(m_j-C_s)-t\lambda^2/8\} \geq1/\alpha. \tag{5} \] For any number of arms this controls the probability of ever rejecting a true null by $\alpha$. If $m_j-\mu\geq\delta$, its rejection probability by $m$ is at least $1-\beta$ whenever \[ m\geq\frac{(\sqrt{\log(1/\alpha)}+ \sqrt{\log(1/\beta)})^2}{2\delta^2}, \qquad\lambda=\sqrt{8\log(1/\alpha)/m}. \]
Proof: For every true null $m_j\leq\mu$, the left side of (5) at each time is at most $\exp\{\lambda\sum_{s=1}^t(\mu-C_s)-t\lambda^2/8\}$. The latter is one nonnegative supermartingale starting at one, so the stopped-process argument above bounds its ever-crossing probability by $\alpha$. This pathwise domination controls all true nulls at once. The power proof is (3) with score $m_j-C_s$, support length one, and error allocation $\alpha$; its conditional mean is $m_j-\mu$. ∎

The gain is a consequence of the one-dimensional ordered family of null means plus the common comparator. Shared controls alone, with unknown unrestricted treatment means, do not create that structure. The general problem is to quantify intermediate gains with estimated arm means, adaptive allocations, and delayed entry while retaining conditional validity. That frontier, and the separate false-discovery-rate variant, remain unresolved here.

References

  • [challenge] C. Cinelli, A. Feller, G. Imbens, E. Kennedy, S. Magliacane and J. Zubizarreta (2025), Challenges in Statistics: A Dozen Challenges in Causality and Causal Inference, arXiv:2508.17099v1, Section 3.1.3, Platform trials. https://arxiv.org/html/2508.17099v1#S3.SS1.SSS3.

Completion Estimate

45%

Go to contribution →