GPT 6 Astra Ultra attempt on Econ-309

Validation of panel estimators under feedback selection: research attempt

Target

Let $A_n=\{V_n\in\mathcal A\}$ be acceptance of a predeclared pretreatment diagnostic, with $P_m(A_n)\geq\eta>0$. The question asks when the worst-case conditional error \[ \sup_m P_m(|\widehat\tau_n-\tau(m)|>\epsilon\mid A_n) \] is informative under treatment selection responding to lagged untreated shocks. A passed diagnostic alone constrains the observed distribution; it need not constrain an observationally equivalent counterfactual law. The source's Section 10.2 explicitly warns that placebo validation can fail under feedback. This attempt supplies an exact finite construction and a calibrated bound under additional assumptions.

Two indistinguishable panels with opposite treatment effects

Fix $a\in(0,1)$ and independent units with a latent variable $Z_i$ equal to $-1$ or $1$ with equal probability. Treatment begins at date 1 and is selected by $D_i=1\{Z_i=1\}$. At every pretreatment date $t\leq0$, set \[ \alpha_i=-Z_i,\qquad \lambda_t=0,\qquad u_{it}=Z_i, \qquad Y_{it}(0)=0. \] Thus the individual effect and the normalized mean-zero shock cancel in all observed pretreatment outcomes. Treatment uses the lagged shock $u_{i0}$, as permitted in the feedback class. There is no anticipation.

Define model A at date 1 by $\lambda_1=0$, $u_{i1}=(1-a)Z_i$ and a constant treatment effect $a$. Define model B by $\lambda_1=a$, $u_{i1}=Z_i$ and a constant treatment effect $-a$. In both models errors have mean zero, the displayed fixed-effects representation is exact, and all observed outcomes are bounded. Under model A, \[ Y_{i1}(0)=-aZ_i,\qquad Y_{i1}=\begin{cases}0&D_i=1,\\a&D_i=0.\end{cases} \] Under model B, untreated outcomes equal $a$ for both groups, and treatment again makes the treated outcome zero. Thus the entire observed panel, including assignment and every pretreatment outcome, has exactly the same law under A and B. Nevertheless their ATT values are $a$ and $-a$.

Any placebo comparison involving only the observed pretreatment outcomes is exactly zero in both models. With both groups present, the ordinary difference-in-differences estimate is $-a$: it is correct in B and wrong by $2a$ in A. The construction is not a counterexample under strict exogeneity, independent individual effects and shocks, or a known serial law; those are restrictions it deliberately does not impose. It demonstrates what the stated broad feedback class still permits.

An estimator-independent conditional lower bound

For the explicit difference-in-differences estimator, conditional error under model A is one whenever $\epsilon<2a$ and the diagnostic also requires both groups to be represented. For arbitrary estimators the general lower bound is one half, not automatically one. Reporting the stronger claim for every estimator would be unjustified.

Additional sample size does not remove this identification obstruction. Replicating the same panel law across more units sharpens knowledge of the observable distribution but never distinguishes models A and B. Likewise, a longer perfectly fitted preperiod does not eliminate the construction: append more dates with the same cancellation.

What a valid error bound must already control

For any error event $E_n$ and acceptance probability at least $\eta$, \[ P_m(E_n\mid A_n)\leq\min\{1,P_m(E_n)/\eta\}. \] Hence an unconditional uniform error bound $\sup_mP_m(E_n)\leq\delta_n$ yields conditional bound $\min(1,\delta_n/\eta)$. The factor $1/\eta$ cannot be improved using event probabilities alone: when $\delta_n\leq\eta$, choose an error event contained in an acceptance event with masses $\delta_n$ and $\eta$.

For a concrete sufficient model, suppose independent bounded unit contributions $T_i\in[-B,B]$ satisfy \[ \widehat\tau_n=n^{-1}\sum_iT_i, \qquad |{\mathbb{E}}_mT_i-\tau(m)|\leq b \] with a uniform bias bound $b$. For $\epsilon>b$, Hoeffding's inequality gives \[ P_m(|\widehat\tau_n-\tau(m)|>\epsilon\mid A_n) \leq\min\left\{1,\frac{2}{\eta} \exp\left(-\frac{n(\epsilon-b)^2}{2B^2}\right)\right\}. \tag{1} \] The bias restriction is doing the causal work; passing the diagnostic does not establish it. For random denominators, estimated propensity weights, or dependence across units, the stated bounded-independent representation must be verified or (1) replaced by a suitable concentration argument.

If the entire diagnostic is computed on a genuinely independent sample and the estimator uses only the other sample, then conditional and unconditional errors coincide for each fixed model. The $1/\eta$ penalty disappears. This sample splitting does not remove structural bias $b$, and it fails when the diagnostic chooses the estimator or its tuning using information shared with the estimation sample. Honest splitting addresses statistical selection, not causal identification.

A transportable randomized validation benchmark

Suppose an external randomized sample identifies a target $\tau_R$, while a declared transport restriction gives $|\tau(m)-\tau_R|\leq\Delta$. If a confidence interval $[L_R,U_R]$ has uniform coverage $1-\alpha$ for $\tau_R$, then $[L_R-\Delta,U_R+\Delta]$ covers the observational-population target with at least the same probability. This follows directly by adding the deterministic transport bound on the coverage event.

A diagnostic can reject observational estimates farther than a prespecified distance from this expanded interval, but accepting then certifies only compatibility with the assumed $\Delta$ and the randomized uncertainty. It does not validate an extrapolated treatment path or an unobserved subgroup automatically. Estimating $\Delta$ requires substantive overlap and transport information.

Completion Estimate

46%

This is a post hoc, heuristic estimate for the full catalogue target. The attempt constructs observationally identical feedback-selected panels with opposite effects and proves conditional validation lower bounds plus calibrated bias, splitting, and randomized-transport guarantees. A sharp characterization for realistic serial-dependence and selection classes with practical panel estimators and diagnostics remains unresolved.

Status and primary source

The broad feedback class admits severe failure of pretreatment validation. Informative conditional guarantees are derived only after adding explicit bias, independence, acceptance-probability or transport restrictions. A sharp characterization for realistic dynamic selection classes, arbitrary panel estimators and practical diagnostics is still unresolved.

Dmitry Arkhangelsky and Guido Imbens, Causal Models for Longitudinal and Panel Data: A Survey, author manuscript arXiv:2311.15458, Section 10.2 (Validation), inspected in the primary PDF. Journal version: The Econometrics Journal 27(3), C1--C61 (2024), DOI 10.1093/ectj/utae014. https://arxiv.org/pdf/2311.15458.