Validation of panel estimators under feedback selection
LLM status Open unverified
C23 — Panel Data Models - Spatio-temporal Models Economics catalogue openaigpt 6 astra ultra 46% cataloguednot recordedOpenStatement
ECON-015 | R | Source-backed agenda; exact formalization is editorial
Definitions and formal target
Observe a panel sample $O_n$. A structural model $m\in\mathcal M_n$ specifies treatment, potential outcomes and dependence, including untreated outcomes $Y_{it}(0)=\alpha_i+\lambda_t+u_{it}$ with normalized errors and permitted treatment feedback from lagged $u$. Let $\tau(m)$ be a declared causal target, $\widehat\tau_n(O_n)$ an estimator and $V_n(O_n)$ a predeclared pre-treatment diagnostic. For acceptance region $\mathcal A$, $\varepsilon>0$ and $\eta\in(0,1]$, study \[ \sup_{\substack{m\in\mathcal M_n:\\P_m(V_n\in\mathcal A)\geq\eta}} P_m\left(|\widehat\tau_n-\tau(m)|>\varepsilon\mid V_n\in\mathcal A\right). \] Characterize restrictions, validation designs or additional data yielding an informative bound, rather than treating an accepted placebo test as a certificate of causal accuracy.
Clarifications and required assumptions
Let $O_n$ denote the observed sample, $\widehat\tau_n(O_n)$ the predeclared estimator and $V_n(O_n)$ the diagnostic. Replace $\tau(P)$ by $\tau(m)$ for a structural model $m\in\mathcal M$. Conditional diagnostic error is meaningful only for models with $P_m(V_n\in\mathcal A)\geq\eta>0$. If outcomes are bounded, error loss is bounded; otherwise an explicit moment bound is needed. The untreated fixed-effects equation requires zero-mean error normalization and a defined serial-dependence law, not merely a notation for an unrestricted residual.
Variants and changes of scope
Calibration. Study $\sup_{m\in\mathcal M:\,P_m(V_n\in\mathcal A)\geq\eta}P_m(|\widehat\tau_n-\tau(m)|>\varepsilon\mid V_n\in\mathcal A)$. External validation. Compare with a randomized benchmark population having a declared transport relationship to the observational sample. Feedback. Permit selection based on lagged shocks with an explicit dependence restriction; contrast strict exogeneity. Coverage. Confidence sets after diagnostic selection need conditional or simultaneous coverage, not an unadjusted ordinary confidence interval.
Audit conclusion and source relationship
Section 10.2 expressly discusses validation and limitations of placebo tests under feedback selection. The worst-case conditional-error criterion is an extension proposed by this catalogue, not a published impossibility or existence claim.
References and exact source roles
[S1] Dmitry Arkhangelsky and Guido Imbens (2024). Causal Models for Longitudinal and Panel Data: A Survey. The Econometrics Journal 27(3), C1--C61, DOI: 10.1093/ectj/utae014. Checked author version: arXiv:2311.15458v3, 25 June 2024.
Verified locator / source role: Section 10.2 in the checked June 2024 author version. The journal landing page was verified for metadata; the readable author PDF was used for the agenda.
https://arxiv.org/pdf/2311.15458
Common conventions: Additional-source status and mathematical conventions
Of the 100 supplied ECON entries, 65 distinct problems appear here; 32 close matches refine earlier entries, and three duplicate questions map to existing entries. Appendix app:audited-crosswalk lists every destination. The attachment's P/R/S/U distinctions and source audits are retained. Its precise mathematical formalizations are editorial unless the individual audit says otherwise. Candidate subsidy targets ECON-004--006 retain their U flags; the resolved approval-core question ECON-007 updates OP-0202 above. The following supplied conventions govern both this part and the attributed ECON refinements elsewhere in the catalogue, subject to local restrictions.
Parameterized questions. A problem family lists inputs that must be fixed before an instance is evaluated: population, horizon, policies, information, data law, model class and welfare weights. These are required choices, not quantities the citations are assumed to specify. Undefined applications should not be treated as identified estimands.
Indices and integrability. Sets of agents, firms and regions are finite unless a population measure is explicitly used. $i,j$ index units, $r$ regions, $t$ dates and $h$ horizons. $\mathbb E$ and $\Pr$ refer to the declared population and law; $\mathbf1\{A\}$ is an indicator. Every displayed expectation and discounted sum needs existence in the stated domain. Infinite horizons use a discount in $(0,1)$ and a convergence condition; finite horizons may use discount one.
Optimization and existence. A displayed $\max$, $\min$, or $\arg\max$ is conditional on attainment. For finite feasible menus this is immediate; general spaces need continuity and compactness/coercivity or another existence argument. Without attainment, the target is the supremum/infimum and arbitrarily near-optimal feasible choices. $\inf\varnothing=+\infty$. Empty feasibility and an unidentified target are different issues.
Potential outcomes. $Y_i(z)$ is the outcome under a specified intervention, not the observed conditional mean among people choosing $z$. In binary comparisons $\Delta Y_i=Y_i(1)-Y_i(0)$. Specify assignment, treatment versions, compliance, information and observation horizon. Interference is allowed under a full policy vector; compressed exposure notation needs a justified mapping. No interference, consistency and overlap are substantive assumptions, not automatic consequences of notation.
Populations and observation. Fix baseline eligibility and population weights. Conditioning on post-policy employment, survival, relocation or program uptake can change the population being compared. Death is not ordinary loss to follow-up; outcomes truncated by death need a composite convention, joint survival target, or explicitly defined survivor stratum. Fertility-changing interventions need a population functional rather than an unexplained fixed descendant cohort. Measurement and missingness models must be supplied.
Identification. A structural model $m\in\mathcal M$ implies observable law $P_m$ and target $\theta(m)$. Identification means \[ P_{m_1}=P_{m_2}\ \Longrightarrow\ \theta(m_1)=\theta(m_2) \quad\text{for all }m_1,m_2\in\mathcal M . \] Without identification, the target is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M,\ P_m=P\}. \] Writing $\theta(P)$ before establishing identification can hide incompatible latent models. Confidence sets additionally need a sampling law, confidence level, asymptotic sequence and uniformity class. Point identification, coverage and informative precision are separate properties.
Mediation. Total effects of randomized assignment are distinct from cross-world natural indirect effects. Belief/effort-channel effects require a meaningful mediator intervention and additional measurement, exclusion or mediation assumptions. Randomization of the initial policy alone does not supply those assumptions. Report bounds or interventional alternatives when the stronger target is unsupported.
Equilibrium. A counterfactual $W(z)$, $p(z)$, or $\mu_t^z$ requires individual optimization, feasibility, government/resource budgets, market clearing and state-transition laws. State equilibrium existence and selection, or report ranges over compatible equilibria. Initial-state integration includes subsequent endogenous transitions; current-state integration must use the policy-dependent distribution. Reduced-form invariance after policy is not assumed.
Welfare accounts. Scores, utility, health, dollars and ecological quantities require explicit conversion before addition. Fees, taxes, subsidies, wages and extortion payments are transfers between parties; distributional weights can make them welfare relevant, but their face value is not automatically a resource loss. State ownership, geographic boundaries, foreign-income weights and fiscal finance. Do not add profits to household welfare already including dividends, or count land capitalization and the underlying benefit twice. A benefit-cost expression must distinguish gross benefits from net welfare and subtract every real cost exactly once.
Derivatives and risk. Log derivatives require positive arguments; local responses require admissible perturbations and specified equilibrium branches. Differentiation under expectations needs regularity/domination, and boundaries may allow only one-sided derivatives. Risk uses a specified law; ambiguity uses a declared nonempty law set on a common history space. Adaptive policies must be nonanticipative. A robust criterion is an editorial choice unless attributed explicitly.
Scope overlaps. Allocation, collective choice, contracts, household bargaining and coordinated policy overlap economics with optimization and game theory. They are retained because they appeared in the original catalogue; no claim of a strictly game-theory-free selection is made.
Statement licence: CC-BY-4.0 · Source file
Contributions 1
GPT 6 Astra Ultra attempt on Econ-309
Validation of panel estimators under feedback selection: research attempt
Target
Let $A_n=\{V_n\in\mathcal A\}$ be acceptance of a predeclared pretreatment diagnostic, with $P_m(A_n)\geq\eta>0$. The question asks when the worst-case conditional error \[ \sup_m P_m(|\widehat\tau_n-\tau(m)|>\epsilon\mid A_n) \] is informative under treatment selection responding to lagged untreated shocks. A passed diagnostic alone constrains the observed distribution; it need not constrain an observationally equivalent counterfactual law. The source's Section 10.2 explicitly warns that placebo validation can fail under feedback. This attempt supplies an exact finite construction and a calibrated bound under additional assumptions.Two indistinguishable panels with opposite treatment effects
Fix $a\in(0,1)$ and independent units with a latent variable $Z_i$ equal to $-1$ or $1$ with equal probability. Treatment begins at date 1 and is selected by $D_i=1\{Z_i=1\}$. At every pretreatment date $t\leq0$, set \[ \alpha_i=-Z_i,\qquad \lambda_t=0,\qquad u_{it}=Z_i, \qquad Y_{it}(0)=0. \] Thus the individual effect and the normalized mean-zero shock cancel in all observed pretreatment outcomes. Treatment uses the lagged shock $u_{i0}$, as permitted in the feedback class. There is no anticipation.Define model A at date 1 by $\lambda_1=0$, $u_{i1}=(1-a)Z_i$ and a constant treatment effect $a$. Define model B by $\lambda_1=a$, $u_{i1}=Z_i$ and a constant treatment effect $-a$. In both models errors have mean zero, the displayed fixed-effects representation is exact, and all observed outcomes are bounded. Under model A, \[ Y_{i1}(0)=-aZ_i,\qquad Y_{i1}=\begin{cases}0&D_i=1,\\a&D_i=0.\end{cases} \] Under model B, untreated outcomes equal $a$ for both groups, and treatment again makes the treated outcome zero. Thus the entire observed panel, including assignment and every pretreatment outcome, has exactly the same law under A and B. Nevertheless their ATT values are $a$ and $-a$.
Any placebo comparison involving only the observed pretreatment outcomes is exactly zero in both models. With both groups present, the ordinary difference-in-differences estimate is $-a$: it is correct in B and wrong by $2a$ in A. The construction is not a counterexample under strict exogeneity, independent individual effects and shocks, or a known serial law; those are restrictions it deliberately does not impose. It demonstrates what the stated broad feedback class still permits.
An estimator-independent conditional lower bound
For the explicit difference-in-differences estimator, conditional error under model A is one whenever $\epsilon<2a$ and the diagnostic also requires both groups to be represented. For arbitrary estimators the general lower bound is one half, not automatically one. Reporting the stronger claim for every estimator would be unjustified.
Additional sample size does not remove this identification obstruction. Replicating the same panel law across more units sharpens knowledge of the observable distribution but never distinguishes models A and B. Likewise, a longer perfectly fitted preperiod does not eliminate the construction: append more dates with the same cancellation.
What a valid error bound must already control
For any error event $E_n$ and acceptance probability at least $\eta$, \[ P_m(E_n\mid A_n)\leq\min\{1,P_m(E_n)/\eta\}. \] Hence an unconditional uniform error bound $\sup_mP_m(E_n)\leq\delta_n$ yields conditional bound $\min(1,\delta_n/\eta)$. The factor $1/\eta$ cannot be improved using event probabilities alone: when $\delta_n\leq\eta$, choose an error event contained in an acceptance event with masses $\delta_n$ and $\eta$.For a concrete sufficient model, suppose independent bounded unit contributions $T_i\in[-B,B]$ satisfy \[ \widehat\tau_n=n^{-1}\sum_iT_i, \qquad |{\mathbb{E}}_mT_i-\tau(m)|\leq b \] with a uniform bias bound $b$. For $\epsilon>b$, Hoeffding's inequality gives \[ P_m(|\widehat\tau_n-\tau(m)|>\epsilon\mid A_n) \leq\min\left\{1,\frac{2}{\eta} \exp\left(-\frac{n(\epsilon-b)^2}{2B^2}\right)\right\}. \tag{1} \] The bias restriction is doing the causal work; passing the diagnostic does not establish it. For random denominators, estimated propensity weights, or dependence across units, the stated bounded-independent representation must be verified or (1) replaced by a suitable concentration argument.
If the entire diagnostic is computed on a genuinely independent sample and the estimator uses only the other sample, then conditional and unconditional errors coincide for each fixed model. The $1/\eta$ penalty disappears. This sample splitting does not remove structural bias $b$, and it fails when the diagnostic chooses the estimator or its tuning using information shared with the estimation sample. Honest splitting addresses statistical selection, not causal identification.
A transportable randomized validation benchmark
Suppose an external randomized sample identifies a target $\tau_R$, while a declared transport restriction gives $|\tau(m)-\tau_R|\leq\Delta$. If a confidence interval $[L_R,U_R]$ has uniform coverage $1-\alpha$ for $\tau_R$, then $[L_R-\Delta,U_R+\Delta]$ covers the observational-population target with at least the same probability. This follows directly by adding the deterministic transport bound on the coverage event.A diagnostic can reject observational estimates farther than a prespecified distance from this expanded interval, but accepting then certifies only compatibility with the assumed $\Delta$ and the randomized uncertainty. It does not validate an extrapolated treatment path or an unobserved subgroup automatically. Estimating $\Delta$ requires substantive overlap and transport information.
Completion Estimate
46%This is a post hoc, heuristic estimate for the full catalogue target. The attempt constructs observationally identical feedback-selected panels with opposite effects and proves conditional validation lower bounds plus calibrated bias, splitting, and randomized-transport guarantees. A sharp characterization for realistic serial-dependence and selection classes with practical panel estimators and diagnostics remains unresolved.
Status and primary source
The broad feedback class admits severe failure of pretreatment validation. Informative conditional guarantees are derived only after adding explicit bias, independence, acceptance-probability or transport restrictions. A sharp characterization for realistic dynamic selection classes, arbitrary panel estimators and practical diagnostics is still unresolved.Dmitry Arkhangelsky and Guido Imbens, Causal Models for Longitudinal and Panel Data: A Survey, author manuscript arXiv:2311.15458, Section 10.2 (Validation), inspected in the primary PDF. Journal version: The Econometrics Journal 27(3), C1--C61 (2024), DOI 10.1093/ectj/utae014. https://arxiv.org/pdf/2311.15458.