we-solve (beta)

Econ-344 · economics · Economics

External validity and scaling of randomized interventions

LLM status Open unverified

Statement

Original atlas ID: D10 (entry 40). Classification: Editorial transport sensitivity and scale-up problem; includes a known identification formula.

Original atlas question: When does a treatment estimated in one site predict effects elsewhere or at national scale?

Definitions and assumptions

Let $S=1$ denote a study population and $S=0$ a target population, with pretreatment covariates $X\in\mathbb R^p$ for a fixed dimension $p$. Let $F_s$ be the law of $X$ in population $S=s$. A precisely specified binary treatment has potential outcomes $Y(1),Y(0)$ on a common measurement scale. Assume consistency, randomized assignment $D$ within the study, $0<\Pr(D=1\mid X=x,S=1)<1$ for $F_0$-almost-every $x$, and integrable outcomes. For a transport benchmark, assume conditional effect exchangeability, \[ \mathbb E[Y(1)-Y(0)\mid X=x,S=0] =\mathbb E[Y(1)-Y(0)\mid X=x,S=1], \] for $F_0$-almost-every $x$, and measure-theoretic overlap $F_0\ll F_1$ (the target law is absolutely continuous with respect to the study law). Topological support inclusion alone is insufficient when the target has atoms or singular components absent from the study. This is a substantive assumption about effect modifiers and treatment versions, not a consequence of randomization.

Formal research question

Under these assumptions the target average effect is identified by \[ \tau_0=\int\!\left\{\mathbb E[Y\mid D=1,X=x,S=1] -\mathbb E[Y\mid D=0,X=x,S=1]\right\}\,dF_0(x), \] where $F_0$ is the target covariate distribution. The research target is to test supported transport predictions prospectively and derive sharp bounds on $\tau_0$ when exchangeability or overlap is weakened by explicit sensitivity restrictions. For scale-up, define $Y_i(\mathbf D,a,p)$ with coverage vector $\mathbf D$, implementer $a$, and equilibrium prices $p$. Identify the target policy effect under its allocation and price responses, rather than applying the individual-treatment formula to unsupported saturation.

Interpretation and scope

The displayed transport formula is a known identification consequence, not an open theorem. Missing effect modifiers can invalidate it even with flawless within-study randomization. Limited overlap can make estimation unstable before it destroys point identification, so support and precision require separate discussion. Changing implementers, delivery intensity, prices, or political responses changes the treatment or environment. Hierarchical pooling supplies predictions under its exchangeability and prior assumptions; it does not identify national equilibrium effects from unrelated local trials alone. Prospective multisite validation should measure moderators before treatment, specify common outcomes, and account for sampling and publication selection. Uncertainty intervals should include both within-site sampling uncertainty and the stated between-site extrapolation uncertainty.

Source audit and status

[S1] reviews experiments, [S2] critiques interpretation and generalization, and [S3] provides cross-study heterogeneity evidence. They motivate the editorial sensitivity-and-scaling program without posing its exact formulas as unresolved conjectures. The atlas's question is therefore a collection of conditional empirical and methodological targets; no universal failure or universal 2026 open status is inferred.

References

    },leftmargin=*]
  1. Abhijit V. Banerjee and Esther Duflo (2009). The Experimental Approach to Development Economics. Annual Review of Economics 1, 151--178. https://doi.org/10.1146/annurev.economics.050708.143235. Locator: Primary publisher, working-paper, or author record: bibliographic information and abstract; full-text theorem verification is not claimed. Role: Review of experimental research strengths and limitations.
  2. Angus Deaton and Nancy Cartwright (2018). Understanding and Misunderstanding Randomized Controlled Trials. Social Science & Medicine 210, 2--21. https://www.sciencedirect.com/science/article/pii/S0277953617307359. Locator: Primary publisher, working-paper, or author record: bibliographic information and abstract; full-text theorem verification is not claimed. Role: Methodological analysis of inference, transport, and causal explanation.
  3. Eva Vivalt (2020). How Much Can We Generalize from Impact Evaluations?. Journal of the European Economic Association 18(6), 3045--3089. https://doi.org/10.1093/jeea/jvaa019. Locator: Primary publisher, working-paper, or author record: bibliographic information and abstract; full-text theorem verification is not claimed. Role: Cross-study evidence on effect heterogeneity; motivates prospective transport validation.

Integrated refinement: ECON-018

Transporting micro causal estimates to aggregate policy changes

This refinement adopts the additional source conventions (audited:conventions), subject to its local restrictions. Scaling from micro effects to an equilibrium policy contrast. For coverage $s$, let $p(s)$ be equilibrium prices and wages and let $Y_i(d,p,s)$ include treatment, prices and spillover exposure. Define \[ \Delta(s_1,s_0)=\mathbb E[Y_i(D_i(s_1),p(s_1),s_1) -Y_i(D_i(s_0),p(s_0),s_0)]. \] Identify or bound this nonlocal contrast using declared equilibrium restrictions and additional observations. A local experimental or panel effect at $(p(s_0),s_0)$ does not identify it automatically. Fix the baseline population, assignment/coverage policies, treatment versions, equilibrium closure and selection, and the common welfare/outcome units. Separate price responses, spillovers and implementation changes rather than applying a reweighting formula that holds the environment fixed.

Supplied source audit for ECON-018

Local [S1], [S2], etc. in this audit and the references immediately below belong to ECON-018, independently of the original entry's reference numbering. Section 10.3 supports micro--macro links and the policy-invariance concern. The equilibrium contrast is editorial. The citation is to an inspected readable author version, rather than relying on the inaccessible body of the journal page.

References for integrated refinement ECON-018

[S1] Dmitry Arkhangelsky and Guido Imbens (2024). Causal Models for Longitudinal and Panel Data: A Survey. The Econometrics Journal 27(3), C1--C61, DOI: 10.1093/ectj/utae014. Checked author version: arXiv:2311.15458v3, 25 June 2024.

Verified locator / source role: Section 10.3 in the checked June 2024 author version. The journal landing page was verified for metadata; the readable author PDF was used for the agenda.

https://arxiv.org/pdf/2311.15458

Common conventions: Source mathematical conventions

Unless an entry states otherwise, agent and firm sets are finite. State and action spaces are measurable, strategies and outcome maps are measurable, probability laws are specified, and expectations exist for the targets under discussion. Infinite-horizon discount factors lie in $(0,1)$ and discounted objectives are finite. Norms, tolerances and complexity input sizes must be specified for computational comparisons. Constants or primitives that appear locally are inputs, not empirical facts asserted by this catalogue.

Identification. A statistical model $m\in\mathcal M$ implies an observable law $P_m$ and target $\theta(m)$. At law $P$, its identified set is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M,\ P_m=P\}. \] Point identification holds when this set is a singleton, or equivalently when $P_{m_1}=P_{m_2}$ implies $\theta(m_1)=\theta(m_2)$ for all compatible models. Sharp bounds mean bounds attained, or approached when the boundary is not attained, by compatible models. Writing this set is a definition, not a solution: the substantive task is to establish informative restrictions and derive or estimate the set. An unrestricted-confounding model may give only trivial bounds.

Inference. Identification, estimability and valid uncertainty quantification are separate questions. Fix $\alpha\in(0,1)$. A confidence procedure $C_n$ over a declared law class $\mathcal P$ has asymptotic uniform coverage at level $1-\alpha$ when \[ \liminf_{n\to\infty}\inf_{P\in\mathcal P}\Pr_P\{\theta(P)\in C_n\}\geq1-\alpha. \] For set-identified targets the coverage event must be defined separately, for example coverage of the full identified set. If an entry uses identification-indexed models instead of uniquely indexed laws, the same coverage convention is applied over those models. Pointwise limits do not establish uniform validity, and asymptotic validity does not guarantee finite-sample precision. Sample sizes, dependence structures and weak-identification sequences are part of each application.

Counterfactuals. $Y_i(a)$ is a potential outcome under a specified intervention, including its timing, implementation and versions. It is not $Y_i$ conditional on voluntarily choosing $a$. A full intervention vector permits interference; shorthand scalar treatment notation does not silently impose no interference. Assignment, selection, attrition, anticipation and measurement must be specified. A claim about an instrument's local effect is not automatically a population policy effect.

Economic equilibrium. A counterfactual model includes best responses, constraints, feasibility, market clearing, state transitions and expectations consistent with the stated information. When several equilibria exist, either a declared selector is an input or the target is set-valued. Comparisons across policy regimes must use the stated selection convention. Reduced-form relationships are not presumed invariant after an equilibrium-changing intervention.

Optimization and welfare. Optimization is over the stated feasible set. If attainment is not established, $\sup$ or $\inf$ and $\varepsilon$-optimal choices replace an asserted maximizer or minimizer. For example, with value $V=\sup_{a\in\mathcal A}W(a)<\infty$, an $\varepsilon$-optimal set is $\{a\in\mathcal A:W(a)\geq V-\varepsilon\}$ for $\varepsilon>0$. Benefits, costs, risk and health or environmental outcomes can be aggregated only using declared units and conversion weights. Interpersonal utility weights, the treatment of fiscal transfers and foreign profits, discounting, and the behavioral welfare criterion are explicit inputs. A comparative-static derivative additionally requires existence and appropriate differentiability of the selected counterfactual.

Sources. Local labels [S1], [S2], and so forth restart in every question. Locators identify the relevant abstract, model, section or result at the level actually checked; a bibliographic or abstract-level verification is not represented as inspection of an entire proof. Working papers are distinguished from later publications and changing versions. Source summaries are paraphrased. All URL checks and editorial formulations refer to the audit date stated above.

Common conventions: Additional-source status and mathematical conventions

Of the 100 supplied ECON entries, 65 distinct problems appear here; 32 close matches refine earlier entries, and three duplicate questions map to existing entries. Appendix app:audited-crosswalk lists every destination. The attachment's P/R/S/U distinctions and source audits are retained. Its precise mathematical formalizations are editorial unless the individual audit says otherwise. Candidate subsidy targets ECON-004--006 retain their U flags; the resolved approval-core question ECON-007 updates OP-0202 above. The following supplied conventions govern both this part and the attributed ECON refinements elsewhere in the catalogue, subject to local restrictions.

Parameterized questions. A problem family lists inputs that must be fixed before an instance is evaluated: population, horizon, policies, information, data law, model class and welfare weights. These are required choices, not quantities the citations are assumed to specify. Undefined applications should not be treated as identified estimands.

Indices and integrability. Sets of agents, firms and regions are finite unless a population measure is explicitly used. $i,j$ index units, $r$ regions, $t$ dates and $h$ horizons. $\mathbb E$ and $\Pr$ refer to the declared population and law; $\mathbf1\{A\}$ is an indicator. Every displayed expectation and discounted sum needs existence in the stated domain. Infinite horizons use a discount in $(0,1)$ and a convergence condition; finite horizons may use discount one.

Optimization and existence. A displayed $\max$, $\min$, or $\arg\max$ is conditional on attainment. For finite feasible menus this is immediate; general spaces need continuity and compactness/coercivity or another existence argument. Without attainment, the target is the supremum/infimum and arbitrarily near-optimal feasible choices. $\inf\varnothing=+\infty$. Empty feasibility and an unidentified target are different issues.

Potential outcomes. $Y_i(z)$ is the outcome under a specified intervention, not the observed conditional mean among people choosing $z$. In binary comparisons $\Delta Y_i=Y_i(1)-Y_i(0)$. Specify assignment, treatment versions, compliance, information and observation horizon. Interference is allowed under a full policy vector; compressed exposure notation needs a justified mapping. No interference, consistency and overlap are substantive assumptions, not automatic consequences of notation.

Populations and observation. Fix baseline eligibility and population weights. Conditioning on post-policy employment, survival, relocation or program uptake can change the population being compared. Death is not ordinary loss to follow-up; outcomes truncated by death need a composite convention, joint survival target, or explicitly defined survivor stratum. Fertility-changing interventions need a population functional rather than an unexplained fixed descendant cohort. Measurement and missingness models must be supplied.

Identification. A structural model $m\in\mathcal M$ implies observable law $P_m$ and target $\theta(m)$. Identification means \[ P_{m_1}=P_{m_2}\ \Longrightarrow\ \theta(m_1)=\theta(m_2) \quad\text{for all }m_1,m_2\in\mathcal M . \] Without identification, the target is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M,\ P_m=P\}. \] Writing $\theta(P)$ before establishing identification can hide incompatible latent models. Confidence sets additionally need a sampling law, confidence level, asymptotic sequence and uniformity class. Point identification, coverage and informative precision are separate properties.

Mediation. Total effects of randomized assignment are distinct from cross-world natural indirect effects. Belief/effort-channel effects require a meaningful mediator intervention and additional measurement, exclusion or mediation assumptions. Randomization of the initial policy alone does not supply those assumptions. Report bounds or interventional alternatives when the stronger target is unsupported.

Equilibrium. A counterfactual $W(z)$, $p(z)$, or $\mu_t^z$ requires individual optimization, feasibility, government/resource budgets, market clearing and state-transition laws. State equilibrium existence and selection, or report ranges over compatible equilibria. Initial-state integration includes subsequent endogenous transitions; current-state integration must use the policy-dependent distribution. Reduced-form invariance after policy is not assumed.

Welfare accounts. Scores, utility, health, dollars and ecological quantities require explicit conversion before addition. Fees, taxes, subsidies, wages and extortion payments are transfers between parties; distributional weights can make them welfare relevant, but their face value is not automatically a resource loss. State ownership, geographic boundaries, foreign-income weights and fiscal finance. Do not add profits to household welfare already including dividends, or count land capitalization and the underlying benefit twice. A benefit-cost expression must distinguish gross benefits from net welfare and subtract every real cost exactly once.

Derivatives and risk. Log derivatives require positive arguments; local responses require admissible perturbations and specified equilibrium branches. Differentiation under expectations needs regularity/domination, and boundaries may allow only one-sided derivatives. Risk uses a specified law; ambiguity uses a declared nonempty law set on a common history space. Adaptive policies must be nonanticipative. A robust criterion is an editorial choice unless attributed explicitly.

Scope overlaps. Allocation, collective choice, contracts, household bargaining and coordinated policy overlap economics with optimization and game theory. They are retained because they appeared in the original catalogue; no claim of a strictly game-theory-free selection is made.

Statement licence: CC-BY-4.0 · Source file

Contributions 1

Econ-344-A1 · Exploration · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-344

External validity and scaling of randomized interventions: a research attempt

Target and outcome

Status: unresolved. This entry combines transport of a randomized intervention to a new population with changes in implementation and equilibrium at scale. The following attempt gives sharp bounds for a finite-stratum transport model with limited overlap and an aggregate sensitivity budget. It then constructs scale-up responses that agree at all observed experimental saturations but differ at full coverage. Neither result validates a nationwide policy empirically.

Vivalt's primary publisher abstract [vivalt] documents the relevance of heterogeneity across evaluations. It motivates checking transport restrictions, but does not establish the restrictions below. The bounds and counterexample are derived here as conditional mathematical results; no novelty over the general partial-identification literature is claimed.

Data and the precise restricted target

Let $X$ take finitely many values. The target population has known weights $w_x\geq0$ summing to one. In the study, treatment is randomized with positive probability for each arm in the observed strata $O$, so the study effect \[ a_x={\mathbb{E}}[Y(1)-Y(0)\mid X=x,S=1],\qquad x\in O, \] is identified. Outcomes lie in $[0,1]$. Target observations reveal covariate frequencies only; they do not reveal target potential-outcome means. Define the target effects $t_x={\mathbb{E}}[Y(1)-Y(0)\mid X=x,S=0]$ and target mean $T=\sum_xw_xt_x$.

For overlapping strata impose declared restrictions \[ |t_x-a_x|\leq\delta_x,\qquad \sum_{x\in O}w_x|t_x-a_x|\leq\Gamma, \quad \delta_x,\Gamma\geq0. \] For $x\notin O$ impose only $-1\leq t_x\leq1$. These are assumptions on target effects, not consequences of randomization. The aggregate budget permits departures to be distributed across strata but limits their total target-population weight. Zero-weight strata can be omitted.

Set \[ T_O=\sum_{x\in O}w_xa_x,\quad W_N=\sum_{x\notin O}w_x, \] \[ U=\sum_{x\in O}w_x\min(\delta_x,1-a_x),\qquad L=\sum_{x\in O}w_x\min(\delta_x,1+a_x). \]

Proposition (Sharp transport bounds): The identified set for $T$ is the entire interval \[ \left[T_O-\min(\Gamma,L)-W_N,\quad T_O+\min(\Gamma,U)+W_N\right]. \]
Proof: For the upper endpoint, each positive deviation $t_x-a_x$ is bounded by both $\delta_x$ and $1-a_x$. Its total weighted sum is bounded by $U$, and the absolute-deviation budget also bounds it by $\Gamma$. Negative deviations cannot improve the objective. The unsupported strata contribute at most $W_N$. This proves the upper inequality; reversing signs proves the lower one.

To attain the upper bound, allocate nonnegative weighted deviations across the overlapping strata until their sum reaches $\min(\Gamma,U)$, never exceeding the individual capacities $w_x\min(\delta_x,1-a_x)$. A sequential filling procedure succeeds because the capacities sum to $U$. Set all unsupported effects to one. For the lower bound, fill the corresponding downward capacities and set unsupported effects to minus one.

Every resulting effect vector is realizable: in target stratum $x$, choose Bernoulli potential outcomes with means $(1+t_x)/2$ and $(1-t_x)/2$, with any joint coupling. This changes no observed target outcome law because only target covariates are observed. Keep the entire study law unchanged. The restrictions define a convex compact set of effect vectors; convex interpolation between endpoint vectors retains all restrictions and realizes every intermediate value of $T$. Thus the interval is sharp for the specified data and model class. ∎

For example, take target weights $(0.3,0.4,0.3)$, with only the first two strata represented in the study. Let $(a_1,a_2)=(0.2,-0.1)$, $(\delta_1,\delta_2)=(0.1,0.2)$ and $\Gamma=0.05$. Then $T_O=0.02$, $L=U=0.11$ and the exact interval is $[-0.33,0.37]$. The unsupported 30 percent of the population dominates the remaining uncertainty. Reporting $0.02$ as an overall transported effect would silently assign zero effect to that group.

Why ordinary overlap language is insufficient

With continuous $X$, equality of effects almost everywhere under the study law supports the usual transport integral only when the target covariate law is absolutely continuous with respect to the study covariate law. Topological support containment does not suffice. For example, let the study covariate be uniform on $[0,1]$ and let the target covariate equal zero with probability one. The point zero belongs to the study's support but has study probability zero. Two response functions differing only at zero generate the same study law and arbitrary different target effects in $[-1,1]$. A continuity or smoothness restriction can change this conclusion, but must be stated.

The finite-stratum formula has the correct limiting diagnostics. If $W_N=0$ and $\Gamma=0$, it collapses to the transported study average. If $\Gamma$ is large, the individual $\delta_x$ and outcome bounds determine the endpoints. If there is no overlap, the interval is $[-1,1]$. Additional observed target outcomes or cross-stratum shape restrictions generally change the sharp set; the construction does not assert sharpness after adding such information.

A scale-up obstruction even with accurate local effects

A separate issue arises when outcomes depend on coverage $s$. Fix an observed experimental coverage $s_0\in(0,1)$ and define, for $d\in\{0,1\}$ and $s\in[0,1]$, \[ Y_c(d,s)=\tfrac14+\tfrac1{10}d+c\,s(s-s_0)^2, \qquad |c|\leq\tfrac1{10}. \] These outcomes remain in $[0,1]$. Treatment effects at every given coverage are exactly $1/10$, independently of $c$. At both $s=0$ and $s=s_0$ the spillover term vanishes. Its derivative also vanishes at $s=s_0$. Thus experimental arm means at $s_0$, untreated outcomes at zero coverage, and even the local derivative of aggregate outcomes at $s_0$ agree for every $c$.

Under independent treatment with probability $s$, the aggregate mean is \[ \overline Y_c(s)=\tfrac14+\tfrac1{10}s+c\,s(s-s_0)^2. \] The full-coverage change relative to no intervention equals $1/10+c(1-s_0)^2$, which varies with $c$. Hence those local observations do not identify the scale-up target. This is a potential-outcome nonidentification example with an explicit coverage map; it is not a claim to have built a calibrated economic equilibrium. Stronger equilibrium restrictions or coverage experiments near one may exclude some of these curves.

What remains open

The first result resolves the stated finite-dimensional bounding problem conditional on known study effects and sensitivity parameters. Finite-sample uncertainty can be accommodated by projecting a simultaneous confidence region for the study means through the same constraints; its informativeness depends on actual sample sizes. Selecting $\delta_x$ or $\Gamma$ from the same unvalidated target assumptions is not empirical verification.

The broader target requires evidence about target selection, implementation capacity, coverage-induced prices, supply and interference, as well as prospective performance on new programs. The counterexample explains why increasing the precision of an existing small trial alone may not close that gap. No assertion is made that external validity is impossible under all informative designs.

Completion Estimate

49%

This is a post hoc, heuristic estimate for the full catalogue target. The attempt derives sharp finite stratum transport bounds under individual and aggregate sensitivity restrictions and constructs indistinguishable local experiments with different full coverage outcomes. The original target still requires prospective validation and credible implementation, spillover, and equilibrium price restrictions for the national policy contrast.

References

  • [vivalt] E. Vivalt (2020), How Much Can We Generalize from Impact Evaluations?, Journal of the European Economic Association 18(6), 3045--3089. Publisher abstract checked: https://doi.org/10.1093/jeea/jvaa019.

Go to contribution →