we-solve (beta)

Econ-562 · economics · Economics

Scaling behavioral interventions

LLM status Open unverified

Statement

Original subject group: Behavioral and experimental economics

Audit classification: Published research agenda. DellaVigna–Linos March 2020 manuscript explicitly asks about scaling and leaves forecasting accuracy needing more trials. The causal saturation surface is editorial.

Definitions and precise research target

Model and research target. Index interventions by $j$ and implementation scale by $s$, defined as the fraction of an eligible population reached. Let $Y_{ij}(d,s)$ be participant $i$'s outcome under receipt $d\in\{0,1\}$, allowing implementation capacity and spillovers to depend on scale. Define the direct treatment effect $\tau_j(s)=\mathbb E[Y_{ij}(1,s)-Y_{ij}(0,s)]$. Let $X_j$ contain preregistered intervention features and $C_j(s)$ contain measured implementation features. The prediction target for a large-scale rollout is \[\widehat\tau_j(s_H)=g\left(\widehat\tau_j(s_L),X_j,C_j(s_L),C_j(s_H)\right),\qquad R=\mathbb E[(\widehat\tau_j(s_H)-\tau_j(s_H))^2].\] Estimate and validate $g$ on interventions held out from training, using complete trial registries rather than only published significant effects. Separate differences caused by publication selection, population composition, intervention versions, and administrative constraints from the causal effect of changing scale itself. Observing academic and government trials at different scales does not identify $\tau_j(s_H)-\tau_j(s_L)$ for a common intervention. Identification of that contrast requires randomized saturation, replicated implementation, or explicit transport assumptions. Assess both rank prediction and magnitude prediction, and include uncertainty from estimated trial effects in $R$.

Definitions and maintained assumptions (editorial unless a source definition is named). Let $s\in[0,1]$ denote a specified treatment saturation within a defined interference cluster; exposure mapping must justify $Y_{ij}(d,s)$ rather than dependence on every other individual’s assignment. $\tau_j(s)$ is the direct treatment effect at saturation $s$, not total rollout welfare or the outcome effect on untreated people. Randomized two-stage assignment first sets cluster saturation, then individual receipt, with support for $d=0,1$ at each studied $s$. Population composition, implementation version and implementer must be held fixed or included in the transport model. Define a target law $\mathcal J$ over new interventions and implementations for risk $R=\mathbb E_{j\sim\mathcal J}[(\widehat\tau_j(s_H)-\tau_j(s_H))^2]$; without $\mathcal J$, prediction risk is unspecified. Training and test interventions are independent under a declared sampling scheme. Trial-effect estimation noise must be separated from latent forecast error. Complete registries reduce publication selection but do not randomize implementation scale.

Variants and assumption changes

  1. Causal scale effect (editorial). Randomize saturation for the same intervention, implementer and population. Estimate both direct effects $\tau_j(s)$ and spillover contrasts among untreated units.
  2. Cross-institution forecast (source-related editorial). Train on registered small and large trials with specified feature coding and target intervention law. Test predictions on entirely held-out interventions, reporting transport assumptions rather than treating academic--government differences as scale effects.

References and source audit

  1. §1 p.1; end of §4 p.21 in March2020 manuscript. Supports the stated research area; exact displayed specification is editorial. https://eml.berkeley.edu/~sdellavi/wp/NudgeToScale2020-03-20.pdf
  1. Stefano DellaVigna; Elizabeth Linos. 2020. RCTs to Scale: Comprehensive Evidence from Two Nudge Units. Author manuscript dated March 2020, §1, printed p. 1; end of §4 and start of §5, printed p. 21 (PDF pp. 2 and 22).. https://eml.berkeley.edu/~sdellavi/wp/NudgeToScale2020-03-20.pdf.

Common conventions: Source mathematical conventions

These conventions apply to each entry unless explicitly replaced.

Models and identification. A primitive model $m$ specifies all probability laws, feasible choices, information, equilibrium and measurement rules used by its target. The admissible class $\mathcal M$ is fixed independently of outcomes. If $P_O(m)$ is its observable probability law and $\tau(m)$ the target, its identified set at observable law $P$ is \[ \mathcal I_\tau(P)=\{\tau(m):m\in\mathcal M,\ P_O(m)=P\}. \] Point identification means that this set is a singleton. An empty set signals incompatibility of data and assumptions. Coordinate bounds need not describe the joint set. Bounds are sharp only if every retained target is attainable or arbitrarily approachable by compatible models. Finite bounds require explicit restrictions on unobserved quantities; unrestricted confounding, hidden wealth, extrapolation or arbitrary equilibrium selection can yield uninformative sets.

Causal interventions. A potential outcome $Y_i(p)$ is the outcome under a complete policy regime $p$, including timing, financing, information and market spillovers. The target population and units are fixed before treatment. With interference, use the complete assignment vector or a justified exposure mapping; individual no-interference notation alone cannot identify an economy-wide intervention. A difference of mean potential outcomes does not require the cross-world joint distribution. Conditioning on post-treatment migration, survival, enrollment or employment changes the estimand and needs separate justification.

Statistical statements. An estimator, confidence set, forecast or test specifies a sampling law, model class and loss/coverage criterion. Uniform $(1-\alpha)$-coverage means $\inf_{m\in\mathcal M}\Pr_m\{\tau(m)\in C_n\}\geq1-\alpha$, or an explicitly asymptotic version. The whole parameter space trivially has coverage, so informativeness requires a stated width, risk or power objective. A forecasting improvement is relative to a named benchmark, information set and evaluation population.

Equilibrium and optimization. An equilibrium comprises feasible plans, optimality, beliefs where needed, budget constraints and all market/resource clearing. The primitives or a stated benchmark define each correspondence $\mathcal E(p;m)$. Nonemptiness is assumed or proved, never inferred just from compact policy sets. A selected-equilibrium target uses a declared selection rule; a robust target quantifies over all permitted equilibria. Use suprema or infima unless attainment is established. An $\varepsilon$-optimal policy is feasible and within $\varepsilon$ of the finite optimum value. Report an empty feasible set separately.

Welfare and accounting. Normative utility, interpersonal normalization, social weights, numeraire, discounting and the use of public receipts are primitives. Behavioral utility need not coincide with the welfare criterion. Household welfare includes ownership income and rents once; adding profits again to compensating variation generally double-counts them. A monetary equivalent is defined by an explicit transfer and price convention, with monotonicity and range conditions. Average effects, mechanism contributions, transfers and welfare losses are different targets. Infinite sums require convergence and dynamic feasible plans require terminal or no-Ponzi conditions.

Source versions. A working-paper date, revision date and journal publication year are distinguished. A DOI for a journal version is not automatically the DOI of an earlier manuscript. Locators refer to the stated version and printed pages where specified. A metadata-only check does not verify a claim about a full-text conclusion. Every entry's source subsection narrows the provenance claim accordingly.

Statement licence: CC-BY-4.0 · Source file

Contributions 1

Econ-562-A1 · Exploration · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-562

Scaling behavioral interventions: honest forecast evaluation with noisy trial effects

Target and scope

For a target law over interventions and implementations, let $L_j=\tau_j(s_L)$ and $H_j=\tau_j(s_H)$. The prediction target is \[ R(f)=E[(f(T_j,F_j)-H_j)^2],\qquad T_j=L_j+\varepsilon_j, \] where $F_j$ records prespecified intervention and implementation features. The low-scale estimate $T_j$, rather than the latent $L_j$, is available when the forecast is made. DellaVigna and Linos [scale] explicitly discuss magnitude and rank forecasting and the need for more interventions to assess it. The results below distinguish trial noise, intervention-level generalization and causal scale effects.
Theorem (Held-intervention risk correction): Suppose test interventions are iid from the declared target law and independent of training. For intervention $j$, a high-scale validation trial returns $V_j=H_j+\eta_j$. Conditional on the latent effects, observed forecast inputs and training, assume $E\eta_j=0$ and $E\eta_j^2=v_j$, where $v_j$ is known or replaced by a conditionally unbiased estimate with finite expectation. Then \[ \widehat R(f)=\frac1J\sum_{j=1}^J\{(f(T_j,F_j)-V_j)^2-v_j\} \] is unbiased for the latent forecast risk, conditional on training. For any two trained rules $f,g$, their uncorrected loss difference is already unbiased for $R(f)-R(g)$; no variance subtraction is needed.
Proof: Expand $(f-H-\eta)^2=(f-H)^2-2(f-H)\eta+\eta^2$ and take the specified conditional expectation. The cross term vanishes and the last term equals $v$. Averaging over target interventions proves the first claim. Subtracting the corresponding expansions for $f$ and $g$ cancels $\eta^2$ identically, leaving the latent loss difference minus $2(f-g)\eta$, whose conditional mean is zero. Training independence is essential: fitting the forecaster to the validation outcomes can correlate the forecast error with $\eta$. ∎
For a fixed pair of rules and homoskedastic validation variance $v$, conditional centering also yields \[ \operatorname{Var}\{(f-V)^2-(g-V)^2\} =\operatorname{Var}\{(f-H)^2-(g-H)^2\}+4vE[(f-g)^2]. \] This follows because the two terms in the preceding proof have zero covariance. The effective sample size for assessing new interventions is $J$, not the number of individuals pooled across their trials. If effects, predictions and trial estimates are bounded in $[-B,B]$, the loss difference lies in $[-4B^2,4B^2]$; Hoeffding gives an honest radius $8B^2\sqrt{\log(2/\alpha)/(2J)}$. A preregistered finite pool of comparisons can be handled by a union bound. Selecting the most favorable comparison afterward without that adjustment changes the guarantee.
Proposition (Magnitude prediction under a Gaussian intervention law): Suppose $(L,H)$ is jointly Gaussian with means $(\mu_L,\mu_H)$, variances $(v_L,v_H)$, covariance $c$, and independent low-trial noise $\varepsilon\sim N(0,v_\varepsilon)$. Assume $v_L+v_\varepsilon>0$. The unique mean-square optimal forecast, up to almost-sure equality, is \[ f^*(T)=\mu_H+\frac{c}{v_L+v_\varepsilon}(T-\mu_L), \] and its risk is $v_H-c^2/(v_L+v_\varepsilon)$.
Proof: The pair $(T,H)$ is jointly Gaussian, with covariance $c$ and variance of $T$ equal to $v_L+v_\varepsilon$. Subtract the displayed linear function from $H$. The residual is Gaussian and uncorrelated, hence independent, of $T$, with variance $v_H-c^2/(v_L+v_\varepsilon)$. It has mean zero. For any other measurable forecast $f$ of finite risk, orthogonality gives $E(f-H)^2=E(f-f^*)^2+E(f^*-H)^2$. This proves both optimality and the risk formula. ∎
Even perfect knowledge of the cross-intervention regression leaves variation in large-scale effects not predicted by small-scale effects. Increasing the high-scale validation sample improves evaluation of $R$ but does not remove that irreducible prediction component.
Proposition (An exact rank-prediction ceiling): Under the preceding Gaussian law, assume $v_H>0$ and $c\ne0$. For two independent interventions, the probability that the optimal forecast correctly orders their large-scale effects is \[ \frac12+\frac1\pi\arcsin\rho,\qquad \rho=\frac{|c|}{\sqrt{v_H(v_L+v_\varepsilon)}}. \]
Proof: The differences in forecasts and true effects are centered jointly Gaussian. Scaling them to unit variance gives correlation $\rho\ge0$. Represent this pair as $(G_1,\rho G_1+\sqrt{1-\rho^2}G_2)$ for independent standard Gaussians. The direction of $(G_1,G_2)$ is uniform on the circle, because its density depends only on radius. The two linear forms have opposite signs on wedges of total angle $2\arccos\rho$. Hence their signs agree with probability $1-\arccos\rho/\pi$, equal to the stated expression. Perfect correlation follows by continuity. With $c=0$, forecasts tie and independent fair tie-breaking gives probability $1/2$. ∎
The Gaussian assumption is a declared prediction model, not evidence that actual intervention effects are Gaussian. It makes the distinction between ranking and magnitude accuracy quantitative: neither is established simply by a significant average treatment effect.

Causal scaling requires a different experiment

Suppose each intervention is observed at only one saturation $s_j$. Even exact knowledge of its effect there does not identify its scale response: the families $\tau_j(s)=a_j+b_j(s-s_j)$ all agree at the observed saturation for arbitrary slopes compatible with outcome bounds. Complete publication registries do not alter this observational equivalence.
Proposition (Two-stage saturation identification): Consider independent interference clusters using the same intervention version and target composition. Randomize each cluster to saturation regime $S\in\{s_L,s_H\}$ with positive probabilities $q_s$, then randomize individual receipt $D$ with known positive conditional probabilities $p_d(s)$. Assume the exposure mapping makes outcomes $Y_i(d,s)$ well defined, with no interference across clusters. The cluster average of \[ \frac{1\{S=s\}}{q_s} \left\{\frac{D_iY_i}{p_1(s)}-\frac{(1-D_i)Y_i}{p_0(s)}\right\} \] is unbiased for the direct effect at regime $s$, averaged over the fixed baseline units. Subtracting the two regime scores identifies the causal change in the direct effect.
Proof: Condition on the cluster's complete potential outcomes. Two-stage randomization and consistency give expectation $Y_i(1,s)-Y_i(0,s)$ for each score. Average over units and then the declared population of clusters. Dependence within a cluster affects uncertainty but not this expectation. If literal fixed treated fractions make own receipt change peer exposure, the exposure label must include that distinction; the stipulated mapping cannot silently equate the two peer configurations. ∎

Remaining gap

The note provides honest forecast evaluation and a quantitative restricted prediction model, not an empirically validated universal scaling function. Changing implementers, treatment versions or target intervention laws requires new transport information. Causal saturation experiments answer a different question from forecasts across institutions, and neither intervention registries nor noise correction remove that distinction.

Completion Estimate

65%

This is a post hoc, heuristic estimate for the full catalogue target. The attempt derives held-intervention risk correction, Gaussian magnitude and ranking limits, and two-stage identification of causal saturation differences. Full behavioral scaling still requires registered matched implementations, empirically trained and independently validated prediction rules, and transport restrictions distinguishing population, version, implementer, publication, and administrative changes.

References

Go to contribution →