we-solve (beta)

Econ-643 · economics · Economics

Lab–online–field population generalizability

LLM status Open unverified

Statement

Original identifier: B10 (atlas item 70). Classification: Population parameter generalizability research program. Original tractability label: Easy (editorial, not a complexity result).

Definitions and assumptions

Let populations $p\in\mathcal P$ represent laboratory, online, and field recruitment frames, all facing a common experimental protocol $G$. For person $i$, let $\theta_i\in\Theta$ be a latent behavioral parameter and $X_i$ observed pretreatment characteristics. The measurement model is $P(A_i\mid G,\theta_i,X_i,p)$, with common normalization and a specified noise distribution. Let $R_i=1$ denote participation and define the target population distribution independently of recruitment. Parameter comparability requires measurement invariance: identical latent types facing the same protocol generate identical conditional action distributions across frames, unless population-specific measurement differences are modeled.

Formal research question

Identify or sharply bound $F_{\theta\mid p}$ and the extent to which between-population differences are attributable to observed composition. Under conditional latent-type exchangeability and overlap, evaluate \[ F_{\theta\mid p_t}(z)=\int F_{\theta\mid X=x,R=1,p_s}(z)\,dF_{X\mid p_t}(x),\quad z\in\Theta, \] interpreted componentwise for vector parameters. Determine sensitivity to violations of sampling exchangeability and measurement invariance. Test whether parameters calibrated on one frame predict choices on held-out frames within a declared discrepancy metric and tolerance, accounting for first-stage estimation uncertainty.

Interpretation and scope

This target differs from B1: it concerns transport of a latent parameter distribution, not merely transport of one treatment effect. Latent parameters may be unidentified even when an average treatment effect is experimentally identified. Subject-pool regressions should therefore account for estimation noise, rather than regress noisy individual estimates as if observed without error. Education or digital literacy can explain statistical variation without being its causal source; causal statements about these traits require a separate intervention or identification argument. Representative panels reduce some sampling discrepancies but do not eliminate participation selection or protocol differences. Common instructions, incentives, device access, and comprehension checks help support measurement invariance, although checks conducted after treatment must not create new selection bias. A benchmark infrastructure should preregister target populations and protocols, document exclusions, and evaluate both population-average and distributional predictions. Good fit in observed populations cannot establish universal generalizability to every future subject pool.

Source audit and status

[S1] and [S2] verify the original generalizability citations. [S3] again concerns laboratory replicability, so its role is background quality control rather than direct evidence of population parameter transport. The source overlap with B1 is retained transparently while the formal targets are distinguished. This editorial measurement and transportability problem does not inherit a universally easy classification from the atlas; identification difficulty depends on sampling and measurement restrictions.

References

    },leftmargin=*]

  1. Glenn W. Harrison and John A. List (2004). Field Experiments. Journal of Economic Literature 42(4), 1009--1055. https://www.aeaweb.org/articles?id=10.1257/0022051043004577

    Source role: Taxonomy of field settings; methodological foundation. Locator: Abstract; six dimensions of field context.

  2. Steven D. Levitt and John A. List (2007). What Do Laboratory Experiments Measuring Social Preferences Reveal About the Real World?. Journal of Economic Perspectives 21(2), 153--174. https://www.aeaweb.org/articles?id=10.1257/jep.21.2.153

    Source role: Methodological discussion of generalizability. Locator: Abstract; discussion of scrutiny, context, selection, and stakes.

  3. Colin F. Camerer, Anna Dreber, Eskil Forsell, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Johan Almenberg, Adam Altmejd, Taizan Chan, Emma Heikensten, Felix Holzmeister, Taisuke Imai, Siri Isaksson, Gideon Nave, Thomas Pfeiffer, Michael Razen, and Hang Wu (2016). Evaluating Replicability of Laboratory Experiments in Economics. Science 351(6280), 1433--1436. https://doi.org/10.1126/science.aaf0918

    Source role: Evidence on laboratory replicability, not field transportability. Locator: Abstract; author repository metadata and supplemental materials.

Common conventions: Source mathematical conventions

Unless an entry states otherwise, agent and firm sets are finite. State and action spaces are measurable, strategies and outcome maps are measurable, probability laws are specified, and expectations exist for the targets under discussion. Infinite-horizon discount factors lie in $(0,1)$ and discounted objectives are finite. Norms, tolerances and complexity input sizes must be specified for computational comparisons. Constants or primitives that appear locally are inputs, not empirical facts asserted by this catalogue.

Identification. A statistical model $m\in\mathcal M$ implies an observable law $P_m$ and target $\theta(m)$. At law $P$, its identified set is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M,\ P_m=P\}. \] Point identification holds when this set is a singleton, or equivalently when $P_{m_1}=P_{m_2}$ implies $\theta(m_1)=\theta(m_2)$ for all compatible models. Sharp bounds mean bounds attained, or approached when the boundary is not attained, by compatible models. Writing this set is a definition, not a solution: the substantive task is to establish informative restrictions and derive or estimate the set. An unrestricted-confounding model may give only trivial bounds.

Inference. Identification, estimability and valid uncertainty quantification are separate questions. Fix $\alpha\in(0,1)$. A confidence procedure $C_n$ over a declared law class $\mathcal P$ has asymptotic uniform coverage at level $1-\alpha$ when \[ \liminf_{n\to\infty}\inf_{P\in\mathcal P}\Pr_P\{\theta(P)\in C_n\}\geq1-\alpha. \] For set-identified targets the coverage event must be defined separately, for example coverage of the full identified set. If an entry uses identification-indexed models instead of uniquely indexed laws, the same coverage convention is applied over those models. Pointwise limits do not establish uniform validity, and asymptotic validity does not guarantee finite-sample precision. Sample sizes, dependence structures and weak-identification sequences are part of each application.

Counterfactuals. $Y_i(a)$ is a potential outcome under a specified intervention, including its timing, implementation and versions. It is not $Y_i$ conditional on voluntarily choosing $a$. A full intervention vector permits interference; shorthand scalar treatment notation does not silently impose no interference. Assignment, selection, attrition, anticipation and measurement must be specified. A claim about an instrument's local effect is not automatically a population policy effect.

Economic equilibrium. A counterfactual model includes best responses, constraints, feasibility, market clearing, state transitions and expectations consistent with the stated information. When several equilibria exist, either a declared selector is an input or the target is set-valued. Comparisons across policy regimes must use the stated selection convention. Reduced-form relationships are not presumed invariant after an equilibrium-changing intervention.

Optimization and welfare. Optimization is over the stated feasible set. If attainment is not established, $\sup$ or $\inf$ and $\varepsilon$-optimal choices replace an asserted maximizer or minimizer. For example, with value $V=\sup_{a\in\mathcal A}W(a)<\infty$, an $\varepsilon$-optimal set is $\{a\in\mathcal A:W(a)\geq V-\varepsilon\}$ for $\varepsilon>0$. Benefits, costs, risk and health or environmental outcomes can be aggregated only using declared units and conversion weights. Interpersonal utility weights, the treatment of fiscal transfers and foreign profits, discounting, and the behavioral welfare criterion are explicit inputs. A comparative-static derivative additionally requires existence and appropriate differentiability of the selected counterfactual.

Sources. Local labels [S1], [S2], and so forth restart in every question. Locators identify the relevant abstract, model, section or result at the level actually checked; a bibliographic or abstract-level verification is not represented as inspection of an entire proof. Working papers are distinguished from later publications and changing versions. Source summaries are paraphrased. All URL checks and editorial formulations refer to the audit date stated above.

Statement licence: CC-BY-4.0 · Source file

Contributions 1

Econ-643-A1 · Exploration · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-643

Lab--online--field population generalizability Latent population transport with an explicit measurement operator

The object that must generalize

The catalogue concerns distributions of latent preference parameters across laboratory, online, and field sampling frames. A treatment effect can be identified in an experiment while the distribution of individual preference parameters remains unidentified. This attempt solves a finite-type measurement-and-transport model, supplies an exact failure example when types are unrestricted, and bounds sensitivity to measurement drift. These are conditional results; they do not establish empirical lab--online--field generalizability.

Let $X$ have finitely many prespecified cells $x$. Let $R=1$ denote participation in the source frame, and suppose the protocol produces a finite response pattern $A\in\{1,\ldots,m\}$. A latent type takes values in a fixed ordered grid $\theta_1<\cdots<\theta_k$ under a common normalization. Define the known measurement matrix and participant type proportions by \[ K_x(a,j)={\mathbb{P}}(A=a\mid\theta_j,X=x,R=1),\qquad \pi_{xj}={\mathbb{P}}(\theta=\theta_j\mid X=x,R=1). \] Every column of $K_x$ is a probability vector. The observed conditional law is $q_x=K_x\pi_x$. Knowing $K_x$ is a substantive structural assumption; randomizing experimental tasks alone does not usually supply it.

Identification, transport, and sharp bounds

Assume latent conditional exchangeability between source participants and the target population: \[ {\mathbb{P}}_t(\theta=\theta_j\mid X=x)=\pi_{xj}, \tag{1} \] and overlap for every target cell. Let $w_x={\mathbb{P}}_t(X=x)$, observed from a representative target sample. The target latent cumulative distribution is \[ F_t(z)=\sum_xw_x c_z^\mathsf{T}\pi_x, \qquad c_z(j)=\mathbf1\{\theta_j\le z\}. \tag{2} \]
Proposition: If each $K_x$ has full column rank, (1)--(2) identify the target distribution uniquely. Without full column rank, its pointwise sharp bounds are the minimum and maximum of the right side of (2), subject to \[ K_x\pi_x=q_x,\qquad \pi_x\ge0,\qquad \mathbf1^\mathsf{T}\pi_x=1\quad\hbox{for every }x. \tag{3} \]
Proof: Full column rank makes the linear map injective, hence identifies each feasible probability vector. Generally the feasible set in (3) is a nonempty compact polytope when the model fits. A linear functional attains both extrema. For every feasible collection, construct a target by first drawing $X$ with probabilities $w_x$, then drawing a type with probabilities $\pi_x$, and generate source responses through $K_x$. The resulting model reproduces all observed source response probabilities and target covariate frequencies and obeys (1). Thus each extremum is attainable. ∎

Pointwise bounds for different $z$ must not be presented as an arbitrary rectangle of jointly feasible CDF values. A single collection of vectors $\pi_x$ must generate the whole CDF. Additional target response data, known measurement matrices, or restrictions shared across cells add linear constraints and can narrow the same polytope.

A concrete measurement design illustrates the rank requirement. Suppose $\theta\in\{0,1/2,1\}$ and two conditionally independent binary responses have success probability $\theta$. The count of successes has matrix \[ K=\begin{pmatrix}1&1/4&0\\0&1/2&0\\0&1/4&1\end{pmatrix}. \] Writing count probabilities as $(q_0,q_1,q_2)$ gives \[ \pi_{1/2}=2q_1,\quad \pi_0=q_0-q_1/2,\quad \pi_1=q_2-q_1/2. \tag{4} \] Consequently the observable restrictions include $q_0,q_2\ge q_1/2$. For $q=(3/8,1/4,3/8)$ the recovered weights are $(1/4,1/2,1/4)$, which sum to one and reproduce all three count probabilities. With just one binary response, the observation identifies only $\pi_{1/2}/2+\pi_1$; mass one at $1/2$ and equal masses at zero and one are observationally equivalent.

Why repeated choices alone do not recover an arbitrary distribution

Equation (4) depends on the finite grid. If $\theta\in[0,1]$ is unrestricted, two conditionally independent binary responses identify only ${\mathbb{E}}\theta$ and ${\mathbb{E}}\theta^2$. Compare \[ H_1=\operatorname{Uniform}[0,1],\qquad H_2=\tfrac12\delta_{1/2-1/\sqrt{12}} +\tfrac12\delta_{1/2+1/\sqrt{12}}. \] Both have first moment $1/2$ and second moment $1/3$. Under either law the success count probabilities are $(1/3,1/3,1/3)$, yet the latent CDFs differ. For example, at $z=1/4$, $H_1(z)=1/4$ whereas $H_2(z)=1/2$. Even infinitely many participants completing two tasks cannot distinguish the two models. With $T$ repetitions the likelihood is a polynomial of degree $T$ in $\theta$, so it depends only on the first $T$ moments. More participants estimate these moments more precisely; they do not create new moment orders. Conditional independence, absence of learning, and a stable response probability across repetitions are themselves testable only partly and must be defended for the actual elicitation protocol.

A quantitative sensitivity result

Return to a finite grid. Suppose $K_x$ has smallest singular value $\sigma_x>0$ and the target measurement law is $K'_x=K_x+E_x$, with operator norm $\|E_x\|_2\le\varepsilon_x$. Suppose target responses satisfy $q'_x=K'_x\pi'_x$, while source responses satisfy $q_x=K_x\pi_x$. Then \[ \|\pi'_x-\pi_x\|_2 \le\frac{\|q'_x-q_x\|_2+\varepsilon_x}{\sigma_x}. \tag{5} \] Indeed $K_x(\pi'_x-\pi_x)=q'_x-q_x-E_x\pi'_x$; apply the singular-value lower bound and $\|\pi'_x\|_2\le1$. Thus small changes in observed response frequencies imply small changes in latent weights only when the measurement operator is sufficiently well-conditioned and its change is controlled. By Cauchy--Schwarz, \[ |F'_t(z)-\sum_xw_xc_z^\mathsf{T}\pi_x| \le\sum_xw_x\|c_z\|_2 \frac{\|q'_x-q_x\|_2+\varepsilon_x}{\sigma_x}, \tag{6} \] clipped at one. This is a valid conservative bound, not a sharp bound. Full rank without a useful lower bound on $\sigma_x$ can be practically uninformative.

Selection sensitivity is different from measurement drift. If target conditional type distributions are within total variation $\delta_x$ of source participant distributions, every CDF changes by at most $\sum_xw_x\delta_x$, directly from the definition of total variation. This assumption limits unmeasured selection; observing covariate balance does not prove it. Selection and measurement restrictions must both hold when both adjustments are invoked.

Inference and an operational validation check

For source cell samples $n_x$, simultaneous Hoeffding intervals for all response probabilities have half-width $\sqrt{\log(2L/\alpha)/(2n_x)}$, where $L$ counts the prespecified cell--response probabilities. Intersect those intervals with probability simplexes. Replace each equality in (3) by the corresponding interval constraint and optimize (2) over the expanded feasible set. On the joint coverage event, its extrema contain the true target CDF under (1). Target cell frequencies have their own simultaneous multinomial-coordinate intervals and simplex constraint. Optimizing over both $w$ and $\pi$ remains a finite compact optimization, though generally bilinear rather than a linear program. Ignoring uncertainty in either stage undercovers.

A held-out protocol with a specified measurement matrix $K^{\rm hold}_x$ yields predictions $K^{\rm hold}_x\pi_x$. Comparing these predictions with independent target choices tests joint implications of the latent model and transport assumptions. It cannot establish latent invariance against all observationally equivalent models. Dropping participants based on post-treatment comprehension tests additionally changes the sample and needs its own selection analysis.

Relation to the sources and unresolved claim

Harrison and List's publisher abstract distinguishes experimental contexts through participant pools, information, goods, rules, stakes, and setting. Levitt and List's publisher abstract explains why social choices can depend on scrutiny, context, and selection. Those considerations motivate the measurement and selection restrictions here; neither source is being cited as proving the finite-mixture proposition. Its linear algebra and moment counterexample are elementary conditional calculations.

The unresolved original task is to specify and validate realistic measurement operators, latent normalizations, and selection sensitivity ranges across actual lab, online, and field populations. The conditional reconstruction above provides a reviewable test of proposed assumptions, not empirical proof that the transported preference distribution is correct.

Completion Estimate

50%

This is a post hoc, heuristic estimate for the full catalogue target. The attempt solves finite-grid latent-type transport with sharp conditional bounds, exhibits unrestricted-distribution nonidentification and quantifies measurement drift and selection sensitivity. Realistic cross-frame generalizability still requires validated measurement operators and latent normalizations, credible selection restrictions and held-out prediction tests with both estimation stages accounted for.

References

Go to contribution →