GPT 6 Astra Ultra attempt on Econ-642

Stability of social-preference measures Prediction stability, calibration, and repeated measurement

Target and scope

The catalogue asks whether fixed, source-trained social-preference survey scores improve prediction of incentivized behavior in new populations and contexts. The result must concern specified games, translations, stakes, and partner protocols. It is stronger than finding one significant survey coefficient. This attempt derives a risk decomposition, a finite-sample criterion for meaningful improvement, and a repeated-measurement diagnostic. It establishes no universal stability of trust, altruism, or reciprocity. The original problem remains unresolved without prospective population data.

Normalize one game outcome to $Y\in[0,1]$ and clip predictions to $[0,1]$. Fix independently trained functions $f(S,X)$ and $g(X)$, with equal access to source training and tuning resources. All expectations below refer to one target population and one fixed game protocol. Under squared loss define \[ V={\mathbb{E}}[(Y-g)^2-(Y-f)^2],\qquad m={\mathbb{E}}[Y\mid S,X],\quad m_0={\mathbb{E}}[Y\mid X]. \] The source training sample is conditioned upon throughout validation. Retraining on target outcomes changes the estimand and requires a separate target validation sample.

What predictive success would establish

Equation (1) gives two distinct failure modes. The survey can have little conditional behavioral signal, or a source-trained rule can miss a strong target signal. Conversely, a weak covariate-only competitor can inflate $V$ through its own prediction error. Equal tuning budgets help make the comparison interpretable but do not make $g$ the target Bayes rule.

Consider $X$ constant and $S\sim\operatorname{Bernoulli}(1/2)$. In the source, $Y=S$, so $f(S)=S$ is perfectly calibrated and $g=1/2$ gives $V=1/4$. In a target population let $Y=1-S$, preserving the entire marginal law of $S$ and of $Y$. The same functions give \[ {\mathbb{E}}(Y-f)^2=1,\qquad {\mathbb{E}}(Y-g)^2=1/4,\qquad V=-3/4. \] Survey-distribution similarity therefore cannot establish predictive transport. Even equality of game-outcome marginals is insufficient; the joint law matters. This is a mathematical counterexample to an automatic transport assertion, not evidence that any named population reverses its social preferences.

Calibration is also a separate condition. The constant score $f={\mathbb{E}} Y$ is calibrated but supplies no gain over the corresponding constant baseline. For any score $F=f(S,X)$, write $r(F)={\mathbb{E}}[Y\mid F]$. Conditional-mean orthogonality gives \[ {\mathbb{E}}(Y-F)^2={\mathbb{E}}(Y-r(F))^2+{\mathbb{E}}(r(F)-F)^2. \tag{2} \] The second term measures miscalibration; the first can remain large even when that term is zero. Checking both $V$ and calibration avoids equating a correct average prediction with useful discrimination.

A simultaneous, economically meaningful validation criterion

Suppose there are $M$ prespecified population--game--score comparisons. Comparison $j$ has $n_j$ independent validation participants and paired loss differences \[ D_{ij}=(Y_{ij}-g_j(X_{ij}))^2-(Y_{ij}-f_j(S_{ij},X_{ij}))^2\in[-1,1]. \] Let $\widehat V_j$ be their mean and set \[ r_j=\sqrt{\frac{2\log(2M/\alpha)}{n_j}}. \] Hoeffding's inequality and a union bound imply, conditional on all training samples, \[ {\mathbb{P}}\{\,|\widehat V_j-V_j|\le r_j\text{ for all }j\,\}\ge1-\alpha. \tag{3} \] Independence between different comparisons is unnecessary: the same person may complete several games, provided observations within each comparison are independent across sampled participants. Household or session correlation instead requires independent cluster-level bounded statistics and the appropriate cluster sample size.

For a chosen practical threshold $v_0>0$, the event $\widehat V_j-r_j>v_0$ for every $j$ certifies the finite collection of population improvements at simultaneous confidence $1-\alpha$. It says nothing about unmeasured populations. In particular, rejection of $V_j=0$ is weaker than this statement. For two candidate scores on the same game, use the paired difference of their losses directly; unrelated confidence intervals waste the covariance information.

For prespecified score bins $B_b$, a necessary implication of calibration is \[ a_b={\mathbb{E}}[(Y-F)\mathbf1\{F\in B_b\}]=0. \] Each sample summand lies in $[-1,1]$, so the same bound applies jointly to these moments after enlarging the number of comparisons. The conditional bin bias is $a_b/{\mathbb{P}}(F\in B_b)$ when that probability is positive. A tiny unconditional moment does not certify a rare bin: one needs a lower bound on bin mass. Passing finitely many bin checks does not prove pointwise calibration, because positive and negative errors can cancel inside bins. These limitations are substantive, even with unlimited computational effort.

Separating repeatability from changes in behavior

A restrictive measurement model makes the required assumptions visible. At time $t$, suppose a survey measure and a game measure satisfy \[ S_t=T_t+e_t,\qquad G_t=T_t+u_t. \] Assume finite second moments, mean-zero errors independent of the entire latent process $(T_t)$, and errors mutually independent across methods and times. The equal unit loadings are maintained assumptions, not consequences of observing positive correlations. Then \[ {\operatorname{Cov}}(S_t,G_t)={\operatorname{Var}}(T_t),\quad {\operatorname{Cov}}(S_t,G_s)={\operatorname{Cov}}(T_t,T_s)\quad(t\ne s), \tag{4} \] while \[ {\operatorname{Var}}(e_t)={\operatorname{Var}}(S_t)-{\operatorname{Cov}}(S_t,G_t),\qquad {\operatorname{Var}}(u_t)={\operatorname{Var}}(G_t)-{\operatorname{Cov}}(S_t,G_t). \] These follow by expanding covariances and deleting the independent error terms. If both latent variances are positive, the latent temporal correlation is identified by \[ \frac{{\operatorname{Cov}}(S_t,G_s)}{ \sqrt{{\operatorname{Cov}}(S_t,G_t){\operatorname{Cov}}(S_s,G_s)}}. \tag{5} \] Thus increased survey noise can reduce raw survey--game correlations while (5) remains unchanged. For example, take a time-invariant mean-zero latent $T$ of variance one, game-error variance one, and survey-error variance one initially and nine later. The contemporaneous correlations decrease from $1/2$ to $1/\sqrt{20}$ although the latent variable never changes. Bounded independent two-point variables realize these variance calculations; this auxiliary model is not imposing those units on the earlier normalized game outcome.

Correlated method errors, changing loadings, or the effect of the survey itself on subsequent play invalidate (4). A single incentivized game also mixes preferences with beliefs and understanding. Consequently, even a successful covariance decomposition does not identify a uniquely named psychological trait. Randomizing stakes or anonymity while fixing the survey prediction rule supplies direct evidence about protocol dependence; it does not automatically recover an invariant latent preference.

Source relationship and remaining gap

Kosfeld and Sharafi's 2022 working paper examines the external validity of social-preference survey measures using an independent experimental population. The abstract, introduction, and reported comparison of survey items motivate treating predictive stability as an empirical question, with item-specific performance. The calculations here are standard conditional-expectation and concentration arguments, not claims to extend that paper's empirical findings. They concern the 2022 working-paper version; they do not pretend to reproduce the subsequent published study.

The missing result is prospective evidence meeting (3), meaningful calibration tolerances, and a defensible measurement/protocol model across the intended countries and behavioral dimensions. No algebraic identity can determine those population quantities from an unrelated source sample.

Completion Estimate

35%

This is a post hoc, heuristic estimate for the full catalogue target. The attempt separates predictive signal, transport error and calibration, gives simultaneous practical-improvement tests and derives a restrictive repeated-measurement diagnostic. A validated domain of social-preference prediction still requires prospective target-population data, translation and stake comparability, meaningful calibration tolerances and defensible measurement models across contexts.

References

• [ks] Michael Kosfeld and Zahra Sharafi (2022), The Preference Survey Module: New Evidence on Social Preferences from Tehran, IZA Discussion Paper 15006. https://docs.iza.org/dp15006.pdf.