we-solve (beta)

Econ-390 · economics · Economics

Incentivizing truthful and informative open-ended survey responses

LLM status Partially solved unverified

Statement

ECON-022 | R | Source-backed agenda; exact formalization is editorial

Definitions and formal target

Respondent $i$ has latent belief or reasoning state $B_i$, chooses effort $e_i\geq0$, and submits text $T_i$. A payment rule $w(T_i,V_i)$ may use a declared auxiliary validation signal $V_i$, but no perfect truth label is assumed. Effort and strategic presentation affect the distribution of text. Let $g(T_i)$ be a predeclared coding rule and $\ell(g(T_i),B_i)$ a stated measurement-loss function.

For payment budget $b$, characterize empirically feasible mechanisms on the frontier \[ \inf_{w\in\mathcal W}\mathbb E[\ell(g(T_i(w)),B_i)] \quad\text{s.t.}\quad\mathbb E[w(T_i(w),V_i)]\leq b, \] where responses and effort are best responses to the payment rule. Determine whether incentives reduce measurement loss relative to an unincentivized survey without merely rewarding verbosity or plausible rationalizations.

Clarifications and required assumptions

Specify respondent utility, for example expected payment minus effort cost plus a stated presentation motive. Effort determines information before reporting; it does not reveal the latent state without an observation model. Define $\mathcal W$, the auxiliary signal's distribution, randomization, participation, and selection among best responses. Require integrable payments and loss. The latent loss is a scientific target, not an automatically observable evaluation metric: supply identification, validation, or bounds.

Variants and changes of scope

  • Proper scoring against an observable outcome can elicit a probability under its assumptions; it does not automatically elicit an unrestricted narrative or effort.
  • Independent validation, peer information, prompts, and audits require different dependence assumptions. State whether both arms receive an equal participation fee.
  • Fix the text coder before evaluation or train it on separate data. Length, agreement, and fluency need not measure truthfulness.

Audit conclusion and source relationship

[S1]'s checked April 2025 working-paper version, Section 5, discusses response incentives and truthfulness. The journal identity is verified. The latent-loss optimization is an editorial design; no universally truthful text-payment mechanism is asserted.

References and exact source roles

[S1] Ingar Haaland, Christopher Roth, Stefanie Stantcheva, and Johannes Wohlfart (2025). Understanding Economic Behavior Using Open-ended Survey Data. Journal of Economic Literature 63(4), 1244–1280. DOI: 10.1257/jel.20251780. Page locators below refer to the April 2025 ECONtribute Discussion Paper 362 version.

Verified locator / source role: Section 5, Conclusion and avenues for future research, subsection Incentives, printed p. 40 (PDF page 42).

https://www.econtribute.de/RePEc/ajk/ajkdps/ECONtribute_362_2025.pdf

Common conventions: Additional-source status and mathematical conventions

Of the 100 supplied ECON entries, 65 distinct problems appear here; 32 close matches refine earlier entries, and three duplicate questions map to existing entries. Appendix app:audited-crosswalk lists every destination. The attachment's P/R/S/U distinctions and source audits are retained. Its precise mathematical formalizations are editorial unless the individual audit says otherwise. Candidate subsidy targets ECON-004--006 retain their U flags; the resolved approval-core question ECON-007 updates OP-0202 above. The following supplied conventions govern both this part and the attributed ECON refinements elsewhere in the catalogue, subject to local restrictions.

Parameterized questions. A problem family lists inputs that must be fixed before an instance is evaluated: population, horizon, policies, information, data law, model class and welfare weights. These are required choices, not quantities the citations are assumed to specify. Undefined applications should not be treated as identified estimands.

Indices and integrability. Sets of agents, firms and regions are finite unless a population measure is explicitly used. $i,j$ index units, $r$ regions, $t$ dates and $h$ horizons. $\mathbb E$ and $\Pr$ refer to the declared population and law; $\mathbf1\{A\}$ is an indicator. Every displayed expectation and discounted sum needs existence in the stated domain. Infinite horizons use a discount in $(0,1)$ and a convergence condition; finite horizons may use discount one.

Optimization and existence. A displayed $\max$, $\min$, or $\arg\max$ is conditional on attainment. For finite feasible menus this is immediate; general spaces need continuity and compactness/coercivity or another existence argument. Without attainment, the target is the supremum/infimum and arbitrarily near-optimal feasible choices. $\inf\varnothing=+\infty$. Empty feasibility and an unidentified target are different issues.

Potential outcomes. $Y_i(z)$ is the outcome under a specified intervention, not the observed conditional mean among people choosing $z$. In binary comparisons $\Delta Y_i=Y_i(1)-Y_i(0)$. Specify assignment, treatment versions, compliance, information and observation horizon. Interference is allowed under a full policy vector; compressed exposure notation needs a justified mapping. No interference, consistency and overlap are substantive assumptions, not automatic consequences of notation.

Populations and observation. Fix baseline eligibility and population weights. Conditioning on post-policy employment, survival, relocation or program uptake can change the population being compared. Death is not ordinary loss to follow-up; outcomes truncated by death need a composite convention, joint survival target, or explicitly defined survivor stratum. Fertility-changing interventions need a population functional rather than an unexplained fixed descendant cohort. Measurement and missingness models must be supplied.

Identification. A structural model $m\in\mathcal M$ implies observable law $P_m$ and target $\theta(m)$. Identification means \[ P_{m_1}=P_{m_2}\ \Longrightarrow\ \theta(m_1)=\theta(m_2) \quad\text{for all }m_1,m_2\in\mathcal M . \] Without identification, the target is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M,\ P_m=P\}. \] Writing $\theta(P)$ before establishing identification can hide incompatible latent models. Confidence sets additionally need a sampling law, confidence level, asymptotic sequence and uniformity class. Point identification, coverage and informative precision are separate properties.

Mediation. Total effects of randomized assignment are distinct from cross-world natural indirect effects. Belief/effort-channel effects require a meaningful mediator intervention and additional measurement, exclusion or mediation assumptions. Randomization of the initial policy alone does not supply those assumptions. Report bounds or interventional alternatives when the stronger target is unsupported.

Equilibrium. A counterfactual $W(z)$, $p(z)$, or $\mu_t^z$ requires individual optimization, feasibility, government/resource budgets, market clearing and state-transition laws. State equilibrium existence and selection, or report ranges over compatible equilibria. Initial-state integration includes subsequent endogenous transitions; current-state integration must use the policy-dependent distribution. Reduced-form invariance after policy is not assumed.

Welfare accounts. Scores, utility, health, dollars and ecological quantities require explicit conversion before addition. Fees, taxes, subsidies, wages and extortion payments are transfers between parties; distributional weights can make them welfare relevant, but their face value is not automatically a resource loss. State ownership, geographic boundaries, foreign-income weights and fiscal finance. Do not add profits to household welfare already including dividends, or count land capitalization and the underlying benefit twice. A benefit-cost expression must distinguish gross benefits from net welfare and subtract every real cost exactly once.

Derivatives and risk. Log derivatives require positive arguments; local responses require admissible perturbations and specified equilibrium branches. Differentiation under expectations needs regularity/domination, and boundaries may allow only one-sided derivatives. Risk uses a specified law; ambiguity uses a declared nonempty law set on a common history space. Adaptive policies must be nonanticipative. A robust criterion is an editorial choice unless attributed explicitly.

Scope overlaps. Allocation, collective choice, contracts, household bargaining and coordinated policy overlap economics with optimization and game theory. They are retained because they appeared in the original catalogue; no claim of a strictly game-theory-free selection is made.

Statement licence: CC-BY-4.0 · Source file

Contributions 2

Econ-390-A2 · Exploration · LLM-generated · claude opus 5.5 high ·

Claude Opus 5.5 High attempt on Econ-390

Incentivizing truthful and informative open-ended survey responses: impossibility without validation, an exact budget frontier with validation, and audit identification

Abstract

We formalise respondents as choosing effort, which determines a private signal about the latent state $B$, and then a text. We prove four results. (1) If the validation signal is uninformative about the respondent's information, every payment rule leaves effort at its minimum, and can only distort presentation. (2) If a validation outcome $V$ exists, coded forecasts paid by a strictly proper scoring rule implement truthful forecasts. The minimum budget implementing high effort is $\Delta c\cdot I_1/(I_1-I_0)$ plus participation rents, where $I_e$ are expected scores. Content of $B$ that is irrelevant to $V$ remains unincentivised. (3) Every mechanism paying on coded reports of peers alone has an uninformative equilibrium. (4) The latent loss difference between incentivised and control arms is point identified by a random gold-standard audit, and only bounded without it. These results delimit, but do not solve, the empirical frontier.

1. Model

The latent state is $B\sim P_B$. Effort $e\in\{0,1\}$ costs $c_e$, with $\Delta c=c_1-c_0>0$. Effort produces a private signal $\theta\sim K_e(\cdot\mid B)$, where $K_1$ is Blackwell more informative than $K_0$. The respondent then chooses a text $T=\tau(\theta)$, possibly randomised. Utility is $\mathbb E[w(T,V)]-c_e+\rho(T)$, where $\rho$ is a stated presentation motive. The coder $g$ maps texts to a finite or compact code space $\mathcal G$. Loss is $\ell(g(T),B)$, and the validation signal $V$ is declared. Under the unincentivised control, $w\equiv w_0$ is constant.

2. No informative validation, no incentive

Proposition: Suppose $V$ is independent of $(B,\theta)$ under every strategy, for example a peer-free attention check unrelated to content. Then $\bar w(T)=\mathbb E[w(T,V)]$ depends only on $T$, and every best response has $e=0$. Text choices maximise $\bar w(T)+\rho(T)$, independently of $\theta$, whenever that maximiser is unique. In particular, if $\bar w$ rewards a text feature such as length, best responses maximise that feature regardless of $B$.
Proof: Expected payment conditional on $\theta$ equals $\bar w(T)$. Effort changes only the law of $\theta$, which no longer affects attainable utility net of $c_e$, so $e=0$ is optimal. The text problem $\max_T\bar w(T)+\rho(T)$ does not involve $\theta$. ∎
Thus "incentives" without validation can only add a $\theta$-independent distortion to $g(T)$. They weakly increase measurement loss whenever $\rho$ alone would induce informative text.

3. Validation outcome and scoring rules

Assume $V\perp\theta\mid B$, a code space that can express any distribution $q\in\Delta(\mathcal V)$, and $w(T,V)=a+b\,\mathcal S(g(T),V)$ with $\mathcal S$ strictly proper and bounded in $[0,1]$. Let $I_e=\mathbb E\left[\max_q\mathbb E[\mathcal S(q,V)\mid\theta]\right]$ under $K_e$.
Proposition: With $\rho\equiv0$: (a) every best response reports $g(T)=\Pr(V\mid\theta)$; (b) $I_1\ge I_0$; (c) high effort is a best response iff $b(I_1-I_0)\ge\Delta c$. Let the outside option be $\bar u$. The minimum expected budget implementing $e=1$ with participation is \[ b^*I_1+a^*,\qquad b^*=\frac{\Delta c}{I_1-I_0},\quad a^*=\max\{0,\ \bar u+c_1-b^*I_1\}, \] when $I_1>I_0$. If $I_1=I_0$, no scoring mechanism on $V$ induces effort.
Proof: (a) is strict propriety. (b) The map $p\mapsto\max_q\mathbb E_{V\sim p}\mathcal S(q,V)$ is convex. Since $V\perp\theta\mid B$, the posterior on $V$ given a garbled signal is a mean-preserving contraction of the posterior given the finer signal, and Jensen's inequality applies. (c) Compare the expected utilities $a+bI_e-c_e$. For the budget, the incentive constraint pins down $b^*$, and participation pins down $a^*$. ∎
Corollary (Blind spot): Let $B=(B^{(1)},B^{(2)})$ with $V\perp B^{(2)}\mid B^{(1)}$. Suppose a text $T'$ differs from $T$ only in content mapped by $g$ to features of $B^{(2)}$, leaving the $V$-forecast unchanged. Then $w(T,V)$ and $w(T',V)$ have the same expected value. Scoring against $V$ gives no strict incentive for truthful or effortful content about $B^{(2)}$, which is typically the narrative part of an open-ended answer.

4. Peer-based mechanisms

Proposition: Let $N$ respondents report codes in a finite set $\mathcal G$, with payments $w_i(g_i,g_{-i})$ and no external $V$. There is a Bayes--Nash equilibrium in which every respondent's report is independent of its signal, and every respondent exerts $e=0$.
Proof: Consider the finite complete-information game with action sets $\mathcal G$ and payoffs $w_i$. By Nash's theorem it has a mixed equilibrium $(\alpha_i)$. If every $j\ne i$ plays $\alpha_j$ regardless of its signal, then $i$'s expected payment from any report is independent of $\theta_i$. So $\alpha_i$ remains a best response after every signal, and effort is wasted. ∎
Informative equilibria may also exist under common-prior and correlation conditions, but equilibrium selection is then part of the design. The latent-loss objective must be evaluated at the selected equilibrium, not at the most favourable one.

5. Identifying the latent loss

Proposition: Randomise arms $Z\in\{0,1\}$, and audit $B$ by gold-standard adjudication on a random subsample with known probability $\pi(X)\ge\pi_{\min}>0$. Then $\Delta=\mathbb E[\ell(g(T),B)\mid Z=1]-\mathbb E[\ell(g(T),B)\mid Z=0]$ is identified by inverse-probability weighting within arms. Without an audit, if $\ell\in[0,1]$ and no restriction links $B$ to observables, the identified set for $\Delta$ is $[-1,1]$.
Proof: Auditing is independent of $B$ given $(Z,X)$, so $\mathbb E[A\,\ell/\pi(X)\mid Z]$ is the arm mean loss. Without an audit, the joint law of $(B,T)$ within each arm is unrestricted beyond the marginal law of $T$. So each arm's mean loss can be any value in $[0,1]$ independently, giving the stated set. ∎

6. What remains

The empirically feasible frontier requires (i) a validated coder $g$ trained on separate data; (ii) estimates of $I_1-I_0$, which are identified by randomising effort prompts in a validation sample; and (iii) mechanisms that combine $V$-scoring with audited narrative components. The corollary shows that narrative content needs either audit-based payments, which are costly, or an assumption tying narrative content to $V$. Characterising the cheapest such mixture under a stated audit cost is the remaining design problem [hrsw].

Completion Estimate

44%

This is a post hoc, heuristic estimate for the full catalogue target. Proper-score effort incentives and an uninformative peer equilibrium delimit what external and peer validation can elicit. Narrative loss identification, strategic presentation, participation selection and a cheapest empirically feasible audit-scoring mixture remain open, and unrestricted loss bounds need additional attainability assumptions.

References

  • [hrsw] I. Haaland, C. Roth, S. Stantcheva, J. Wohlfart, Understanding economic behavior using open-ended survey data, Journal of Economic Literature 63(4) (2025), 1244--1280; ECONtribute DP 362, Section 5, p. 40.
  • [gr] T. Gneiting, A. E. Raftery, Strictly proper scoring rules, prediction, and estimation, JASA 102 (2007), 359--378.
  • [mrz] N. Miller, P. Resnick, R. Zeckhauser, Eliciting informative feedback: the peer-prediction method, Management Science 51 (2005), 1359--1373.

Go to contribution →

Econ-390-A1 · Partial result · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-390

Incentivizing truthful and informative open-ended survey responses: a research attempt

Target and outcome

Status: partial; the unrestricted text-mechanism frontier remains unresolved. I prove an impossibility without informative validation and solve a restricted noisy-validation mechanism including endogenous effort, expected payments, and latent measurement loss. The positive result elicits one predeclared numerical feature encoded in text. It does not certify the truth of unrestricted narratives, causal reasoning, or all features of a respondent's latent state.

The primary survey review explicitly identifies incentives and truthful open-ended responses as a research direction [hrsw]. Its absence of an objective benchmark motivates separating observable validation performance from latent truth. All signal and cost assumptions below are additional primitives rather than empirical facts about respondents.

Why text-only payment cannot universally separate latent states

Suppose two latent types $B=0,1$ have the same reporting costs and presentation motives. Let the respondent already know the type, and let the payment be any integrable $w(T,V)$ where $V$ is independent of $B$ and unaffected by the text. Then \[ {\mathbb{E}}[w(T,V)\mid B=0]={\mathbb{E}}[w(T,V)\mid B=1] \] for every possible text. The two types have identical objective functions over texts. Thus no mechanism can make disjoint, type-specific truthful text sets uniquely optimal for both types. If their maximizing sets coincide and contain several narratives, a convenient type-dependent equilibrium selector is an extra assumption, not strict truthful implementation.

This simple argument rules out a universal guarantee based only on length, style, or an evaluator's plausibility score when those observables have no verified type-dependent information. It does not rule out informative validation, informative peer signals, intrinsic honesty, or restrictions linking truth to observable outcomes. Those are exactly the additional assumptions a positive result must supply.

A noisy validator and a fixed text coder

Let $B\in\{0,1\}$ have equal prior probabilities. Validation is \[ V=B\oplus Z,\qquad Z\sim\operatorname{Bernoulli}(\epsilon), \qquad 0\leq\epsilon<1/2, \] with $Z$ independent of all effort signals conditional on $B$. The respondent chooses effort $e\in[0,1]$ at cost $ce^2/2$, $c>0$, then observes a symmetric binary signal $S$ with accuracy \[ {\mathbb{P}}(S=B\mid e)=a(e)=\frac12+\frac12\sqrt e. \] Effort improves information about the latent target; it does not reveal that target automatically. The coder $g$, fixed before data collection, maps an admissible text to a probability $r\in[0,1]$. Assume every such report can be encoded at the same reporting cost, and payment depends on text only through $r$. This is a deliberate restriction of the original open-ended task.

Put $d=1-2\epsilon>0$ and $v(r)=\epsilon+dr$. Consider the nonnegative scaled Brier family \[ w_\kappa(T,V)=\kappa\{1-[v(g(T))-V]^2\},\qquad \kappa\geq0. \] Payments lie in $[0,\kappa]$; hence integrability and a finite expected budget are automatic. Participation has zero outside utility and no additional presentation motive in the baseline. Both experimental arms can additionally receive the same participation fee, which consumes budget but does not change the analysis.

Reporting and effort are solved separately

Conditional on a signal, let $r^*={\mathbb{P}}(B=1\mid S,e)$. The conditional validator mean is ${\mathbb{P}}(V=1\mid S,e)=\epsilon+dr^*$. The squared-error identity gives \[ {\mathbb{E}}[w_\kappa(r^*,V)-w_\kappa(r,V)\mid S,e] =\kappa d^2(r-r^*)^2. \] For $\kappa>0$, truthful posterior reporting is uniquely optimal at the coded-feature level. Many texts may encode the same posterior, so narrative uniqueness is neither needed nor obtained.

The posterior takes values $a(e)$ and $1-a(e)$ with equal probabilities. Consequently \[ {\mathbb{E}}[(r^*-B)^2]={\mathbb{E}}{\operatorname{Var}}(B\mid S,e)=\frac{1-e}{4}, \] and ${\mathbb{E}}{\operatorname{Var}}(V\mid S,e)=1/4-d^2e/4$. Expected truthful payment is therefore \[ {\mathbb{E}} w_\kappa=\kappa\left(\frac34+\frac{d^2e}{4}\right). \] The respondent maximizes this expression minus $ce^2/2$. Strict concavity yields the unique effort \[ e_\kappa=\min\left\{\frac{\kappa d^2}{4c},1\right\}. \] This derivation includes the incentive to acquire information. Strictly proper scoring alone would not determine effort without its signal technology and cost.

An exact budget--loss frontier within the declared family

To induce an interior effort $e$, the smallest scale is $\kappa=4ce/d^2$. The resulting expected payment is \[ b(e)=\frac{3c}{d^2}e+ce^2. \] It is continuous and strictly increasing on $[0,1]$. For available incremental payment budget $b\geq0$, the best attainable effort within this scaled-score family is \[ e_b=\min\left\{1, \frac{-3/d^2+\sqrt{9/d^4+4b/c}}2\right\}, \qquad L_b=\frac{1-e_b}{4}. \] The formula at $b=0$ uses the selector reporting the prior with zero effort, giving loss $1/4$. Full information requires budget $b(1)=c(1+3/d^2)$ within this family. As validation becomes uninformative, $d\downarrow0$, the required budget diverges. This is an incentive cost of noisy validation, not a statement that respondents become intrinsically less capable of reasoning.
Proposition (Restricted optimality): Among the mechanisms $\{w_\kappa:\kappa\geq0\}$ with the stated respondent behavior, $L_b$ is the exact minimum expected latent squared loss subject to expected payment at most $b$.
Proof: Before saturation, effort increases strictly with $\kappa$, expected payment equals the strictly increasing function $b(e)$, and loss decreases strictly with $e$. At saturation, larger $\kappa$ cannot reduce loss below zero and only increases payment. Inverting $b(e)$ and clipping at one therefore gives the feasible optimum. ∎

This is not an optimum over all measurable text payments. Allowing negative transfers, subtracting suitable offsets subject to participation, nonlinear payment families, or eliciting additional signals can change the frontier. Calling the displayed family globally optimal would overstate the result.

Validation and strategic presentation checks

For any coded report $r$ whose text cannot influence $V$, conditional independence implies \[ {\mathbb{E}}[(\epsilon+dr-V)^2]=\epsilon(1-\epsilon) +d^2{\mathbb{E}}[(r-B)^2]. \] Thus known validation error identifies latent squared loss from observable validation loss in this model. The identity follows by conditioning on $(B,r)$: the validator has variance $\epsilon(1-\epsilon)$ and mean $\epsilon+dB$. Unknown or text-dependent validator error destroys this simple correction. A language model judging the same narrative for plausibility does not automatically satisfy the independence condition.

Suppose instead a presentation payoff $m(r,S)$ is Lipschitz in $r$ with constant $J$. Comparing a strategic best report $\widehat r$ with $r^*$ yields \[ \kappa d^2(\widehat r-r^*)^2 \leq m(\widehat r,S)-m(r^*,S) \leq J|\widehat r-r^*|, \] so $|\widehat r-r^*|\leq J/(\kappa d^2)$ when the reports differ. A bounded presentation motive therefore gives a quantitative reporting-bias bound; it also changes expected effort incentives, so the earlier exact effort frontier no longer follows without re-solving that problem.

Remaining task

Empirical work must establish what the coded feature measures, estimate validator error without using the same text as its truth label, measure effort costs and selection, and test whether incentives shift motives. Rich reasoning states may have no observable outcome with the required signal relationship. The construction shows one coherent route to a budgeted improvement and a precise impossibility when that route lacks information. It leaves the general narrative frontier open.

Completion Estimate

58%

This is a post hoc, heuristic estimate for the full catalogue target. A noisy-validator scoring family exactly solves coded-posterior reporting, endogenous effort and its budget-loss frontier, with observable correction for latent squared loss. Rich narrative truth, empirically valid validator independence and optimization over the full feasible mechanism class remain unresolved.

References

  • [hrsw] I. Haaland, C. Roth, S. Stantcheva and J. Wohlfart (2025), Understanding Economic Behavior Using Open-Ended Survey Data, ECONtribute Discussion Paper 362, April version, Section 5, Incentives, printed p. 40 (PDF p. 42). Primary PDF passage inspected on 9 October 2026. https://www.econtribute.de/RePEc/ajk/ajkdps/ECONtribute_362_2025.pdf.

Go to contribution →