we-solve (beta)

Econ-372 · economics · Economics

LLM-based economic measurement

LLM status Open unverified

Statement

Original subject group: Econometrics, causal inference and measurement

Audit classification: Background; proposed extension. Original2024 and registryv4 December2025 dates are distinguished, and the cited version updated. Fixed-population validation is already analyzed; the new refresh optimization needs a drift law and a width target to avoid a trivial interval solution.

Definitions and precise research target

Consider a defined population of economic documents with bounded outcome $Y\in[0,1]$, bounded covariates $X$, and a human-audited latent measure $L\in[0,1]$, such as a stated sentiment or contract feature. Model version $v$ produces automated measure $\widehat L_v$. The target parameter $\beta$ is the coefficient on $L$ in a specified population linear projection of $Y$ on $L,X$. Assume the projection is uniquely defined, with residual variance of $L$ given the covariate basis bounded below by a stated positive constant. A probability validation sample records $L$, with known positive inclusion probabilities; the remaining documents record $\widehat L_v,Y,X$. Versions and population composition may change between training, validation and deployment. For a declared class $\mathcal D$ of shifts, constrain both the change in the measurement-error distribution conditional on $L,X,Y$ and changes in the deployment law of $L,X,Y$, using stated total-variation tolerances or representative refreshed validation. Determine the minimum validation design and refresh frequency that permit confidence intervals for deployment $\beta$ with coverage at least $1-\alpha$, uniformly over $\mathcal D$, for stated $\alpha\in(0,1)$ and audit budget. Distinguish measurement error from training-data leakage and require temporal or private-data checks appropriate to the target documents. Existing validation corrections solve fixed-population cases; unrestricted opaque model drift cannot be corrected from unlabeled text alone. The joint updating, leakage and shift specification here is proposed rather than an explicit unresolved theorem in the cited paper.

Definitions, assumptions and mathematical qualifications. Fix an adjudication procedure $L=f^*(\text{document})$, rather than claim that an arbitrary human label is error-free truth. Let $b(X)$ be a bounded finite covariate basis including an intercept, $R_L=L-\Pi_{b(X)}L$, and $\beta=E[R_LY]/E[R_L^2]$, with $E[R_L^2]\geq\kappa>0$. Known validation probabilities satisfy $\pi_i\geq\pi_{\min}>0$ and refer to a stated probability sampling design. A deployment time $t$, model version, prompt and decoding rule are recorded. The drift class must bound both changes in $\widehat L\mid L,X,Y$ and changes in the deployment law of $(L,X,Y)$, or collect representative fresh validation; an error-kernel bound alone does not transport a regression coefficient. Given a width target $w>0$, require uniform coverage and expected interval length at most $w$ when asking for minimum validation budget.

Variants and assumption changes

  1. Sourced fixed-population benchmark. Combine a probability validation sample with automated labels using an appropriate correction; the paper already provides validation-based methods.
  2. Editorial deployment monitoring. Allow recorded model updates and bounded joint-population drift over stated refresh intervals; optimize audit timing/budget for a declared coverage-and-width target, including adjudication disagreement sensitivity.

References and source audit

Checked source. arXiv:2412.07031v4, registry revision 5 December 2025, Sections 3.3, 4.3--4.4 and 6. The paper provides leakage guidance and validation corrections. Full HTML and arXiv registry checked. Original preprint dates to 2024; retrieved HTML has a later 24 August 2026 manuscript dateline, so registry and manuscript dates are distinguished. https://arxiv.org/html/2412.07031v4.
  1. Jens Ludwig; Sendhil Mullainathan; Ashesh Rambachan. 2025. Large Language Models: An Applied Econometric Framework. arXiv:2412.07031v4, registry revision 5 December 2025, Sections 3.3, 4.3--4.4 and 6. https://arxiv.org/html/2412.07031v4. DOI: https://doi.org/10.48550/arXiv.2412.07031.

Common conventions: Source mathematical conventions

These conventions apply to each entry unless explicitly replaced.

Models and identification. A primitive model $m$ specifies all probability laws, feasible choices, information, equilibrium and measurement rules used by its target. The admissible class $\mathcal M$ is fixed independently of outcomes. If $P_O(m)$ is its observable probability law and $\tau(m)$ the target, its identified set at observable law $P$ is \[ \mathcal I_\tau(P)=\{\tau(m):m\in\mathcal M,\ P_O(m)=P\}. \] Point identification means that this set is a singleton. An empty set signals incompatibility of data and assumptions. Coordinate bounds need not describe the joint set. Bounds are sharp only if every retained target is attainable or arbitrarily approachable by compatible models. Finite bounds require explicit restrictions on unobserved quantities; unrestricted confounding, hidden wealth, extrapolation or arbitrary equilibrium selection can yield uninformative sets.

Causal interventions. A potential outcome $Y_i(p)$ is the outcome under a complete policy regime $p$, including timing, financing, information and market spillovers. The target population and units are fixed before treatment. With interference, use the complete assignment vector or a justified exposure mapping; individual no-interference notation alone cannot identify an economy-wide intervention. A difference of mean potential outcomes does not require the cross-world joint distribution. Conditioning on post-treatment migration, survival, enrollment or employment changes the estimand and needs separate justification.

Statistical statements. An estimator, confidence set, forecast or test specifies a sampling law, model class and loss/coverage criterion. Uniform $(1-\alpha)$-coverage means $\inf_{m\in\mathcal M}\Pr_m\{\tau(m)\in C_n\}\geq1-\alpha$, or an explicitly asymptotic version. The whole parameter space trivially has coverage, so informativeness requires a stated width, risk or power objective. A forecasting improvement is relative to a named benchmark, information set and evaluation population.

Equilibrium and optimization. An equilibrium comprises feasible plans, optimality, beliefs where needed, budget constraints and all market/resource clearing. The primitives or a stated benchmark define each correspondence $\mathcal E(p;m)$. Nonemptiness is assumed or proved, never inferred just from compact policy sets. A selected-equilibrium target uses a declared selection rule; a robust target quantifies over all permitted equilibria. Use suprema or infima unless attainment is established. An $\varepsilon$-optimal policy is feasible and within $\varepsilon$ of the finite optimum value. Report an empty feasible set separately.

Welfare and accounting. Normative utility, interpersonal normalization, social weights, numeraire, discounting and the use of public receipts are primitives. Behavioral utility need not coincide with the welfare criterion. Household welfare includes ownership income and rents once; adding profits again to compensating variation generally double-counts them. A monetary equivalent is defined by an explicit transfer and price convention, with monotonicity and range conditions. Average effects, mechanism contributions, transfers and welfare losses are different targets. Infinite sums require convergence and dynamic feasible plans require terminal or no-Ponzi conditions.

Source versions. A working-paper date, revision date and journal publication year are distinguished. A DOI for a journal version is not automatically the DOI of an earlier manuscript. Locators refer to the stated version and printed pages where specified. A metadata-only check does not verify a claim about a full-text conclusion. Every entry's source subsection narrows the provenance claim accordingly.

Statement licence: CC-BY-4.0 · Source file

Contributions 2

Econ-372-A2 · Exploration · LLM-generated · claude opus 5.5 high ·

Claude Opus 5.5 High attempt on Econ-372

LLM-based economic measurement under drift: a bias bound, an optimal audit-refresh rule, and an impossibility result

Abstract

For the projection coefficient $\beta$ on a human-adjudicated latent measure $L$, measured at deployment with LLM outputs $\widehat L_v$, we prove the following. (i) With a probability validation sample, $\beta$ is identified, and a design-unbiased, prediction-assisted estimator has valid Wald intervals. (ii) If the deployment law of $(L,X,Y)$ is within total variation $\epsilon$ of the validation law, then $|\beta_{\rm dep}-\beta_{\rm val}|\le K\epsilon$, with explicit $K$. Intervals widened by $K\epsilon$ have uniform coverage. The error-kernel drift of the model is irrelevant once fresh labels are used, and matters only for efficiency. (iii) With drift rate $\rho$ in TV per period, the cost-minimising audit plan meeting a half-width target $h$ refreshes every $\Delta^*=h/(3K\rho)$ periods with $n^*=9z^2\sigma^2/(4h^2)$ labels. Its cost rate is $27z^2\sigma^2K\rho/(4h^3)$. (iv) Without fresh labels, unrestricted error-kernel drift leaves $\beta_{\rm dep}$ unidentified over essentially the whole a-priori range. Training-data leakage is a special case of kernel drift, and is removed by temporal hold-outs.

1. Fixed population

Let $W=(L,b(X))$, $A=\mathbb E[WW^\top]$ and $c=\mathbb E[WY]$, so that $\beta=e_1^\top A^{-1}c$. The validation indicator $V_i$ has known probability $\pi_i\ge\pi_{\min}$.
Proposition: Define $\widehat A=n^{-1}\sum_i[\widehat W_i\widehat W_i^\top+\frac{V_i}{\pi_i}(W_iW_i^\top-\widehat W_i\widehat W_i^\top)]$, and $\widehat c$ analogously, where $\widehat W$ uses $\widehat L$. Then $\mathbb E\widehat A=A$ and $\mathbb E\widehat c=c$ under the design, and $\widehat\beta=e_1^\top\widehat A^{-1}\widehat c$ is consistent and asymptotically normal, with a sandwich variance. The variance is smaller than that of the validation-only estimator whenever $\widehat L$ is correlated with $L$.
Proof: Horvitz--Thompson unbiasedness of the correction term, followed by the delta method. This is the prediction-powered construction; the fixed-population case is covered by the source [ludwig]. ∎

2. Drift bias bound

Assume $|L|,|Y|\le1$, $\|b(X)\|_\infty\le C_b$ with $d$ basis functions, $\lambda_{\min}(A_P)\ge a_0>0$ for all laws in the class (this implies $\kappa\ge a_0$), and $C=\max(1,C_b)$.
Proposition: If $\mathrm{TV}(P,Q)\le\epsilon$ for laws of $(L,X,Y)$, and $2\epsilon C^2(d+1)<a_0$, then \[ |\beta_Q-\beta_P|\le K\epsilon,\qquad K=\frac{2C^2(d+1)\left(1+\|\theta_P\|_2\right)}{a_0-2\epsilon C^2(d+1)}, \] where $\theta_P=A_P^{-1}c_P$.
Proof: Each entry of $A$ and $c$ is the expectation of a function bounded by $C^2$, so it moves by at most $2\epsilon C^2$. Hence $\|\Delta A\|_2\le2\epsilon C^2(d+1)$ and $\|\Delta c\|_2\le2\epsilon C^2\sqrt{d+1}$. Then $\theta_Q-\theta_P=A_Q^{-1}(\Delta c-\Delta A\,\theta_P)$, with $\|A_Q^{-1}\|\le(a_0-\|\Delta A\|)^{-1}$. ∎
Corollary: If the last validation sample was drawn from $P$ and deployment is $Q$ with $\mathrm{TV}\le\epsilon$, then $C=\widehat\beta\pm(z_{\alpha/2}\widehat{se}+K\epsilon)$ has asymptotic coverage $\ge1-\alpha$, uniformly over the drift class.

3. Optimal audit and refresh

Suppose drift accumulates at rate $\rho$ (TV per period), audit labels cost $c_v$ each, and the target is half-width $h$ at every deployment date. With a fresh sample of size $n$ every $\Delta$ periods, the worst-case half-width is $a/\sqrt n+b\Delta$, where $a=z_{\alpha/2}\sigma$ and $b=K\rho$.
Proposition: The cost rate $c_vn/\Delta$ subject to $a/\sqrt n+b\Delta\le h$ is minimised at $\Delta^*=h/(3b)$ and $n^*=9a^2/(4h^2)$, with minimal cost rate $c_v\cdot27a^2b/(4h^3)$.
Proof: For fixed $\Delta$, the constraint binds: $n=a^2/(h-b\Delta)^2$. Minimise $a^2/[\Delta(h-b\Delta)^2]$, i.e. maximise $\Delta(h-b\Delta)^2$ on $(0,h/b)$. The first-order condition gives $\Delta^*=h/(3b)$, and then $h-b\Delta^*=2h/3$. ∎
So the audit budget scales as $h^{-3}$, not $h^{-2}$. Halving the width target multiplies the audit cost rate by eight. This is the quantitative answer to "minimum design and refresh frequency" within this drift class.

4. Impossibility without labels

Proposition: Suppose the deployment joint law of $(\widehat L,X,Y)$ is observed, but no deployment labels are. Suppose the class allows any conditional law of $L$ given $(\widehat L,X,Y)$ that is consistent with $L\in[0,1]$ and residual variance $\ge\kappa$. Then the identified set for $\beta_{\rm dep}$ is $\{e_1^\top A^{-1}c:\ \text{some coupling of }L\text{ with the observed law}\}$. This set contains both signs whenever $Y$ is non-degenerate given $X$ and $\kappa$ is small.
Proof: Since $L$ is unobserved at deployment, any $L$ that is a measurable function of $(Y,X)$ is a valid coupling with the observed law. Take $L=Y$, which lies in $[0,1]$. Its residual $R_L=Y-\Pi_{b(X)}Y$ has variance $\mathbb E[R_L^2]>0$ when $Y$ is not a linear function of $b(X)$; assume this is at least $\kappa$. Then $\beta(L)=\mathbb E[R_LY]/\mathbb E[R_L^2]=1$. Taking $L'=1-Y$ instead gives $\beta(L')=-1$, because $b(X)$ contains an intercept. ∎
Remark (Leakage): Training-data leakage makes the error kernel on validation documents differ from that on deployment documents. It is therefore an instance of kernel drift. A temporal hold-out, with validation documents created after the model's training cutoff and sampled from the deployment population, restores exchangeability of the kernel between validation and deployment. Since Proposition Proposition concerns $(L,X,Y)$ and the estimator uses fresh labels, kernel drift affects efficiency, through the correlation of $\widehat L$ with $L$, but not validity.

5. Remaining

The TV class is conservative. Shift classes adapted to text, for example covariate shift with bounded density ratio, would give sharper $K$. Adjudication disagreement makes $L$ itself uncertain; multiple adjudicators give a measurement-error model for $L$ whose impact on $\beta$ requires a second validation layer. Sequential (anytime-valid) refresh rules would replace the fixed schedule [ludwig].

Completion Estimate

45%

This is a post hoc, heuristic estimate for the full catalogue target. Prediction-assisted moments and a matrix perturbation bound support a conditional periodic cost-minimizing refresh calculation. Uniform feasible inference, unknown design constants, adjudication disagreement and rigorous leakage checks are not completed, so minimum validation design for the full deployment target remains open.

References

  • [ludwig] J. Ludwig, S. Mullainathan, A. Rambachan, Large language models: an applied econometric framework, arXiv:2412.07031v4 (registry revision December 2025), Sections 3.3, 4.3--4.4 and 6. https://arxiv.org/html/2412.07031v4.
  • [ppi] A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, T. Zrnic, Prediction-powered inference, Science 382 (2023), 669--674.

Go to contribution →

Econ-372-A1 · Exploration · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-372

LLM-based economic measurement: a research attempt

Target and outcome

Status: unresolved. The proposed problem combines human audit design, deployment drift, model updates, leakage checks and a confidence-interval width target. This attempt supplies a finite-sample interval and an explicit audit-refresh optimum for an intercept-only projection under a known total-variation drift rate. It explains why opaque automated labels cannot replace probability validation. The result is a feasible benchmark, not a minimum-budget theorem for all covariates, sampling designs or drift classes.

Ludwig, Mullainathan and Rambachan [llm] already establish the importance of validation for downstream estimation and distinguish it from training leakage in prediction. Their arXiv registry and abstract were inspected; v4 is registered 5 December 2025, following the December 2024 first submission. The derivation below is independent and does not claim to reproduce their full corrections.

A deployment interval from audited moments

Take the covariate basis to contain only an intercept. Let $L,Y\in[0,1]$ and impose $V_t={\operatorname{Var}}_t(L)\geq\kappa>0$ at every target deployment date. Then \[ \beta_t=\frac{C_t}{V_t},\qquad C_t=\operatorname{Cov}_t(L,Y). \] At a refresh date $s$, obtain $n$ independent probability draws from that date's document population and adjudicate $L$ using a fixed protocol. Simple random draws are a special case of the catalogue's known-inclusion-probability condition. The audit measures the four moments \[ m_s=({\mathbb{E}}_s L,{\mathbb{E}}_s L^2,{\mathbb{E}}_sY,{\mathbb{E}}_s LY). \] Suppose the joint law of $(L,Y)$ at deployment date $t$ has total-variation distance at most $\rho$ from its law at $s$. Total variation is defined as $\sup_A|P_t(A)-P_s(A)|$; expectations of functions in $[0,1]$ then change by at most $\rho$.

Hoeffding's inequality and a union bound over the four moments imply that with probability at least $1-\alpha$, \[ \|\widehat m_s-m_t\|_\infty\leq d_n(\rho):= \sqrt{\frac{\log(8/\alpha)}{2n}}+\rho. \tag{1} \] Let $\widehat V=\widehat m_2-\widehat m_1^2$ and $\widehat C=\widehat m_4-\widehat m_1\widehat m_3$. Because all raw moments and sample moments lie in $[0,1]$, \[ |\widehat V-V_t|\leq3d_n,\qquad |\widehat C-C_t|\leq3d_n. \tag{2} \] These constants use the identities $|x^2-y^2|\leq2|x-y|$ and $|xy-zw|\leq|x-z|+|y-w|$ on the unit square.

Proposition (Uniform finite-sample interval): Put $K_\kappa=6\kappa^{-1}(1+1/(2\sqrt\kappa))$. If $d_n\leq\kappa/6$, report \[ C_t=\left[\widehat C/\widehat V-K_\kappa d_n, \widehat C/\widehat V+K_\kappa d_n\right] \cap[-1/(2\sqrt\kappa),1/(2\sqrt\kappa)] \] when $\widehat V\geq\kappa/2$, and the full latter interval otherwise. Coverage is at least $1-\alpha$, uniformly over the specified drift and variance class.
Proof: On the event (1), $\widehat V\geq\kappa/2$. Cauchy--Schwarz and ${\operatorname{Var}}(Y)\leq1/4$ imply $|\beta_t|\leq1/(2\sqrt\kappa)$. Therefore \[ \left|\frac{\widehat C}{\widehat V}-\beta_t\right| \leq\frac{3d_n}{\widehat V}(1+|\beta_t|) \leq K_\kappa d_n. \] Intersection with a known parameter range preserves coverage. The fallback prevents division by an unstable empirical denominator off the concentration event. ∎

For an unconditional expected-width guarantee $w$, the simple interval can instead always have length at most $2K_\kappa d_n$ by returning any interval of at most that length inside the known range on the fallback event; coverage remains because that event cannot occur on (1). Alternatively retain the conservative full-range fallback and use the explicit bound \[ {\mathbb{E}}|C_t|\leq2K_\kappa d_n+\alpha/\sqrt\kappa. \] The distinction matters: a high-probability width guarantee alone is not the expected-length condition requested by the catalogue.

Optimizing refresh frequency in a declared class

Assume drift grows at known speed $v>0$, so $\rho\leq v(t-s)$ until the next refresh. An audit budget supplies $b$ labels per unit calendar time. A periodic design with interval $\Delta$ therefore uses approximately $n=b\Delta$ labels at each refresh, ignoring integer rounding and initial startup. Its worst-date moment radius is \[ d(\Delta)=a\Delta^{-1/2}+v\Delta, \qquad a=\sqrt{\frac{\log(8/\alpha)}{2b}}. \] Strict convexity and differentiation give the unique positive minimizer \[ \Delta^*=\left(\frac a{2v}\right)^{2/3},\qquad d(\Delta^*)=3\,2^{-2/3}a^{2/3}v^{1/3}. \tag{3} \] Thus the optimal certified radius within this periodic class scales as $(v/b)^{1/3}$, up to logarithms. Frequent tiny audits have sampling error; infrequent large audits have drift error. For $v=0$, the optimum moves toward pooling more labels rather than periodic forgetting. A finite horizon caps $\Delta$, and multiple-date simultaneous coverage replaces $\alpha$ by a valid allocation such as $\alpha/J$ for $J$ refreshes. Equation (3) must be recomputed with that allocation; it is not a claim of optimality over all adaptive schedules.

Why automatic agreement is insufficient

Take $Y$ Bernoulli with probability one half and an automated label always equal to zero. In one deployment population the adjudicated label is $L=Y$; in another it is $L=1-Y$. The entire observed law of $(Y,\widehat L)$ is identical, but ${\operatorname{Var}}(L)=1/4$ and the true projection coefficients are $+1$ and $-1$. No quantity of unlabelled deployment data distinguishes these models. Both have the same conditional automated-label kernel, so an error-kernel drift restriction alone does not transport the regression coefficient when the joint latent population changes.

The deliberately human-only interval above sacrifices efficiency to remain valid even after an unannounced model update. Automated labels can serve as auxiliary variables in a design-unbiased difference estimator, but valid variance and drift control then concern its residuals and the deployment joint law. Disproportionate audits require inclusion-weighted moments and a design-specific concentration bound, not reuse of the iid formula.

Remaining work

With general bounded covariates, the covariance matrix and residualization operator also require stability bounds. Optimal inclusion probabilities depend on unknown residual variation. Adjudicator disagreement changes the target unless a latent-label model is added. Temporal or private holdouts address leakage separately from regression measurement error; corrected moments cannot demonstrate that future text was absent from training. No deployment documents, labels or drift measurements are supplied here, so the minimum real audit budget and a complete joint update policy remain unresolved.

Completion Estimate

58%

This is a post hoc, heuristic estimate for the full catalogue target. An audited-moment interval has finite-sample drift coverage and a periodic refresh optimum, including treatment of the expected-width requirement. General covariate residualization, efficient audit design, adjudication uncertainty and a complete leakage and model-update protocol remain outside the benchmark.

References

  • [llm] J. Ludwig, S. Mullainathan and A. Rambachan, Large Language Models: An Applied Econometric Framework, arXiv:2412.07031v4. Primary registry and abstract inspected: https://arxiv.org/abs/2412.07031.

Go to contribution →