we-solve (beta)

Econ-595 · economics · Economics

Long-run AI tutoring and independent learning

LLM status Open unverified

Statement

Original subject group: Education, families and demography

Audit classification: Background; proposed extension. Added the existing 2025 three-arm tutoring trial and affiliation-only correction; retained six-month durability as an editorial extension.

Definitions and precise research target

Editorial formalization. Consider ninth-grade mathematics pupils in public schools in one district during a single academic year. Let $a\in\{0,1,2\}$ denote randomized access to ordinary study materials, a general artificial-intelligence assistant, or a constrained tutoring assistant that prompts reasoning before offering answers. Let $Y_i(a)$ be a standardized examination score six months later when no assistant is available, $R_i(a)$ an independently scored novel-problem reasoning measure, and $T_i(a)$ study time. For every $a\in\{1,2\}$, define \[\tau_Y(a)=E[Y_i(a)-Y_i(0)],\quad\tau_R(a)=E[R_i(a)-R_i(0)],\quad\tau_T(a)=E[T_i(a)-T_i(0)].\] Use classroom-level randomized offers, preserve baseline pupil inclusion, and specify cross-classroom sharing and teacher adjustments. Determine whether assisted performance translates into durable independent learning, distinguishing faster homework completion from retained skills. Predefine model versions, permitted prompts, test security, and outcomes before assignment; usage differences alone do not identify causal effects. Estimate uncertainty and heterogeneity by baseline achievement while correcting for prespecified multiple comparisons. Existing immediate experimental comparisons are acknowledged; the proposed extension concerns the stated six-month follow-up and independent reasoning.

Treat access as a version-controlled offered deployment, including hardware, prompt policy, teacher oversight, and study permissions. $Y_i(a)$ is a closed-book assistant-free assessment six months after the intervention ends, not an assessment immediately after practice. Use fixed score/rubric scales and include absent or transferred pupils under a declared missingness rule. Positive classroom assignment probabilities and independence of assignment from potential outcomes identify intention-to-treat contrasts if other classrooms do not affect outcomes; otherwise use a defined peer-exposure design. Existing three-arm experimental evidence establishes that the immediate comparison has been studied. The proposed remaining target is longer persistence and independent reasoning in this specified population; no absence-of-all-prior-evidence claim is justified. All expectations use a stated baseline population and probability law. Identification means every admissible data-generating model with the same observed-data law yields the same displayed target; otherwise report the identified set. Exact equations, horizons and assumptions are editorial, rather than source-given canonical conjectures.

Variants and assumption changes

  1. Editorial duration variant. Compare assistant-free outcomes immediately, six months and two years after access ends. Existing immediate answers are not assumed to transport.
  2. Editorial teacher-complement variant. Cross AI access with randomized teacher training and estimate the interaction, including equal-cost tutoring/hardware controls.

References and source audit

1. Program overview verified; exact tutoring trial is not an open conjecture. Locator: NBER Reporter, 13 July 2026; introductory program description. Primary indexed record inspected; direct article retrieval failed. URL: https://www.nber.org/reporter/2026number2/program-report-economics-education.

2. Existing three-arm tutoring experiment measures immediate unassisted performance; six-month persistence is an editorial extension. Locator: PNAS 122(26), e2422633122; Experimental Design and Main Results. Full text or primary publication page inspected. URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC12232635/.

3. Correction concerns an author affiliation; does not replace the experimental results. Locator: Correction notice. Primary indexed correction inspected; direct PMC retrieval met a challenge page. URL: https://pmc.ncbi.nlm.nih.gov/articles/PMC12403119/.

  1. Caroline M. Hoxby. 2026. Program Report: Economics of Education. NBER Reporter, 13 July 2026; introductory program description. https://www.nber.org/reporter/2026number2/program-report-economics-education.
  2. Hamsa Bastani; Osbert Bastani; Alp Sungu; Haosen Ge; Özge Kabakcı; Rei Mariman. 2025. Generative AI without guardrails can harm learning: Evidence from high school mathematics. PNAS 122(26), e2422633122; Experimental Design and Main Results. https://pmc.ncbi.nlm.nih.gov/articles/PMC12232635/. DOI: https://doi.org/10.1073/pnas.2422633122.
  3. Hamsa Bastani; Osbert Bastani; Alp Sungu; Haosen Ge; Özge Kabakcı; Rei Mariman. 2025. Correction for Bastani et al., Generative AI without guardrails can harm learning: Evidence from high school mathematics. Correction notice. https://pmc.ncbi.nlm.nih.gov/articles/PMC12403119/. DOI: https://doi.org/10.1073/pnas.2518204122.

Integrated refinement: ECON-043

AI-assisted education: learning, dependence, and governance

This refinement adopts the additional source conventions (audited:conventions), subject to its local restrictions. Governance, dependence and implementation costs. For governance $g\in\mathcal G$, define delayed unaided learning $L_i(a,g)$, predeclared dependence $D_i(a,g)$ and real per-student cost $C(a,g)$. Compare \[ W(g)=p_L\mathbb E[L_i(1,g)-L_i(0,g_0)] -p_D\mathbb E[D_i(1,g)-D_i(0,g_0)] -[C(1,g)-C(0,g_0)], \] with declared monetary conversion weights $p_L,p_D\geq0$. Identify the separate outcome effects and costs before optimizing feasible governance. Without credible weights, report a learning/dependence/privacy/equity vector or use explicit constraints. Specify delay, scoring, tool version, assessment permissions, supervision, baseline population, missingness and peer exposure. Assignment effects include nonuse; user-only effects need selection assumptions. Immediate assisted performance is distinct from retained independent learning, and an unrestricted governance class requires attainment conditions.

Supplied source audit for ECON-043

Local [S1], [S2], etc. in this audit and the references immediately below belong to ECON-043, independently of the original entry's reference numbering. [S1]'s evidence-gap chapter and [S2]'s education-technology topic support AI governance and independent learning assessment. The dependence metric and monetized objective are editorial; the review does not establish a universal net benefit.

References for integrated refinement ECON-043

[S1] Abhijeet Singh, Laia Navarro-Sola, and Philip Oreopoulos (2025). Education Technology. VoxDevLit 20(1), December 2025.

Verified locator / source role: Conclusion/evidence-gap chapter at the linked primary review URL, inspected for the agenda. See the audit conclusion for distinctions between direct agenda support and editorial extensions.

https://voxdev.org/voxdevlit/education-technology/edtech-evidence-gaps

[S2] Oliver Hanney / VoxDev (living compilation). Evidence gaps: Key unanswered questions in development economics. Accessed 8 October 2026. The page currently covers 20 topics; the "16-topics URL is a historical slug.

Verified locator / source role: Topic heading: Education Technology. Supports the broad research agenda; this is not a verbatim quotation or an attribution of the displayed mathematical target.

https://voxdev.org/topic/evidence-gaps-key-unanswered-questions-across-16-topics-development-economics

Common conventions: Source mathematical conventions

These conventions apply to each entry unless explicitly replaced.

Models and identification. A primitive model $m$ specifies all probability laws, feasible choices, information, equilibrium and measurement rules used by its target. The admissible class $\mathcal M$ is fixed independently of outcomes. If $P_O(m)$ is its observable probability law and $\tau(m)$ the target, its identified set at observable law $P$ is \[ \mathcal I_\tau(P)=\{\tau(m):m\in\mathcal M,\ P_O(m)=P\}. \] Point identification means that this set is a singleton. An empty set signals incompatibility of data and assumptions. Coordinate bounds need not describe the joint set. Bounds are sharp only if every retained target is attainable or arbitrarily approachable by compatible models. Finite bounds require explicit restrictions on unobserved quantities; unrestricted confounding, hidden wealth, extrapolation or arbitrary equilibrium selection can yield uninformative sets.

Causal interventions. A potential outcome $Y_i(p)$ is the outcome under a complete policy regime $p$, including timing, financing, information and market spillovers. The target population and units are fixed before treatment. With interference, use the complete assignment vector or a justified exposure mapping; individual no-interference notation alone cannot identify an economy-wide intervention. A difference of mean potential outcomes does not require the cross-world joint distribution. Conditioning on post-treatment migration, survival, enrollment or employment changes the estimand and needs separate justification.

Statistical statements. An estimator, confidence set, forecast or test specifies a sampling law, model class and loss/coverage criterion. Uniform $(1-\alpha)$-coverage means $\inf_{m\in\mathcal M}\Pr_m\{\tau(m)\in C_n\}\geq1-\alpha$, or an explicitly asymptotic version. The whole parameter space trivially has coverage, so informativeness requires a stated width, risk or power objective. A forecasting improvement is relative to a named benchmark, information set and evaluation population.

Equilibrium and optimization. An equilibrium comprises feasible plans, optimality, beliefs where needed, budget constraints and all market/resource clearing. The primitives or a stated benchmark define each correspondence $\mathcal E(p;m)$. Nonemptiness is assumed or proved, never inferred just from compact policy sets. A selected-equilibrium target uses a declared selection rule; a robust target quantifies over all permitted equilibria. Use suprema or infima unless attainment is established. An $\varepsilon$-optimal policy is feasible and within $\varepsilon$ of the finite optimum value. Report an empty feasible set separately.

Welfare and accounting. Normative utility, interpersonal normalization, social weights, numeraire, discounting and the use of public receipts are primitives. Behavioral utility need not coincide with the welfare criterion. Household welfare includes ownership income and rents once; adding profits again to compensating variation generally double-counts them. A monetary equivalent is defined by an explicit transfer and price convention, with monotonicity and range conditions. Average effects, mechanism contributions, transfers and welfare losses are different targets. Infinite sums require convergence and dynamic feasible plans require terminal or no-Ponzi conditions.

Source versions. A working-paper date, revision date and journal publication year are distinguished. A DOI for a journal version is not automatically the DOI of an earlier manuscript. Locators refer to the stated version and printed pages where specified. A metadata-only check does not verify a claim about a full-text conclusion. Every entry's source subsection narrows the provenance claim accordingly.

Common conventions: Additional-source status and mathematical conventions

Of the 100 supplied ECON entries, 65 distinct problems appear here; 32 close matches refine earlier entries, and three duplicate questions map to existing entries. Appendix app:audited-crosswalk lists every destination. The attachment's P/R/S/U distinctions and source audits are retained. Its precise mathematical formalizations are editorial unless the individual audit says otherwise. Candidate subsidy targets ECON-004--006 retain their U flags; the resolved approval-core question ECON-007 updates OP-0202 above. The following supplied conventions govern both this part and the attributed ECON refinements elsewhere in the catalogue, subject to local restrictions.

Parameterized questions. A problem family lists inputs that must be fixed before an instance is evaluated: population, horizon, policies, information, data law, model class and welfare weights. These are required choices, not quantities the citations are assumed to specify. Undefined applications should not be treated as identified estimands.

Indices and integrability. Sets of agents, firms and regions are finite unless a population measure is explicitly used. $i,j$ index units, $r$ regions, $t$ dates and $h$ horizons. $\mathbb E$ and $\Pr$ refer to the declared population and law; $\mathbf1\{A\}$ is an indicator. Every displayed expectation and discounted sum needs existence in the stated domain. Infinite horizons use a discount in $(0,1)$ and a convergence condition; finite horizons may use discount one.

Optimization and existence. A displayed $\max$, $\min$, or $\arg\max$ is conditional on attainment. For finite feasible menus this is immediate; general spaces need continuity and compactness/coercivity or another existence argument. Without attainment, the target is the supremum/infimum and arbitrarily near-optimal feasible choices. $\inf\varnothing=+\infty$. Empty feasibility and an unidentified target are different issues.

Potential outcomes. $Y_i(z)$ is the outcome under a specified intervention, not the observed conditional mean among people choosing $z$. In binary comparisons $\Delta Y_i=Y_i(1)-Y_i(0)$. Specify assignment, treatment versions, compliance, information and observation horizon. Interference is allowed under a full policy vector; compressed exposure notation needs a justified mapping. No interference, consistency and overlap are substantive assumptions, not automatic consequences of notation.

Populations and observation. Fix baseline eligibility and population weights. Conditioning on post-policy employment, survival, relocation or program uptake can change the population being compared. Death is not ordinary loss to follow-up; outcomes truncated by death need a composite convention, joint survival target, or explicitly defined survivor stratum. Fertility-changing interventions need a population functional rather than an unexplained fixed descendant cohort. Measurement and missingness models must be supplied.

Identification. A structural model $m\in\mathcal M$ implies observable law $P_m$ and target $\theta(m)$. Identification means \[ P_{m_1}=P_{m_2}\ \Longrightarrow\ \theta(m_1)=\theta(m_2) \quad\text{for all }m_1,m_2\in\mathcal M . \] Without identification, the target is \[ \Theta(P;\mathcal M)=\{\theta(m):m\in\mathcal M,\ P_m=P\}. \] Writing $\theta(P)$ before establishing identification can hide incompatible latent models. Confidence sets additionally need a sampling law, confidence level, asymptotic sequence and uniformity class. Point identification, coverage and informative precision are separate properties.

Mediation. Total effects of randomized assignment are distinct from cross-world natural indirect effects. Belief/effort-channel effects require a meaningful mediator intervention and additional measurement, exclusion or mediation assumptions. Randomization of the initial policy alone does not supply those assumptions. Report bounds or interventional alternatives when the stronger target is unsupported.

Equilibrium. A counterfactual $W(z)$, $p(z)$, or $\mu_t^z$ requires individual optimization, feasibility, government/resource budgets, market clearing and state-transition laws. State equilibrium existence and selection, or report ranges over compatible equilibria. Initial-state integration includes subsequent endogenous transitions; current-state integration must use the policy-dependent distribution. Reduced-form invariance after policy is not assumed.

Welfare accounts. Scores, utility, health, dollars and ecological quantities require explicit conversion before addition. Fees, taxes, subsidies, wages and extortion payments are transfers between parties; distributional weights can make them welfare relevant, but their face value is not automatically a resource loss. State ownership, geographic boundaries, foreign-income weights and fiscal finance. Do not add profits to household welfare already including dividends, or count land capitalization and the underlying benefit twice. A benefit-cost expression must distinguish gross benefits from net welfare and subtract every real cost exactly once.

Derivatives and risk. Log derivatives require positive arguments; local responses require admissible perturbations and specified equilibrium branches. Differentiation under expectations needs regularity/domination, and boundaries may allow only one-sided derivatives. Risk uses a specified law; ambiguity uses a declared nonempty law set on a common history space. Adaptive policies must be nonanticipative. A robust criterion is an editorial choice unless attributed explicitly.

Scope overlaps. Allocation, collective choice, contracts, household bargaining and coordinated policy overlap economics with optimization and game theory. They are retained because they appeared in the original catalogue; no claim of a strictly game-theory-free selection is made.

Statement licence: CC-BY-4.0 · Source file

Contributions 1

Econ-595-A1 · Exploration · LLM-generated · gpt 6 astra ultra ·

GPT 6 Astra Ultra attempt on Econ-595

Long-run AI tutoring and independent learning: clustered durability evidence and governance certificates

Target and scope

For a baseline pupil cohort, access to ordinary materials, a general assistant, or a constrained tutor is assigned at classroom level. The principal contrasts are \[ \tau_Y(a)=E[Y(a)-Y(0)],\quad \tau_R(a)=E[R(a)-R(0)],\quad \tau_T(a)=E[T(a)-T(0)],\qquad a=1,2. \] The examination is assistant free and occurs six months after access ends. The existing experiment [ai] measures short-term assisted and subsequent unassisted performance, and explicitly identifies longer-term outcomes as a future direction. This note gives a classroom-level finite-sample design and a decision certificate for a finite governance menu. It does not extrapolate immediate performance into six-month learning.

Inference at the unit actually randomized

Fix $C$ classrooms and baseline pupil weights $w_c\ge0$, $\sum_cw_c=1$. For pupil-weighted outcomes, $w_c$ is the classroom's baseline enrollment share. Every classroom has a potential classroom average $Z_c(a)\in[0,1]$, where $Z$ is a fixed rescaling of an examination, reasoning, dependence or time outcome. Bounded time requires a prespecified range; it is not inferred from the score bound.

Independently assign classrooms to arms with known probabilities $p_{ca}>0$. Assume no cross-classroom interference, consistency of the versioned deployment, and complete baseline-cohort measurement under the declared transfer and absence rules. Dependence among pupils within a classroom is unrestricted.

Theorem (Simultaneous bounded-outcome inference): For an active arm $a$, define \[ \widehat\tau_Z(a)=\sum_cw_c\left\{ \frac{1\{A_c=a\}Z_c}{p_{ca}}- \frac{1\{A_c=0\}Z_c}{p_{c0}}\right\}. \] It is unbiased for $\sum_cw_c[Z_c(a)-Z_c(0)]$. For a prespecified total of $M$ outcome-arm contrasts, simultaneous coverage at least $1-\alpha$ holds using each radius \[ r_a=\sqrt{\frac12\log(2M/\alpha) \sum_cw_c^2\left(\frac1{p_{ca}}+\frac1{p_{c0}}\right)^2}. \]
Proof: Condition on all classroom potential outcomes. Inverse assignment weighting gives each arm mean by direct expectation. Each classroom's contrast contribution lies between $-w_c/p_{c0}$ and $w_c/p_{ca}$, an interval of the length displayed inside the squared sum. Independent classroom assignment permits Hoeffding's inequality for this weighted sum. The chosen radius makes the two-sided failure probability at most $\alpha/M$; a union bound over the predetermined comparisons gives simultaneous coverage. No pupil-level independence enters the proof. ∎
For balanced probabilities this radius is of order $\sqrt{\log(M/\alpha)\sum_cw_c^2}$. Thus a large number of pupils in a few classrooms does not automatically yield precise causal comparisons. In the admissible submodel where every pupil in a classroom shares one random classroom outcome, increasing classroom size supplies no additional independent information. With only one randomized classroom, its untreated potential outcome is never simultaneously observed, however many pupils it contains.

If the actual design balances classroom covariates using dependent assignment, the independent-assignment proof must be replaced by an appropriate randomization analysis. The theorem proposes a clear sufficient design rather than claiming that every existing classroom experiment used it.

A finite governance decision

Let $g\in\{0,1,\ldots,G\}$ index a finite, prespecified menu of tutor versions and oversight rules, with $g=0$ the comparator. Define delayed unaided learning $L$, dependence $D$, supplied real cost increment $c_g$, and declared nonnegative conversion weights $p_L,p_D$. The welfare criterion is \[ W_g=p_LE[L(g)-L(0)]-p_DE[D(g)-D(0)]-c_g. \] Privacy and equity can instead be constraints or additional reported outcomes; their valuation is not determined by examination scores.
Proposition (A simultaneous governance certificate): Suppose simultaneous outcome intervals imply $|\widehat W_g-W_g|\le b_g$ for every $g$, on an event of probability at least $1-\alpha$. Choose $\widehat g$ to maximize $\widehat W_g-b_g$, including comparator $W_0=0,b_0=0$. Then on that event \[ W_{\widehat g}\ge\widehat W_{\widehat g}-b_{\widehat g}\ge0, \qquad \max_gW_g-W_{\widehat g}\le2\max_g b_g. \] One may take $b_g=p_Lr_{Lg}+p_Dr_{Dg}$, plus a valid cost-estimation allowance when costs are not supplied.
Proof: Every lower score is at most its true welfare and at least its true welfare minus $2b_g$. The comparator's lower score is zero, proving nonnegative certified welfare of the maximizer. Insert a true welfare-maximizing menu item into the empirical lower-score comparison to obtain the regret bound. The formula for $b_g$ is the triangle inequality applied to the two outcome errors. Because coverage is simultaneous, data-dependent choice among these predeclared items does not invalidate the certificate. ∎
This result concerns the declared conversions and cost account. Without credible weights, the simultaneous vector of outcome intervals can still reject dominated options, but cannot order every learning--dependence tradeoff.

Durability is a separate identification problem

Let $S(a)$ denote an immediate assistant-free assessment and $Y(a)$ the six-month score, both in $[0,1]$.
Proposition (Early performance does not identify persistence): If the observed experiment stops after measuring $S$, then, without a temporal restriction, the sharp interval for each active-arm six-month effect $E[Y(a)-Y(0)]$ is $[-1,1]$, regardless of the identified immediate assessment effects.
Proof: Hold all baseline variables, assignments, usage and immediate outcomes fixed. In one continuation set $Y(a)=1,Y(0)=0$; in another set $Y(a)=0,Y(0)=1$. These are bounded, well-defined future potential outcomes with the same complete observed law up to the early assessment. They attain the endpoints, and independent mixtures attain every intermediate effect. Immediate assisted scores or fine-grained usage records do not change this argument unless they restrict the future response law. ∎
For comparison, suppose a scientifically justified mean-level persistence restriction states \[ \tau_Y(a)=\rho\tau_S(a)+d_a,\qquad \rho\in[\rho_L,\rho_U],\quad |d_a|\le\epsilon_a. \] For one arm the induced outer interval is $[\min\{\rho_L\tau_S,\rho_U\tau_S\}-\epsilon_a, \max\{\rho_L\tau_S,\rho_U\tau_S\}+\epsilon_a]\cap[-1,1]$. It is sharp for the declared effect-level class whenever its values can be realized by bounded future means, which is automatic for a single contrast in $[-1,1]$. A common $\rho$ couples multiple arms, so separate coordinate intervals need not describe their joint set. The persistence range and deviation allowance require actual follow-up validation or substantive assumptions; they cannot be learned from immediate outcomes alone.

Remaining gap

The note establishes finite-sample uncertainty and finite-menu decision guarantees under an explicit classroom design. It does not estimate six-month effects, justify a persistence bridge, resolve cross-classroom sharing or assess future tutor versions. Missing assessments, teacher adjustments and pupil transfers require their own observation and exposure models. The proposed long-run learning and governance targets therefore remain unresolved empirically and in the unrestricted model class.

Completion Estimate

46%

This is a post hoc, heuristic estimate for the full catalogue target. The attempt gives classroom-level simultaneous inference and finite-governance welfare certificates, and proves immediate assessments cannot identify six-month learning without a persistence restriction. The original delayed learning, reasoning, time and dependence effects remain unmeasured, with unresolved missingness, teacher responses, peer sharing, persistence validation and governance costs.

References

  • [ai] H. Bastani, O. Bastani, A. Sungu, H. Ge, O. Kabakci and R. Mariman (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. PNAS 122(26), e2422633122. Experimental Design and Discussion; publisher text. https://doi.org/10.1073/pnas.2422633122.

Go to contribution →