gpt 6 astra ultra attempt on CS-790

Research-attempt provenance

Definitions and mathematical statement

Shared notation.

Standard number systems are $\mathbb N,\mathbb Z,\mathbb Q,\mathbb R,\mathbb C$. Specialized notation and index ranges follow the cited source and the definitions below.

Adam uses running estimates of gradient first and second moments to rescale parameter updates. Gradient descent uses the gradient directly with a step size. Class imbalance means that some output labels occur much more frequently than others.

Record scope and qualifications.

The catalogue title is retained as metadata. Named certificates, contracts, bounds or frameworks in that title are not asserted as results of the original source extract. The supplied registry flags this extraction as unrecoverable. It is displayed as a literal source fragment; the definitions provide context rather than a replacement formal statement.

The excerpt mixes an empirical motivation with incomplete reported results; it does not establish a universal advantage or a precise outstanding theorem.

Source-record limitation: Source flags the statement as unrecoverable: The official report uses this question as the title of a talk and then supplies its proposed mechanism: heavy-tailed class imbalance creates ill-conditioning that Adam’s normalization counteracts. The source text below is rendered literally where necessary; damaged formulas and composite extracts are not silently repaired into a new mathematical claim. The supplied catalogue marks this record as unrecoverable. Its extract and terminology are documented, but a complete original problem has not been reconstructed.

Statement recorded in the supplied source.

Why does Adam work so well? Adam has been shown to outperform gradient descent in optimizing large language transformers empirically, and by a larger margin than on other tasks, but it is unclear why this happens. We show that the heavy-tailed class imbalance found in language modeling tasks leads to difficulties in the optimization dynamics. When training with gradient descent, the loss associated with infrequent words decreases slower than the loss associated with frequent ones. As most samples come from relatively infrequent words, the average loss decreases slowly with gradient descent. On the other hand, Adam and sign-based methods do not suffer from this problem and improve predictions on all classes. To establish that this behavior is indeed caused by class imbalance, we show empirically that it persist through different architectures and data types, on language transformers, vision CNNs, and linear models. We study this phenomenon on a linear classification with cross-entropy loss, showing that heavy-tailed class imbalance leads to ill-conditioning, and that the normalization used by Adam can counteract it. [S2]

Source status.

The supplied catalogue labels this source record partially solved; that metadata was checked against the supplied snapshot on 2026-10-11. The label is preserved for provenance and is not an independent verification of present open status or mathematical correctness. The catalogue literature assessment is dated 2026-08-22.

Short English statement

The fragment asks why Adam performs well and describes a proposed class-imbalance explanation.

Sources

• S1 Ulam AI and the UnsolvedMath Contributors, UnsolvedMath: A Curated Collection of Open Mathematics Problems, record 30006177 (OWR-14299083-003). Catalogue curation and metadata: CC BY 4.0. The source extract is retained; definitions, English summaries and qualifications are editorial additions. https://huggingface.co/datasets/ulamai/UnsolvedMath.
• S2 Overparametrization, Regularization, Identifiability and Uncertainty in Machine Learning $2025$, Oberwolfach Reports, DOI 10.4171/owr/2025/4, pp. 175--208 https://doi.org/10.4171/owr/2025/4.
• S3 Oberwolfach Report 4/2025 https://content.ems.press/assets/public/full-texts/serials/owr/22/1/14299083/online/10.4171-owr-2025-4.pdf.
• S4 Kunstner et al., Heavy-tailed class imbalance and why Adam outperforms gradient descent https://proceedings.neurips.cc/paper_files/paper/2024/hash/350e718ff74062b4bac2c6ffd9e1ac66-Abstract-Conference.html.
• Source-specific Oberwolfach Reports licences, including CC BY-SA 4.0 where applicable, are retained; see THIRD_PARTY_NOTICES.md.

Preparation and review.

The original source and UnsolvedMath contributors retain attribution for the source extract. Codex (GPT-6; exact serving revision not exposed) assisted with definitions, English summaries, scope notes and formatting. No new human mathematical authorship or independently verified solution is asserted. Permanent source ID 30006177 and code OWR-14299083-003 checked against the complete supplied registry and current repository. Distinct upstream records, including repeated questions and broad or damaged extracts, are retained for complete coverage of the source CS category. A repeated record is not a new independent conjecture. Complete mathematical-equivalence and current-literature review has not been performed. Preparation and final review follow contribution prompts C and D, with source gaps disclosed. Structural and rendering checks do not establish a new theorem or open status.

Literature and context check

The primary workshop report, printed p. 198, discusses heavy-tailed class imbalance as one explanation for Adam's empirical behavior. The linked NeurIPS 2024 abstract was also consulted: it reports experiments and continuous-time classification results. Its proof and experiments were not independently reconstructed. The separable quadratic and observation experiment below are different, explicitly limited models, not a solution to the complete empirical question.

Research attempt: Coordinate normalization and the rare-sample barrier

Consider a full-batch loss \[ F(w)=\frac12\sum_{i=1}^m p_i(w_i-a_i)^2,\qquad p_i>0. \] For gradient descent with constant step $\eta$, the error $e_{t,i}=w_{t,i}-a_i$ satisfies $e_{t+1,i}=(1-\eta p_i)e_{t,i}$. Thus when $0<\eta p_i<1$, the time to reduce its magnitude by a factor $\rho\in(0,1)$ is at least $\log\rho/\log(1-\eta p_i)$, which diverges as $p_i\downarrow0$. A step stable in every coordinate cannot simply be increased without regard to the largest $p_i$: if $\eta p_j>2$, nonzero error in coordinate $j$ grows geometrically.

Now consider bias-corrected Adam without weight decay, with $m_0=v_0=0$, $0\leq\beta_1,\beta_2<1$, and positive learning rates. For this full-batch separable loss, take denominator $\sqrt{\widehat v_{t,i}}$ with zero additive stabilization parameter. Compare runs for positive weights $p_i$ and for all weights equal to one, from the same initialization. Whenever updates are defined, induction gives identical $w$ in the two runs and \[ m_{t,i}=p_i m^{(1)}_{t,i},\qquad v_{t,i}=p_i^2v^{(1)}_{t,i}. \] Indeed, at an identical iterate $g_{t,i}=p_i e_{t,i}$, the moving-average recursions preserve these identities; bias corrections are independent of $p_i$; their ratio cancels $p_i$. At an identically zero coordinate gradient history set the zero-over-zero update to zero in both runs. This proves exact coordinate-scale invariance in this particular deterministic model.

The usual positive stabilization parameter $\epsilon$ breaks exact cancellation. Already the first update is \[ -\eta\frac{p_i e_{0,i}}{p_i|e_{0,i}|+\epsilon}. \] For fixed nonzero error and $\epsilon>0$ it tends to zero as $p_i\downarrow0$. Nor does normalization imply convergence with a fixed step: for sign descent on the scalar quadratic and $0<e_0<\eta$, the errors alternate between $e_0$ and $e_0-\eta$, never tending to zero. This is also the Adam update with $\beta_1=\beta_2=0$, $\epsilon=0$.

A sampling obstruction.

Suppose a rare category occurs independently with probability $p$ on each training draw, its unknown target bit is equally likely 0 or 1, and all nonrare examples and side information are independent of that bit. After $n$ draws the category is unobserved with probability $(1-p)^n$. Conditional on that event every learner, normalized or otherwise, has error $1/2$ when tested on the rare category. Therefore its average conditional rare-category risk is at least $\tfrac12(1-p)^n$, and one fixed target bit attains that lower bound. Normalization can cancel deterministic curvature weights in the preceding model but cannot manufacture absent observations. Applying the scale-invariance calculation directly to stochastic rare-token arrival would therefore be an invalid inference.

Verification and adversarial audit

The weighted and unweighted Adam trajectories are compared inductively, not by comparing gradients at unrelated iterates. Zero denominators receive an explicit convention only in the auxiliary zero-$\epsilon$ model. The sign-descent cycle checks that invariant speed is different from convergence. The stochastic lower bound is conditional category risk; overall population risk multiplies it by $p$.

Audit corrections.

Separated the zero-stabilizer full-batch invariance from practical positive-$\epsilon$ behavior and from the unavoidable rare-observation bottleneck.

External computation.

None; symbolic arguments and boundary checks only.

Scope and remaining gaps

The model omits cross-coordinate gradient correlations, nonseparable cross-entropy, minibatch noise, momentum interactions between classes, and neural-network representation changes. It proves neither a general Adam advantage nor the full empirical explanation requested by the source.

Integration review.

Codex (GPT-6; exact serving revision not exposed) reviewed the written arguments and expanded layout and source-reference wrappers. Repository metadata and the production web renderer were checked using Python 3.13 and JavaScript with MathJax 3.2.2. This model-assisted review is not independent human proof verification; no PDF compilation or formal proof checking is asserted.

Completion Estimate

COMPLETION: 0% The calculations below the preserved source statement investigate a concrete route, but establish no substantiated new progress toward the entire original target. Known special cases and failed sufficient conditions are not credited as a solution.

References

The original source references and catalogue attribution are preserved in the Sources section above. The following URLs were actually consulted for this attempt; retention of an additional original reference does not claim fresh consultation of it.

• https://ems.press/content/serial-article-files/51348 (consulted 2026-10-11).
• https://proceedings.neurips.cc/paper_files/paper/2024/hash/350e718ff74062b4bac2c6ffd9e1ac66-Abstract-Conference.html (consulted 2026-10-11).

Model and contributor disclosure

Codex GPT-6 completed written-proof review of this attempt at its declared scope; this is model-assisted review, not independent human proof checking. OpenAI gpt-6-astra, requested reasoning configuration ultra, generated this research attempt on 2026-10-11. The exact serving revision is not exposed. Codex GPT-6 performed orchestration and repository-format review; its exact serving revision is also not exposed. The mathematical route, boundary checks and final adversarial self-audit followed contribution prompts A then B in docs/contribution_prompts.md, after reading CONTRIBUTING.md. Python 3.14.2 assembled the standalone documents; any mathematical computation is specified in Verification. No theorem prover was used. No human mathematical authorship, substantial human proof correction, or independent human proof checking is asserted. Model-assisted review, source retrieval, and structural checks do not establish correctness or novelty. Full private reasoning transcripts are unavailable as reproducibility artifacts; the complete arguments, inputs and outcomes actually used are stated here or in the companion batch audit.

Final status

Unresolved as a new research contribution. This is a concrete route investigation, not a claim of new complete resolution or a current-status certification. A known answer reproduced for context does not become a new contribution. The catalogue's source-status label is preserved; this attempt status does not assert that a source-labelled solved question is open.