Causal-Attention Retrieval Motivation

Research-attempt provenance

Definitions and mathematical statement

Shared notation.

Standard number systems are $\mathbb N,\mathbb Z,\mathbb Q,\mathbb R,\mathbb C$. Specialized notation and index ranges follow the cited source and the definitions below.

Causal attention permits a token to depend only on its predecessors under the stated order. Angles on $\mathbb T=\mathbb R/(2\pi\mathbb Z)$ represent circular states. Retrieval requires a specified decoding map linking a later token state to an earlier input token.

Record scope and qualifications.

The catalogue title is retained as metadata. Named certificates, contracts, bounds or frameworks in that title are not asserted as results of the original source extract. The supplied registry flags this extraction as unrecoverable. It is displayed as a literal source fragment; the definitions provide context rather than a replacement formal statement.

Its differential equation and decoding convention are incomplete or damaged; the full transformer task is not replaced by a reconstructed simplified theorem.

Source-record limitation: Source flags the statement as unrecoverable: The official report uses this question to define its retrieval observable and then derives an asymptotic formula and a rigorous U-shaped retrieval theorem for the stated model; it is not left as a standalone open problem. No separate clean statement is promoted; the original record remains publicly visible and retained for provenance. The source text below is rendered literally where necessary; damaged formulas and composite extracts are not silently repaired into a new mathematical claim. The supplied catalogue marks this record as unrecoverable. Its extract and terminology are documented, but a complete original problem has not been reconstructed.

Statement recorded in the supplied source.

For a distinguished site i∗ = ⌊

\textbackslash{}

sigma0 N ⌋, with

\textbackslash{}

sigma0

\textbackslash{}

in $0, 1$, we ask whether the last token

\textbackslash{}

thetaN (t) at time t decodes to the initial token at position i∗ . Large language models based on causal transformers are routinely deployed in regimes where the available context is much longer than the part of the prompt that actually matters for the final prediction. A striking empirical observation is the so-called lost-in-the-middle effect: retrieval is typically best when the rel- evant information is placed near the beginning or the end of the context, and substantially worse when it is placed in the middle [6]. The aim of this work is to give a rigorous mathematical explanation of this phenomenon in a minimal continuous-time model for causal self-attention with recency bias. Our starting point is the interacting particle system viewpoint on transformers developed in [3, 4, 5]. We consider a decoder-type dynamics on the circle T = R/2

\textbackslash{}

piZ, where the jth token is represented by an angle

\textbackslash{}

thetaj (t) and only attends to its predecessors. After reducing the original spherical model to dimension 2, and replacing the random softmax denominator by a deterministic normalization, we arrive at j X

\textbackslash{}

lambda e− N (j−k)

\textbackslash{}

thetȧj (t) = Pj

\textbackslash{}

lambda W

\textbackslash{}

beta′ (

\textbackslash{}

thetaj (t) −

\textbackslash{}

thetak (t)), W

\textbackslash{}

beta (x) = e

\textbackslash{}

beta cos x . e −N (j−ℓ) k=1 ℓ=1 [S2]

Source status.

The supplied catalogue labels this source record partially solved; that metadata was checked against the supplied snapshot on 2026-10-11. The label is preserved for provenance and is not an independent verification of present open status or mathematical correctness. The catalogue literature assessment is dated 2026-08-22.

Short English statement

The fragment motivates middle-position retrieval in a continuous attention model.

Sources

• S1 Ulam AI and the UnsolvedMath Contributors, UnsolvedMath: A Curated Collection of Open Mathematics Problems, record 30006641 (OWR-14299914-002). Catalogue curation and metadata: CC BY 4.0. The source extract is retained; definitions, English summaries and qualifications are editorial additions. https://huggingface.co/datasets/ulamai/UnsolvedMath.
• S2 Flows on Measure Spaces and Applications in Machine Learning $2026$, Oberwolfach Reports, DOI 10.4171/owr/2026/13, pp. 795--884 https://doi.org/10.4171/owr/2026/13.
• S3 Duerinckx--Geshkovski--Rossi, Kinetic theory for Transformers https://arxiv.org/abs/2605.09213.
• S4 Liu et al., Lost in the Middle https://doi.org/10.1162/tacl_a_00638.
• Source-specific Oberwolfach Reports licences, including CC BY-SA 4.0 where applicable, are retained; see THIRD_PARTY_NOTICES.md.

Preparation and review.

The original source and UnsolvedMath contributors retain attribution for the source extract. Codex (GPT-6; exact serving revision not exposed) assisted with definitions, English summaries, scope notes and formatting. No new human mathematical authorship or independently verified solution is asserted. Permanent source ID 30006641 and code OWR-14299914-002 checked against the complete supplied registry and current repository. Distinct upstream records, including repeated questions and broad or damaged extracts, are retained for complete coverage of the source CS category. A repeated record is not a new independent conjecture. Complete mathematical-equivalence and current-literature review has not been performed. Preparation and final review follow contribution prompts C and D, with source gaps disclosed. Structural and rendering checks do not establish a new theorem or open status.

Literature and context check

The primary report, printed pp. 841--843, supplies the undamaged finite-particle dynamics, a decoding observable, correlation asymptotics, and a theorem giving a U-shaped profile under stated conditions. Thus the retained damaged catalogue fragment is motivation within a reported result, not an independently posed unresolved theorem. The finite-$N$ calculation below reuses that explicit dynamics for an auxiliary observable; it does not claim a new proof of the report's retrieval theorem or a result about full transformers.

Research attempt: Finite-time causal sensitivity and the first correlation term

Fix $N\geq2$, $\beta>0$, and $\lambda\in\mathbb R$. For $k\leq j$, set \[ a_{jk}=\frac{e^{-\lambda(j-k)/N}}{\sum_{\ell=1}^j e^{-\lambda(j-\ell)/N}}, \qquad W(u)=e^{\beta\cos u}. \] Consider the smooth ODE on the $N$-torus \[ \dot\theta_j=\sum_{k\leq j}a_{jk}W'(\theta_j-\theta_k). \] The $k=j$ term is identically zero. Bounded smooth derivatives on the compact torus give existence and uniqueness for every finite time: local ODE solutions cannot escape the compact state space, and a uniform local continuation interval extends them successively. Its triangular structure implies that the first $j$ coordinates depend only on their own initial values, by uniqueness of their closed subsystem. Thus $\partial\theta_j(t)/\partial\theta_i(0)=0$ when $i>j$.

For $i<j$, the off-diagonal Jacobian entry at time zero is \[ J_{ji}(0)=-a_{ji}W''(\theta_j(0)-\theta_i(0)), \] so the direct influence is $tJ_{ji}(0)+O(t^2)$ for fixed data and fixed $N$. Here $|W''|\leq e^\beta(\beta+\beta^2)$, since $W''=(-\beta\cos u+\beta^2\sin^2u)e^{\beta\cos u}$. This gives a finite bound for the differentiations; no limit uniform in $N$ or long time is inferred.

Assume now the initial angles are independent and uniform. Rotation equivariance of the ODE makes every individual angle uniform at every time, so its cosine has zero mean. For fixed $i<N$, define \[ C_i(t)=\mathbb E[\cos\theta_N(t)\cos\theta_i(0)]. \] Initially $C_i(0)=0$. Bounded derivatives justify differentiating under expectation. In the derivative, every term with $k\ne i,N$ has a factor $\cos\theta_i(0)$ independent of the remaining variables and hence mean zero; the self term $k=N$ is zero. With independent uniform $U,V$ this leaves \[ C_i'(0)=a_{Ni}\,\mathbb E[-\sin U\,W'(U-V)\cos V]. \] Put $c_1=(2\pi)^{-1}\int_0^{2\pi}e^{\beta\cos v}\cos v\,dv$. Evenness and a change of variables give $\mathbb E_V[W(U-V)\cos V]=c_1\cos U$, and differentiation in $U$ gives $\mathbb E_V[W'(U-V)\cos V]=-c_1\sin U$. Therefore \[ C_i'(0)=\frac12a_{Ni}c_1>0. \] For completeness, $c_1>0$ follows by pairing the regions with opposite cosine: on $0<v<\pi/2$ their combined integrand is $\cos v(e^{\beta\cos v}-e^{-\beta\cos v})>0$, and symmetry covers the circle.

When $\lambda>0$, $a_{Ni}$ strictly increases with $i$, so the first nonzero short-time term in this cosine correlation is monotone in recency. It does not produce a U-shape. For fixed $N$, bounded smoothness gives $C_i(t)=t a_{Ni}c_1/2+O(t^2)$. Consequently a proposed derivation of the early-position advantage from this direct first-order coupling alone has a precise gap: repeated causal propagation and higher-order terms must enter. This does not contradict the report's asymptotic and macroscopic-time theorem; the observable, scaling, and regime differ.

Verification and adversarial audit

The diagonal self-interaction is identically zero and is not differentiated as if its two arguments were independent. Uniform marginals follow from simultaneous rotation symmetry, not independence at positive time. Only independence at time zero was used to remove terms. The cosine correlation is explicitly distinguished from vocabulary-decoding accuracy and its smoothed version.

Audit corrections.

Used the restored primary equation only in an explicitly separate calculation; distinguished fixed-$N$ short-time recency from the source's U-shaped asymptotic profile.

External computation.

None; symbolic arguments and boundary checks only.

Scope and remaining gaps

The damaged record remains a source-motivation fragment with a reported result. No independent complete reconstruction of its retrieval theorem, uniform-$N$ expansion, decoding guarantee, or full-transformer extension is established.

Integration review.

Codex (GPT-6; exact serving revision not exposed) reviewed the written arguments and expanded layout and source-reference wrappers. Repository metadata and the production web renderer were checked using Python 3.13 and JavaScript with MathJax 3.2.2. This model-assisted review is not independent human proof verification; no PDF compilation or formal proof checking is asserted.

Completion Estimate

COMPLETION: 0% The calculations below the preserved source statement investigate a concrete route, but establish no substantiated new progress toward the entire original target. Known special cases and failed sufficient conditions are not credited as a solution.

References

The original source references and catalogue attribution are preserved in the Sources section above. The following URLs were actually consulted for this attempt; retention of an additional original reference does not claim fresh consultation of it.

• https://ems.press/content/serial-article-files/53606 (consulted 2026-10-11).

Model and contributor disclosure

Codex GPT-6 completed written-proof review of this attempt at its declared scope; this is model-assisted review, not independent human proof checking. OpenAI gpt-6-astra, requested reasoning configuration ultra, generated this research attempt on 2026-10-11. The exact serving revision is not exposed. Codex GPT-6 performed orchestration and repository-format review; its exact serving revision is also not exposed. The mathematical route, boundary checks and final adversarial self-audit followed contribution prompts A then B in docs/contribution_prompts.md, after reading CONTRIBUTING.md. Python 3.14.2 assembled the standalone documents; any mathematical computation is specified in Verification. No theorem prover was used. No human mathematical authorship, substantial human proof correction, or independent human proof checking is asserted. Model-assisted review, source retrieval, and structural checks do not establish correctness or novelty. Full private reasoning transcripts are unavailable as reproducibility artifacts; the complete arguments, inputs and outcomes actually used are stated here or in the companion batch audit.

Final status

Unresolved as a new research contribution. This is a concrete route investigation, not a claim of new complete resolution or a current-status certification. A known answer reproduced for context does not become a new contribution. The catalogue's source-status label is preserved; this attempt status does not assert that a source-labelled solved question is open.