Invariant-Sphere Recurrent State
Source paper: Invariant Sphere Theorem and Ring-Coupled Systems arXiv:2608.28223 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace an unconstrained recurrent transition by a ring-coupled cubic vector field whose radial component drives hidden states toward a prescribed sphere. The angular component remains trainable and can encode information, while the radial Lyapunov dynamics suppress exploding and vanishing state norms during long rollouts.
Formulas
Mathematical statement
Let $h\in\mathbb{R}^{n}$ be the recurrent state, $r=\lVert h\rVert$, and define the ring-coupled homogeneous cubic field $Q(h)$ by $Q_i(h)=q(h_i,h_{i+1})$, with cyclic indexing $h_{n+1}=h_1$ and $q(y,z)=ay^3+by^2z+cyz^2+dz^3$. The continuous recurrent dynamics are $\dot h=\lambda h+Q(h)$, where $\lambda>0$. If the cubic radial form satisfies $\langle u,Q(u)\rangle=-\kappa$ for every unit vector $u$ or approximately satisfies this with bounded variation, then $V(h)=\frac12\lVert h\rVert^2$ obeys $\dot V=\lambda r^2-\kappa r^4$, giving the attracting radius $r_*=\sqrt{\lambda/\kappa}$. More generally, if $\langle u,Q(u)\rangle\leq-\kappa_{\min}<0$ on the unit sphere, then sufficiently large states contract radially. At the pole $u=e_1$, the paper identifies a tangential eigenvalue proportional to $c+1$, so $c=-1$ is a local stability boundary that can be monitored as a trainable bifurcation parameter.
Implementation notes
1. Integration point: use this as the hidden transition of an RNN or state-space layer. Partition a hidden vector into a cyclic ring, compute $Q_i=ah_i^3+bh_i^2h_{i+1}+ch_i h_{i+1}^2+dh_{i+1}^3$, and add input injection separately. Set $\lambda>0$ and initialize coefficients so the sampled radial form is negative. Use a sufficiently small solver step, or compare against an explicit projection to radius $r_*=\sqrt{\lambda/\kappa}$. 2. Pseudocode: u=h/(norm(h)+eps); Q=ring_cubic(h); hdot=lambda*h+Q+W_in*x_t; h=h+dt*hdot; loss=task_loss. Add radial_penalty=(mean(inner)+kappa)^2+variance(inner), where inner=dot(u,Q(u)) over random unit-vector probes. Optionally parameterize $c$ around the predicted boundary $c=-1$. 3. Compute directly from the mechanism the cubic field, Lyapunov derivative, predicted radius, and pole eigenvalue. Estimate empirically the radial coefficient $\kappa=-\operatorname{mean}_{u\sim S^{n-1}}\langle u,Q(u)\rangle$, its variance, the recurrent Jacobian spectral radius, and norm relaxation rate. 4. First experiment: compare equal-width vanilla tanh RNN, orthogonal RNN, and this invariant-sphere RNN on copy-memory and adding tasks at sequence lengths 100, 500, and 1000, using equal parameter counts and optimizer budgets. Include ablations removing the radial penalty and replacing cubic ring coupling by a dense transition. 5. Pre-register the signature: with zero input, the hidden norm should converge to $r_*$ and satisfy $|r_t-r_*|\approx C\exp(-2\lambda t)$ in continuous-time units. Sweeping $c$ should change the pole-direction multiplier at $c=-1$. For Euler integration, the radial linearization predicts a step-size boundary near $dt_c\approx 1/\lambda$. Reject the transfer if the radial form is positive on more than 1 percent of sampled sphere directions, if the measured relaxation rate differs from $2\lambda$ by more than 30 percent, or if norm variance is not reduced by at least 50 percent without improving long-horizon accuracy.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Confirmed ✓
Stage 2 · Practical benchmark result: Significantly worse ✗
Stage 1 — Mechanism check agent confidence 7/10
Built a ring-coupled cubic recurrent state and verified its stability numerically. The field had zero positive radial directions across 20,000 probes, drove initial norms spanning 0.66–17.92 to approximately 2.80–2.83, and the exact scalar case converged to radius 1 with measured relaxation slope -2.31 versus the predicted -2.0. In a small delayed-bit task, the sphere RNN achieved test MSE 0.00013 versus 0.152 for the tanh baseline, a promising but preliminary signal rather than a definitive general result.
- Agent confidence
- 7/10
- Baseline
- tanh RNN: test MSE 0.151553; final hidden norm mean/std 1.16569/0.00737
- Idea
- Invariant-sphere cubic RNN: test MSE 0.000130; final hidden norm mean/std 0.59144/0.00040; radial positive fraction 0/20000; multidimensional final-radius range 2.7975–2.8275; scalar relaxation slope -2.310
Limitations: Only one seed and one short delayed-bit task were tested; no copy/addition benchmark at lengths 500–1000, equal-FLOP or equal-optimization studies, orthogonal-RNN baseline, radial-penalty ablation, dense-coupling ablation, Jacobian spectral measurements, or systematic c and step-size sweeps were performed. The multidimensional field has a direction-dependent radial coefficient rather than the ideal constant-kappa form.
How to run: python3 run_experiment.py
Stage 2 — Benchmark vs. tuned baseline
Significantly WORSE than the tuned baseline (n/a, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0
- Idea mean
- 0
- Effect (Δ)
- 0 (n/a; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- Hurts
- Baseline tuning
- swept over 3 configs
Only the built-in dynamics track was tested; no longer horizons, alternative cubic coefficients, radial-penalty ablation, dense-coupling ablation, or parameter-count-matched angular parameterization was evaluated.
How to run: python3 bench_stage2.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 10/10
The invariant-sphere recurrent transition was implemented and tested on the structurally matched dynamics track using eight paired seeds, a baseline learning-rate sweep, and a matched three-value idea sweep. The trained models showed the predicted negative radial cubic coefficient, but the idea had significantly higher test MSE than the GRU baseline, so it does not qualify as a win.
- Baseline
- GRU rnn_small, lr=0.01, 20 epochs: mean test MSE 6.485018303692414e-06
- Idea
- Invariant-sphere cubic RNN, lr=0.01, lambda=1.0, dt=0.05, cubic_scale=1.0, 20 epochs: mean test MSE 1.4861759154882748e-05
- Idea wins
- 0
- Mechanism confirmed
- true
- Paired delta mean
- 8.376740851190334e-06
- Permutation p value
- 0.0081
Benchmark result
Significantly WORSE than the tuned baseline (n/a, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0
- Idea mean
- 0
- Effect (Δ)
- 0 (n/a; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- Hurts
- Baseline tuning
- swept over 3 configs
Only the built-in dynamics track was tested; no longer horizons, alternative cubic coefficients, radial-penalty ablation, dense-coupling ablation, or parameter-count-matched angular parameterization was evaluated.
How to run: python3 bench_stage2.py
Artifacts
- bench_report.json 4.4 KB View Raw JSON
- bench_stage2.py 5.4 KB View Raw JSON
- report.md 1.5 KB View
- report_bench_2026-09-02T122613.md 3.0 KB View
- results.json 1.5 KB View Raw JSON
- run_experiment.py 4.4 KB View Raw JSON