# Polynomial Orbit-Pattern Regularization

- ID: 1871
- Canonical URL: https://synthcore.org/idea/1871/polynomial-orbit-pattern-regularization
- API JSON: https://synthcore.org/api/idea/1871.json
- API Markdown: https://synthcore.org/api/idea/1871.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.06103](https://arxiv.org/abs/2608.06103)
- Category: dynamics
- Solves: stability, generalization
- ML areas: rnn, ssm, world-model, regularization, training-dynamics
- Math tags: dynamical-systems, statistics, information-theory, combinatorics, control-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Add a trajectory-complexity monitor and regularizer to an RNN, SSM, or world model that limits the number of distinct hidden-state symbol patterns produced over selected time subsets. The paper's nullness criterion suggests targeting polynomial maximal pattern growth rather than merely minimizing one-step Jacobian norms, thereby suppressing combinatorial explosion of long-horizon behaviors while retaining nontrivial dynamics.

## Mathematical statement

Let $(X,T)$ be a compact dynamical system, let $\mathcal{U}=\{U_1,\ldots,U_K\}$ be a finite open cover, and let $F\subset\mathbb{Z}_+$ be a finite set of observation times. The join cover $\bigvee_{j\in F}T^{-j}\mathcal{U}$ distinguishes trajectories by their cover labels at all times in $F$. Its covering number is $N(\mathcal{U})=\min\{|\mathcal{V}|:\mathcal{V}\subseteq\mathcal{U},\ X\subseteq\bigcup_{V\in\mathcal{V}}V\}$, and the maximal pattern complexity is $p_{X,\mathcal{U}}(F)=N(\bigvee_{j\in F}T^{-j}\mathcal{U})$. The paper's mechanism is that nullness is equivalent to polynomial maximal pattern complexity: for every finite cover $\mathcal{U}$, there exist constants $C_{\mathcal{U}}>0$ and $d_{\mathcal{U}}\ge0$ such that $p_{X,\mathcal{U}}(F)\le C_{\mathcal{U}}(|F|+1)^{d_{\mathcal{U}}}$, while equicontinuity corresponds to sublinear growth. For neural trajectories $h_t\in\mathbb{R}^m$, choose quantization cells $U_1,\ldots,U_K$ and estimate the number $\widehat p(F)$ of distinct quantized label strings over initial states or inputs. Penalize excess log-complexity $[\log\widehat p(F)-\log C-d\log(|F|+1)]_+$.

## Key formulas

- $$p_{X,\mathcal{U}}(F)=N\left(\bigvee_{j\in F}T^{-j}\mathcal{U}\right),\qquad F\subset\mathbb{Z}_+,$$
- $$p_{X,\mathcal{U}}(F)\le C_{\mathcal{U}}(|F|+1)^{d_{\mathcal{U}}}\quad\Longleftrightarrow\quad \log p_{X,\mathcal{U}}(F)=O(\log |F|),$$
- $$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{task}}+\lambda\sum_{F\in\mathcal{F}}\left[\log\widehat p(F)-\log C-d\log(|F|+1)\right]_+,$$
- $$\widehat p(F)=\left|\left\{\big(q(h_j^{(b)})\big)_{j\in F}:b=1,\ldots,B\right\}\right|,$$

## Implementation notes

Integrate this at the hidden-state trajectory of an RNN, latent ODE discretization, or state-space model. Run the model for horizons up to $H$ from a batch of $B$ inputs or perturbed initial states, normalize each hidden coordinate using running quantiles, and map each normalized vector $h_t$ to a discrete cell index $q(h_t)\in\{1,\ldots,K\}$ using a fixed grid, random-hyperplane hash, or k-means centroids. For each subset size $r$, sample several time sets $F\subset\{0,\ldots,H-1\}$ with $|F|=r$, form the label string $(q(h_j^{(b)}))_{j\in F}$ for every trajectory, and count distinct strings after merging strings within a small Hamming tolerance. Use the hinge penalty in the formula, or use it only as a controller that reduces the recurrent gain, increases teacher forcing, or clips the recurrent update when the penalty is positive. The exact paper-derived quantities are the join-cover interpretation and the polynomial-versus-sublinear growth criterion; $C$, $d$, quantization scale, and finite-sample corrections must be estimated. The first cheap experiment is a vanilla GRU and gated state-space model on sequential MNIST or Copy Memory, compared with identical models without the penalty. Use $K=8$ random-hyperplane cells, $H=128$, $B=512$, and $r\in\{1,2,4,8,16,32\}$. Plot $\log\widehat p$ versus $\log(r+1)$ and fit the slope. The predicted signature is an approximately linear log-log curve for the regularized model, with slope $\widehat d$ stable as $H$ increases; an unstable unregularized model should show upward curvature and an effective slope increasing with $r$. Sweep recurrent gain or initialization scale and locate the point where fitted slope begins increasing with horizon. The regularizer should shift this transition to larger gains without materially increasing task loss.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a random-hyperplane orbit-pattern monitor and hinge-based recurrent-gain controller, with a reproducible circle-map sweep and TinyGRU comparison. The mechanism appeared clearly: the predicted neutral boundary at gain 1.0 was observed (constant count 4 at exactly 1.0), pattern growth became much larger as gain increased (full-horizon count 1163 at gain 1.6), and the controller reduced that count to 85 while reducing the mean gain to 0.649. However, the task result was not a win: baseline accuracy/loss were 0.973/0.064 versus 0.553/0.700 for the intervention, so this is evidence for complexity suppression, not task-preserving regularization.

### Mechanism check

- Verdict: Built a random-hyperplane orbit-pattern monitor and hinge-based recurrent-gain controller, with a reproducible circle-map sweep and TinyGRU comparison. The mechanism appeared clearly: the predicted neutral boundary at gain 1.0 was observed (constant count 4 at exactly 1.0), pattern growth became much larger as gain increased (full-horizon count 1163 at gain 1.6), and the controller reduced that count to 85 while reducing the mean gain to 0.649. However, the task result was not a win: baseline accuracy/loss were 0.973/0.064 versus 0.553/0.700 for the intervention, so this is evidence for complexity suppression, not task-preserving regularization.
- Confidence: 7/10
- Limitations: The toy uses a two-dimensional circle embedding, fixed random hyperplanes, finite sampled initial states, and short horizons; it does not establish the paper's compact-system theorem or generalization claims. The controller is a detached, non-differentiable intervention rather than a differentiable cardinality regularizer, and only one TinyGRU task, one seed, and one controller setting were tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.43245
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_orbit.py](https://synthcore.org/code/715/bench_orbit.py)
- [bench_report.json](https://synthcore.org/code/715/bench_report.json)
- [orbit_regularization.py](https://synthcore.org/code/715/orbit_regularization.py)
- [report.md](https://synthcore.org/code/715/report.md)
- [report_bench_2026-09-01T012018.md](https://synthcore.org/code/715/report_bench_2026-09-01T012018.md)
- [results.json](https://synthcore.org/code/715/results.json)
- [run_experiment.py](https://synthcore.org/code/715/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/715)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.