Sharp C1 invariant-manifold budget for recurrent layers
Implementation & benchmark of arXiv:2608.04862 — On the sharpness of the $C^1$-norm threshold for perturbations in the normally hyperbolic invariant manifold theorem---a toy model perspective
Source paper: On the sharpness of the $C^1$-norm threshold for perturbations in the normally hyperbolic invariant manifold theorem---a toy model perspective arXiv:2608.04862 ⓘ · analyzed Aug 31, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Construct a recurrent cell with a slow state x and an explicitly contracting auxiliary state y, then constrain the learned nonlinear perturbation in the C1 norm. Set the allowed perturbation size from the normal contraction lambda using the sharp budget (1-sqrt(lambda))^2, so the hidden dynamics retain a differentiable invariant graph and can be reduced safely to the slow coordinate.
Formulas
Mathematical statement
The paper studies F^phi_lambda,alpha(x,y) = (x + alpha_1 + lambda y + phi(x), alpha_2 + lambda y + phi(x)), where x is the tangent coordinate, y is the normal coordinate, lambda in (0,1) is the normal contraction, alpha_1 and alpha_2 are translations, and phi is a periodic C1 perturbation. The unperturbed invariant graph is Gamma_0 = R x {alpha_2/(1-lambda)}. Tangent directions remain bounded while normal directions contract proportionally to lambda^i. The persistence theorem gives a C1 invariant graph when ||phi||_C1 < (1-sqrt(lambda))^2, and the threshold is sharp because equality can produce a unique invariant graph with nondifferentiable points. In the neural adaptation, x is a slow representation, y is an auxiliary memory or error state, and phi_theta is a learned bounded residual. The useful mechanism is the quantitative relation between normal contraction and the perturbation's value and derivative, not merely small residual magnitude.
Implementation notes
1. Integration point: implement a two-channel recurrent cell or state-space block with slow state x_t and auxiliary contracting state y_t. Use x_{t+1}=x_t+a+lambda y_t+phi_theta(x_t) and y_{t+1}=b+lambda y_t+phi_theta(x_t). Parameterize lambda=sigmoid(l), and use a bounded MLP or sinusoidal residual for phi_theta.
2. Pseudocode: on every training batch, evaluate phi_theta and its derivative on current hidden states or on a dense scalar grid. Compute M0=max_j |phi_theta(x_j)| and M1=max_j |d phi_theta(x_j)/d x_j|, set M=max(M0,M1), and calculate epsilon_c=(1-sqrt(lambda))^2. Add L_C1 = softplus(M-(1-rho)epsilon_c)^2 to the task loss, where rho is a safety margin such as 0.1. A cheaper alternative rescales the final residual layer by min(1,(1-rho)epsilon_c/(M+1e-8)).
3. Analytic versus empirical quantities: epsilon_c(lambda) and the normal multiplier lambda are analytic. The global C1 norm is estimated by grid sampling, interval bounds, or spectral normalization combined with activation derivative bounds. Verify the learned graph by fitting y=h(x) from long trajectories and measuring the invariance residual h(x_{t+1})-y_{t+1}, along with finite-difference changes in h'.
4. First experiment: train a one-layer recurrent model on copy-memory and length-1000 periodic-sequence tasks. Compare an unconstrained cell, ordinary spectral normalization, and this C1-budgeted cell. Sweep lambda in {0.25,0.49,0.81}; the predicted critical perturbation norms are 0.25, 0.09, and 0.01. Scale a fixed sinusoidal residual by s and locate the smallest s at which the fitted invariant graph develops a kink or long-horizon errors become nonsmooth. The quantitative prediction is a transition near ||phi||_C1=(1-sqrt(lambda))^2, within approximately 20%; below it, graph smoothness and invariance residual remain stable, while near equality a sharp loss of differentiability appears.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a NumPy MVP for the two-state recurrent map, including exact normal-contraction checks, analytic C1-budget verification, graph-transform diagnostics, and a toy forecasting comparison. The contraction identity and epsilon_c(lambda) scaling matched exactly, but graph smoothness did not transition near the predicted C1 boundary and remained stable substantially above it in several cases. The budgeted forecaster also performed worse than the unconstrained comparison, so this experiment does not demonstrate a practical win.
- Agent confidence
- 9/10
- Baseline
- Toy forecaster MSE 0.134750; unconstrained residual amplitude 0.35. Normal contraction maximum relative error 4.07e-16.
- Idea
- Toy forecaster MSE 0.401789; effective residual amplitude 0.081 versus budget 0.09 for lambda=0.49. Predicted budgets 0.25, 0.09, and 0.01 for lambda 0.25, 0.49, and 0.81 matched exactly, but no consistent graph transition occurred near ratio ||phi||_C1/epsilon_c=1.
Limitations: This was a small NumPy toy experiment, not a trained PyTorch RNN on copy-memory or length-1000 sequence tasks. The graph-transform diagnostic is discretized and the theorem's sufficient C1 persistence bound was not tested as a universal sharp empirical bifurcation criterion. No spectral-normalization baseline, GPU run, or large-scale generalization study was performed.
How to run: python3 c1_budget_experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- c1_budget_experiment.py 4.9 KB View Raw
- report.md 1.7 KB View
- results.json 10.8 KB View Raw