# Unstable-Manifold-Aware Ensemble Averaging

- ID: 2697
- Canonical URL: https://synthcore.org/idea/2697/unstable-manifold-aware-ensemble-averaging
- API JSON: https://synthcore.org/api/idea/2697.json
- API Markdown: https://synthcore.org/api/idea/2697.md
- Verification status: unverified
- Source: [arXiv:2608.20017](https://arxiv.org/abs/2608.20017)
- Category: dynamics
- Solves: accuracy, stability, scalability
- ML areas: rnn, ssm, world-model, inference-speedup
- Math tags: dynamical-systems, statistics, probability, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

For a neural dynamical predictor, train or maintain several independently initialized models and aggregate their multi-step states using the signed displacement along the locally unstable forecast direction. The key mechanism is cancellation of opposite unstable-manifold errors: ordinary averaging should reduce this component at rate N^{-1/2} when errors are independent and centered, while robust aggregation should be activated when validation residuals show heavy tails or persistent bias.

## Mathematical statement

Let F_i be the i-th neural forecaster, x_t the observed state, and xhat_i(t+h)=F_i^(h)(x_t) its h-step prediction. Let u_(t,h) be a unit estimate of the most unstable local direction, obtained from the dominant right singular vector of the ensemble-mean Jacobian J_(t,h)=dF^(h)(x_t)/dx_t, or from the dominant principal component of forecast deviations. Define the signed unstable coordinate z_i=u_(t,h)^T(xhat_i(t+h)-xbar_(t+h)), where xbar_(t+h) is the preliminary ensemble mean. If z_i are independent with zero mean and variance sigma_u^2, then the unstable-coordinate error of the arithmetic ensemble mean has standard deviation sigma_u/sqrt(N), with N models. Estimate the empirical tail index or kurtosis of z_i; use the arithmetic mean when tails are light and a clipped estimator when a few models dominate. The mechanism is useful because unstable directions amplify small signed errors, so cancellation there is more valuable than uniform averaging across all state coordinates.

## Key formulas

- $\hat{x}^{(i)}_{t+h}=F_i^{(h)}(x_t),\qquad \bar{x}_{t+h}=\frac{1}{N}\sum_{i=1}^{N}\hat{x}^{(i)}_{t+h}$
- $u_{t,h}=\operatorname{argmax}_{\|u\|_2=1}\|J_{t,h}u\|_2,\qquad J_{t,h}=\frac{\partial F^{(h)}(x_t)}{\partial x_t}$
- $z_i=u_{t,h}^{\top}(\hat{x}^{(i)}_{t+h}-\bar{x}_{t+h}),\qquad \operatorname{Std}\left[u_{t,h}^{\top}(\bar{x}_{t+h}-x_{t+h})\right]\approx\frac{\sigma_u}{\sqrt{N}}$
- $\tilde{z}=\frac{1}{N}\sum_{i=1}^{N}\operatorname{clip}(z_i,-\tau,\tau)$

## Implementation notes

1. Integration point: use this as an inference-time aggregation layer for an ensemble of RNNs, neural state-space models, or reservoir computers that all receive the same history and produce h-step state forecasts. Keep model parameters independently initialized, or generate diversity through bootstrap data subsets and dropout masks. For a cheap first version, do not backpropagate through the unstable-direction estimate. 2. Pseudocode: compute every forecast y_i=F_i^(h)(x_t); set m=(1/N) sum_i y_i; form the centered forecast matrix D=[y_1-m,...,y_N-m]; take the leading left singular vector of D as u, which estimates the dominant forecast-spread direction without computing Jacobians; calculate z_i=u^T(y_i-m); estimate validation kurtosis K_h of these coordinates; if K_h is below a chosen threshold K_0, return m, otherwise clip z_i at tau=3 validation standard deviations and replace the mean's unstable coordinate by the clipped mean. 3. Computed versus estimated: forecasts and singular vectors are computed online; independence, zero bias, and the N^{-1/2} law are assumptions to test; K_h, sigma_u, and tau are estimated on held-out trajectories. 4. First experiment: train 16 small GRUs or echo-state networks on Lorenz-63 or Mackey-Glass, evaluate horizons h=1,5,10,20, and compare the best single model, ordinary mean, validation-MSE weighting, and this estimator. Plot unstable-coordinate RMSE against ensemble size N=2,4,8,16. The predicted signature is a log-log slope near -1/2 in the light-tailed regime, with flattening or outliers at horizons where kurtosis grows; robust aggregation should reduce this flattening without harming short-horizon mean error.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.