# Residual-to-Symbolic Neural Pruning

- ID: 1639
- Canonical URL: https://synthcore.org/idea/1639/residual-to-symbolic-neural-pruning
- API JSON: https://synthcore.org/api/idea/1639.json
- API Markdown: https://synthcore.org/api/idea/1639.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.00466](https://arxiv.org/abs/2608.00466)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: world-model, ssm, rnn, regularization, training-dynamics
- Math tags: dynamical-systems, statistics, linear-algebra, optimization, control-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Use a two-stage residual augmentation loop: first let a residual network explain model mismatch, then project its learned vector field onto a physically constrained candidate library and replace the flexible residual with the accepted sparse terms. This turns an unconstrained neural correction into a low-complexity dynamical law that is easier to roll out over long horizons and can expose unsupported hidden-state explanations.

## Mathematical statement

Given observed state $z_o$, input $u$, and base vector field $F_o^{obs}$, define the residual target $r(t)=\dot z_o(t)-F_o^{obs}(z_o(t),u(t),p)$. Fit a residual network $R_\theta(z_o,u)$, then project its outputs onto library features $\Phi(z_o,u)$ by solving $\min_B\|R_\theta-B\Phi\|_2^2+\lambda\|B\|_1$. For each candidate coefficient $b_j$, use an estimated standard error $\mathrm{se}(b_j)$ and admit the term only if $|b_j|/\mathrm{se}(b_j)>\tau$ after accounting for the other selected terms. The final model is $\dot z_o=F_o^{obs}+B_{\mathrm{sel}}\Phi$, optionally coupled to a latent neural state estimator. The mechanism is a bias-variance and extrapolation trade: the neural residual supplies discovery flexibility, while the sparse projected field limits rollout amplification.

## Key formulas

- $$r(t)=\dot{\bm z}_{\mathrm o}(t)-\bm F_{\mathrm o}^{\mathrm{obs}}(\bm z_{\mathrm o}(t),\bm u(t),\bm p)$$
- $$\theta^*=\arg\min_\theta\sum_t\|r(t)-R_\theta(z_o(t),u(t))\|_2^2$$
- $$B^*=\arg\min_B\sum_t\|R_{\theta^*}(z_o(t),u(t))-B\Phi(z_o(t),u(t))\|_2^2+\lambda\|B\|_1$$
- $$\text{admit term }j\quad\Longleftrightarrow\quad \left|{\hat b_j\over\mathrm{se}(\hat b_j)}\right|>\tau,\qquad \dot z_o=F_o^{\mathrm{obs}}+B_{\mathrm{sel}}\Phi$$

## Implementation notes

Integrate this into the transition function of a neural ODE, SSM, or recurrent world model. Construct $F_o^{obs}$ from known equations when available; otherwise use a small baseline network frozen for one projection cycle. Define a library containing only dimensionally valid and symmetry-compatible terms, such as $y$, $y^3$, $u$, $yu$, velocity-like finite differences, and cross-coordinate products. First train an MLP residual $R_\theta$ against the one-step derivative residual. Then freeze the MLP, evaluate it on a trajectory buffer, and fit $B$ by elastic net or sequential thresholded least squares. Refit the residual network after replacing its output with $B_{sel}\Phi$, repeating for two to five cycles. For uncertainty, bootstrap trajectories or use the regression covariance $\hat\sigma^2(X^\top X+\lambda I)^{-1}$, where $X$ is the library design matrix; the paper's statistical-gating mechanism is implemented through coefficient $t$-statistics, while the covariance estimate is empirical. The first experiment should use a noisy nonlinear oscillator with a known cubic restoring force and an unobserved auxiliary mode. Compare an unconstrained residual neural ODE, sparse regression alone, and the alternating method on one-step and 100-step rollout error. The falsifiable prediction is that training error will initially favor the unconstrained residual, but after projection the selected model will have lower long-horizon error and lower seed-to-seed variance; accepted terms should remain significant across bootstrap runs, while false library terms should be removed when their $|t|$ statistic is below the chosen threshold.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a reproducible noisy nonlinear-oscillator experiment with an MLP residual, L1 library projection, coefficient t-statistic gating, and sparse-only/neural rollout baselines. The statistical mechanism showed the expected qualitative behavior: signal t-statistics increased with sample size (observed exponent 0.606 versus predicted 0.5) and null acceptance was low (0.0178 versus predicted 0.0124), but the endpoint ratio was 5.34 versus predicted 4.0. The projected symbolic model improved acceleration rollout RMSE over the neural residual (0.05294 versus 0.06442), yet it was identical to sparse regression alone and admitted spurious v and v3 terms, so the claimed residual-to-symbolic advantage and reliable hidden-state term rejection were not demonstrated.

### Mechanism check

- Verdict: Built a reproducible noisy nonlinear-oscillator experiment with an MLP residual, L1 library projection, coefficient t-statistic gating, and sparse-only/neural rollout baselines. The statistical mechanism showed the expected qualitative behavior: signal t-statistics increased with sample size (observed exponent 0.606 versus predicted 0.5) and null acceptance was low (0.0178 versus predicted 0.0124), but the endpoint ratio was 5.34 versus predicted 4.0. The projected symbolic model improved acceleration rollout RMSE over the neural residual (0.05294 versus 0.06442), yet it was identical to sparse regression alone and admitted spurious v and v3 terms, so the claimed residual-to-symbolic advantage and reliable hidden-state term rejection were not demonstrated.
- Confidence: 8/10
- Limitations: The rollout metric is acceleration error rather than full observed-state trajectory error, only one fixed seed was used for the model comparison, no multi-cycle refitting or bootstrap uncertainty was tested, and the latent auxiliary mode was not estimated. The sparse projection was effectively equivalent to direct sparse regression because the final accepted coefficients were refit against the measured residual.

## Artifacts

- [experiment.py](https://synthcore.org/code/610/experiment.py)
- [report.md](https://synthcore.org/code/610/report.md)
- [results.json](https://synthcore.org/code/610/results.json)
- [Download all files as ZIP](https://synthcore.org/download/610)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.