# Sparse symbolic vector-field layer

- ID: 45
- Canonical URL: https://synthcore.org/idea/45/sparse-symbolic-vector-field-layer
- API JSON: https://synthcore.org/api/idea/45.json
- API Markdown: https://synthcore.org/api/idea/45.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.22112](https://arxiv.org/abs/2608.22112)
- Category: architecture
- Solves: memory, generalization, accuracy
- ML areas: world-model, mlp, neural-ode, pruning, regularization
- Math tags: algebra, approximation-theory, optimization, dynamical-systems
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Replace an opaque MLP vector field with a stack of trainable symbolic primitives that can express linear terms, monomials, products, and related analytic operations. Apply an L1 penalty and prune small primitive coefficients after rollout training, yielding a compact dynamics module that is cheaper to evaluate and easier to inspect.

## Mathematical statement

Each operational layer receives $\bm{z}\in\mathbb{R}^{4L(k-1)+n+1}$, consisting of the $n$-dimensional state, an appended constant value $2$, and $4L(k-1)$ outputs from earlier stacks. It emits primitive features; the extracted definitions include $f^{k,l}_{1}(\bm{z})=\bm{w}_{lin}^{k,l}\cdot\bm{z}$ and $f^{k,l}_{2}(\bm{z})=\prod_{i=1}^{d}|z_i|^{w_{pow,i}^{k,l}}$. Here $k$ is the stack index, $l$ is the layer index, $L$ is the number of layers per stack, $d$ is the number of active input coordinates, $\bm{w}_{lin}^{k,l}$ are linear coefficients, and $w_{pow,i}^{k,l}$ are learnable exponents. The complete SymNN composes layers and concatenates prior features, ultimately producing a vector field $\tilde{\bm{f}}(\bm{x})$. The transferable property is a differentiable algebraic basis with adaptive exponents, rather than a fixed polynomial feature map: L1 regularization can eliminate irrelevant terms, producing a sparse approximation of $\bm{f}$ while retaining nonlinear compositional expressivity. For numerical safety, implement the power primitive as $\exp(w_{pow,i}\log(|z_i|+\epsilon))$ with $\epsilon>0$, and constrain exponents to a bounded interval.

## Key formulas

- $$f^{k,l}_{1}(\bm{z})=\bm{w}_{lin}^{k,l}\cdot\bm{z},$$
- $$f^{k,l}_{2}(\bm{z})=\prod_{i=1}^{d}|z_i|^{w_{pow,i}^{k,l}},$$
- $$\tilde{\bm{f}}(\bm{x})=\mathrm{SymNN}_{\theta}\!\left(\begin{bmatrix}\bm{x}\\2\end{bmatrix}\right),\qquad \mathcal{L}(\theta)=\mathcal{L}_{\mathrm{rollout}}(\theta)+\lambda\|\theta\|_{1},$$
- $$\prod_{i=1}^{d}|z_i|^{w_{pow,i}}=\exp\!\left(\sum_{i=1}^{d}w_{pow,i}\log(|z_i|+\epsilon)\right)$$

## Implementation notes

Use this as the vector-field parameterization inside the RK4 rollout, or as a compact replacement for an MLP in a low-dimensional continuous-time world model. For each input state $x\in\mathbb{R}^{n}$, form $z=[x;2]$. In every operational layer, compute a linear feature $u_{lin}=w_{lin}^{\top}z$ and a power feature $u_{pow}=\exp(\sum_i w_{pow,i}\log(|z_i|+\epsilon))$; concatenate these with the layer input and selected outputs from previous stacks, then map the resulting features to the $n$ vector-field coordinates with a final linear readout. Initialize $w_{pow}$ near small integer exponents such as 0 or 1, and bound exponents with $w_{pow}=p_{\max}\tanh(a)$ to avoid singular or explosive powers. Train with rollout MAE plus $\lambda\|\theta\|_1$, applying the L1 term to linear and readout weights and optionally to feature gates. After training, prune terms whose gate or outgoing coefficient is below a threshold, refit the surviving coefficients with a small L1 or L2 penalty, and evaluate the pruned model again. The paper supplies the primitive formulas and sparse-layer motivation; epsilon stabilization, exponent bounds, and pruning thresholds are empirical choices. Begin with 2D pendulum and 3D Lorenz data, using a 3-layer one- or two-stack model against an equal-parameter tanh MLP. Measure active-term count, parameter count, vector-field evaluations per rollout, short-horizon error, and long-time statistics. A useful success criterion is at least 5x fewer active terms with no more than 5% degradation in 32-step prediction error, together with improved extrapolation under modest distribution shift.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a stabilized sparse symbolic vector-field layer with bounded adaptive exponents, RK4 rollout training, L1 regularization, pruning, and a matched tanh-MLP baseline. The power-product identity and finite-difference gradient check passed, but the empirical effect was negative: symbolic rollout MAE was 0.0806 versus 0.0457 for the MLP, with 1.76x higher error, 1.2x more active terms after pruning, and 134 versus 82 parameters. The proposed accuracy and compactness win was not observed in this MVP.

### Mechanism check

- Verdict: Built a stabilized sparse symbolic vector-field layer with bounded adaptive exponents, RK4 rollout training, L1 regularization, pruning, and a matched tanh-MLP baseline. The power-product identity and finite-difference gradient check passed, but the empirical effect was negative: symbolic rollout MAE was 0.0806 versus 0.0457 for the MLP, with 1.76x higher error, 1.2x more active terms after pruning, and 134 versus 82 parameters. The proposed accuracy and compactness win was not observed in this MVP.
- Confidence: 9/10
- Limitations: Only a damped 2D pendulum was tested; Lorenz dynamics, long-time statistics, FLOP-level evaluation cost, and broader hyperparameter searches were not tested. Parameter counts were not exactly equal because the symbolic model includes exponent and gate parameters.

## Artifacts

- [experiment.py](https://synthcore.org/code/58/experiment.py)
- [report.md](https://synthcore.org/code/58/report.md)
- [results.json](https://synthcore.org/code/58/results.json)
- [Download all files as ZIP](https://synthcore.org/download/58)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.