# Fixed-feature invariant output layer

- ID: 119
- Canonical URL: https://synthcore.org/idea/119/fixed-feature-invariant-output-layer
- API JSON: https://synthcore.org/api/idea/119.json
- API Markdown: https://synthcore.org/api/idea/119.md
- Verification status: unverified
- Source: [arXiv:2608.23980](https://arxiv.org/abs/2608.23980)
- Category: architecture
- Solves: speedup, stability, memory
- ML areas: mlp, optimizer, training, loss
- Math tags: linear-algebra, optimization, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Replace repeated nonlinear optimization of a small neural module with a predetermined random-feature basis and a constrained linear output solve. After the least-squares solution, project the output coefficients onto an exactly feasible affine set so that a chosen linear invariant is satisfied despite finite residual error.

## Mathematical statement

The paper defines the transferable representation $u_{\mathrm{NN}}(\bm{x},t)=\sum_{m=1}^{N}\alpha_m(t)\tanh(\bm{w}_m^T\bm{x}+b_m)$, where $\bm{x}$ is the input, $N$ is the number of hidden neurons, $(\bm{w}_m,b_m)$ are fixed hidden parameters, and $\alpha_m(t)$ are time-dependent output coefficients. For samples $x_i$, define the fixed feature matrix $\Phi\in\mathbb{R}^{K\times N}$ by $\Phi_{im}=\tanh(\bm{w}_m^Tx_i+b_m)$. Given a target vector $y\in\mathbb{R}^{K}$, compute the unconstrained least-squares coefficient $\hat{\alpha}=\arg\min_{\alpha}\|\Phi\alpha-y\|_2^2$. Let $c\in\mathbb{R}^{N}$ encode a linear invariant and let $m$ be its desired value, so feasibility is $c^T\alpha=m$. The minimum-change Euclidean projection is $\alpha^*=\arg\min_\alpha\|\alpha-\hat\alpha\|_2^2$ subject to $c^T\alpha=m$, with closed form $\alpha^*=\hat\alpha+c(c^Tc)^{-1}(m-c^T\hat\alpha)$. If the invariant is defined on sampled outputs using quadrature weights $q\in\mathbb{R}^{K}$, then $c=\Phi^Tq$ and $c^T\alpha=q^T(\Phi\alpha)$. For a weighted coefficient metric $W\succ0$, the corresponding projection is $\alpha^*=\hat\alpha+W^{-1}c(c^TW^{-1}c)^{-1}(m-c^T\hat\alpha)$.

## Key formulas

- $$u_{\mathrm{NN}}(\bm{x},t)=\sum_{m=1}^{N}\alpha_m(t)\,\tanh(\bm{w}_m^{T}\bm{x}+b_m),$$
- $$\hat{\alpha}=\arg\min_{\alpha\in\mathbb{R}^{N}}\|\Phi\alpha-y\|_2^2,$$
- $$\alpha^*=\arg\min_{\alpha}\|\alpha-\hat\alpha\|_2^2\quad\mathrm{s.t.}\quad c^T\alpha=m\;=\;\hat\alpha+c(c^Tc)^{-1}(m-c^T\hat\alpha),$$
- $$c_m=\sum_{i=1}^{K}q_i\Phi_{im},\qquad q^T(\Phi\alpha^*)=m.$$

## Implementation notes

Integrate this at the output layer of a compact MLP used as a recurrent state updater, neural operator, or adapter, rather than replacing a full transformer. Inputs are sample coordinates, graph nodes, physical states, or latent positions $x_i$. Choose $N$ fixed hidden features $\phi_m(x)=\tanh(w_m^Tx+b_m)$ and construct $\Phi_{im}=\phi_m(x_i)$ once. Choose a known linear invariant represented by aggregation or quadrature weights $q\in\mathbb{R}^K$; for mass conservation, set $m=q^Ty_{\mathrm{initial}}$. Precompute $c=\Phi^Tq$, factor $\Phi^T\Phi+\lambda I$, and optionally cache its factorization. At each update, form the target $y_t$ from the baseline state update, solve $\hat\alpha=(\Phi^T\Phi+\lambda I)^{-1}\Phi^Ty_t$, then execute `err=m-dot(c,hat_alpha); alpha=hat_alpha+c*err/(dot(c,c)+1e-12)`. Return $\Phi\alpha$ as the output. For multiple invariants, stack rows into $C\in\mathbb{R}^{r\times N}$ and use `alpha=hat_alpha+C.T@solve(C@C.T,b-C@hat_alpha)`. Compute feature conditioning empirically from the singular values of $\Phi$; compute $c$ from known quadrature weights rather than learning it. First test a 2D Allen-Cahn or advection rollout and a graph diffusion task with a 2-layer, 256-feature baseline trained by Adam. Compare against full-network SGD and unconstrained random features at equal parameter count and rollout horizon. Measure wall-clock time, peak memory, task MSE, rollout stability, and invariant drift. Success means at least 3x faster output updates, invariant error near numerical precision, and no more than 5% accuracy loss. Also test transfer by reusing the frozen features on a second domain or initial condition without retraining hidden layers.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.