# Stable Hyperplane Feature Layer

- ID: 2786
- Canonical URL: https://synthcore.org/idea/2786/stable-hyperplane-feature-layer
- API JSON: https://synthcore.org/api/idea/2786.json
- API Markdown: https://synthcore.org/api/idea/2786.md
- Verification status: unverified
- Source: [arXiv:2608.28004](https://arxiv.org/abs/2608.28004)
- Category: architecture
- Solves: scalability, generalization, memory
- ML areas: embedding, mlp, moe-routing
- Math tags: probability, stochastic-processes, geometry, metric-geometry
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a dense directional feature map with a finite stable-zonotope layer: a small set of large random vectors creates a hyperplane arrangement over the input sphere, while a cheaper small-jump residual gives continuous features inside each cell. The layer exposes an explicit compute-versus-resolution knob through the jump cutoff and can be used either as an embedding before an MLP or as a lightweight router for mixture-of-experts blocks.

## Mathematical statement

The paper parametrizes exposed points using $X(u)=\sum_{k=1}^{\infty}\Gamma_k^{-1/\alpha}\varepsilon_k\mathbf{1}_{\{\langle u,\varepsilon_k\rangle>0\}}$, where $u\in S^{d-1}$ is the normalized input direction, $\Gamma_k$ is the $k$th arrival time of a unit-rate Poisson process, $\varepsilon_k$ are independent uniform vectors on $S^{d-1}$, and $0<\alpha<1$. The heavy-tailed weights $\Gamma_k^{-1/\alpha}$ are ordered from large to small. For a finite cutoff, split $X=X_{\mathrm{large}}+X_{\mathrm{small}}$: the signs of the large terms define a finite arrangement of hyperplanes $\langle u,\varepsilon_k\rangle=0$, hence angular cells on which the large contribution is constant. The upper-bound construction uses $T_n=C_N\sum_{Q\in\mathcal Q_n}(1+M_Q)^N V_Q^s$ and $\eta_n=\max_{Q\in\mathcal Q_n}V_Q$, where $N=d-1$, $Q$ are cells, $M_Q$ counts relevant large-jump hyperplanes, and $V_Q$ bounds the image diameter of the small-jump residual on $Q$. The theorem states $\dim_H\overline{X(S^{d-1})}\le N\alpha$, with a matching lower bound, motivating $\alpha$ as a controllable feature-complexity parameter.

## Key formulas

- $$X(u)=\sum_{k=1}^{\infty}\Gamma_k^{-1/\alpha}\varepsilon_k\mathbf{1}_{\{\langle u,\varepsilon_k\rangle>0\}},\qquad u\in S^{d-1},\quad 0<\alpha<1.$$
- $$X_{\mathrm{large}}(u)=\sum_{k=1}^{K}w_k\varepsilon_k\mathbf{1}_{\{\langle u,\varepsilon_k\rangle>0\}},\qquad X_{\mathrm{small}}(u)=\sum_{k=K+1}^{K+R}w_k\varepsilon_k\mathbf{1}_{\{\langle u,\varepsilon_k\rangle>0\}},\quad w_k=\Gamma_k^{-1/\alpha}.$$
- $$\dim_H\overline{X(S^{d-1})}\le (d-1)\alpha,\qquad \dim_H\ext Z_\alpha=(d-1)\alpha\quad\text{almost surely}.$$
- $$T_n=C_N\sum_{Q\in\mathcal Q_n}(1+M_Q)^N V_Q^{\,s},\qquad \eta_n=\max_{Q\in\mathcal Q_n}V_Q.$$

## Implementation notes

Integrate this as a residual embedding or router immediately before an MLP or MoE block. Given an input vector $h\in\mathbb R^d$, compute $u=h/(\|h\|_2+10^{-6})$. Sample and freeze $K+R$ unit vectors $\varepsilon_k$; generate ordered positive weights using $\Gamma_k=\sum_{j=1}^kE_j$ with $E_j\sim\operatorname{Exp}(1)$, then set $w_k=\Gamma_k^{-1/\alpha}$. Clip the weights to $[-w_{\max},w_{\max}]$ or normalize the resulting code by its RMS to avoid rare Poisson outliers. Compute half-space gates $g_k=\mathbf 1[\langle u,\varepsilon_k\rangle>0]$. Form the large code $z_L=\sum_{k=1}^K w_kg_k\varepsilon_k$ and small code $z_S=\sum_{k=K+1}^{K+R}w_kg_k\varepsilon_k$. Concatenate $[u,z_L,z_S]$, apply a learned linear projection, and add the result as a residual to the hidden state. For MoE routing, feed $[g_1,\ldots,g_K,\langle u,\varepsilon_1\rangle,\ldots,\langle u,\varepsilon_K\rangle]$ to a small router and cache the large gates for repeated tokens. The mathematical quantities computed exactly are the Poisson-ordered heavy-tailed weights and hyperplane signs; estimate practical cell complexity by counting distinct $K$-bit gate patterns per minibatch. First test a parameter-matched MLP and a small Transformer on CIFAR-10, TinyImageNet, or a language-modeling subset, against a dense projection and random Fourier features. Sweep $\alpha\in\{0.3,0.5,0.7,0.9\}$ and $K/R\in\{1/4,1/2,1\}$. Record validation accuracy, activation RMS, number of occupied angular cells, router FLOPs, and parameter count. Success is equal or better accuracy at lower learned-parameter count or router cost, with stable activation statistics and useful cell diversity; failure is gate collapse, exploding activations, or no improvement over the dense baseline at equal compute.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.