# Banded Production-Matrix Polynomial Layer

- ID: 2794
- Canonical URL: https://synthcore.org/idea/2794/banded-production-matrix-polynomial-layer
- API JSON: https://synthcore.org/api/idea/2794.json
- API Markdown: https://synthcore.org/api/idea/2794.md
- Verification status: unverified
- Source: [arXiv:2608.28834](https://arxiv.org/abs/2608.28834)
- Category: architecture
- Solves: speedup, memory, stability
- ML areas: mlp, architecture, initialization
- Math tags: linear-algebra, combinatorics, numerical-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a dense learned polynomial-feature transform with a d-orthogonal recurrence whose production matrix is constrained to a (d+2)-banded lower-Hessenberg form. The layer generates successive features using only local recurrence coefficients, giving O(dN) arithmetic and O(dN) parameters for N basis functions instead of O(N^2) dense mixing.

## Mathematical statement

Let p_n(x)=\sum_{k=0}^{n}A_{n,k}x^k be a polynomial basis and let A be its lower-triangular coefficient matrix. Let S be the up-shift matrix, with S_{n,k}=\delta_{k,n+1}. The paper defines the production matrix associated with A^{-1} by P_{A^{-1}}=ASA^{-1}. For an ordinary orthogonal polynomial sequence, P is tridiagonal and encodes a three-term recurrence; for a d-orthogonal sequence, P is (d+2)-banded lower Hessenberg and encodes a (d+2)-term recurrence. In a neural layer, use the equivalent recurrence x p_n(x)=\sum_{j=-d}^{1}P_{n,n-j}p_{n-j}(x), where coefficients outside valid indices are zero; the upper one-step term generates p_{n+1}, while the d lower terms couple only the previous d+1 features. Here x is a scalar feature or a normalized projection of a vector input, n is the polynomial order, P_{i,j} are trainable recurrence coefficients, d controls bandwidth, and N is the number of generated features. The lower-Hessenberg constraint is the transferable structure: it makes basis generation sparse and avoids explicitly forming A or A^{-1}.

## Key formulas

- $$p_n(x)=\sum_{k=0}^{n}A_{n,k}x^k,\qquad P_{A^{-1}}=ASA^{-1},\qquad S_{n,k}=\delta_{k,n+1}.$$
- $$x\,p_n(x)=\sum_{j=-d}^{1}P_{n,n-j}\,p_{n-j}(x),\qquad P_{n,k}=0\ \text{unless}\ n-d-1\le k\le n+1.$$
- $$\begin{aligned}q_0&=1,\\ q_{n+1}&=\frac{1}{P_{n,n+1}}\left(xq_n-\sum_{j=0}^{d}P_{n,n-j}q_{n-j}\right),\qquad n\ge 0,\end{aligned}$$
- $$\text{cost}_{\mathrm{recurrence}}=O(dN)\quad\text{versus}\quad \text{cost}_{\mathrm{dense\ basis}}=O(N^2).$$

## Implementation notes

(1) Integration point: insert the module between an encoder and an MLP head, or use it as a replacement for a dense polynomial-feature block. For an input vector u in R^D, first compute a scalar coordinate x=tanh(w^T u+b), with w normalized or spectrally constrained; optionally use one independent module per coordinate or per channel. The output is h=(q_0(x),...,q_{N-1}(x)), followed by a linear projection. Train the recurrence coefficients jointly with the rest of the network, but parameterize P so that only the diagonals k=n+1,n,n-1,...,n-d are stored. (2) Pseudocode: set q[0]=ones_like(x); for n=0,...,N-2, compute rhs=x*q[n]; subtract P[n,n-j]*q[n-j] for j=0,...,min(d,n); set q[n+1]=rhs/P[n,n+1]. Use softplus(P[n,n+1]) plus epsilon for a positive forward coefficient. Concatenate q or apply a learned projection. (3) The paper's mathematical ingredient is the sparse production-matrix/banded-recurrence correspondence. Estimate no matrix inverse during training; the recurrence coefficients are direct parameters. For stability, estimate the empirical second moment of each q_n and apply per-order normalization, or initialize with Chebyshev coefficients for the three-term case and small lower-band coefficients for d>1. Clip or regularize the lower-band coefficient norm by adding lambda times the squared Frobenius norm of the stored bands. (4) First experiment: compare a dense polynomial feature layer and this recurrence layer on MNIST or CIFAR-10 with a small MLP, using N=32 and d in {1,2,4}, equal hidden width and matched parameter counts. Measure validation accuracy, wall-clock time, peak memory, gradient norm, and activation variance as N increases. Success means comparable or better accuracy with at least 2x lower layer parameters and visibly lower memory, or faster loss descent at equal parameter count; failure is exploding activations, strong order-dependent conditioning, or accuracy loss exceeding 1 percentage point.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.