Carleman-Lifted Polynomial State Space

Integrate this at the transition of a sequence model, replacing the usual RNN update $h_{t+1}=h_t+\Delta t\,\phi(W h_t+U x_t)$ with a truncated lifted state. First choose lift order $N=2$ or $3$, base width $d$, and represent $z_1=h$, $z_2\approx h\otimes h$, and optionally $z_3\approx h\otimes h\otimes h$ using tensor-train or low-rank factors rather than dense vectors of sizes $d^2$ and $d^3$. For each input $x_t$, produce $F_0(x_t)$, $F_1(x_t)$, and $F_2(x_t)$ with small linear projections or an MLP. Compute, for every level $j$, the three contributions corresponding to $A_{j+1}^{j}z_{j+1}$, $A_j^jz_j$, and $A_{j-1}^{j}z_{j-1}$; in code, implement the Kronecker sums as factor-wise contractions, never materializing Kronecker matrices. Use an Euler, midpoint, or exponential-integrator step: $z_j\leftarrow z_j+\Delta t(A_{j+1}^{j}z_{j+1}+A_j^jz_j+A_{j-1}^{j}z_{j-1})$. Initialize $z_j$ from repeated low-rank contractions of $z_1$ or from learned projections. During training, estimate the omitted defect $r_t=\|A_{N+1}^{N}z_{N+1}\|$ using either a temporary approximate $z_{N+1}$ or a low-rank sampled contraction, and add $\lambda\sum_t r_t^2$ to the task loss; alternatively increase $N$ when a moving average of $r_t$ exceeds a threshold. The mathematical quantities are the block couplings and defect form; low-rank rank, step size, and defect threshold are empirical hyperparameters. Start with a 2-layer GRU and a Carleman-Lifted model of matched parameter count on adding, copy-memory, and long-range oscillatory synthetic tasks, then test WikiText-2 character modeling. Success means lower long-horizon error, fewer exploding trajectories, or faster loss descent at equal parameter count and FLOPs; failure is higher cost without improved stability or accuracy.