Spectral Memory-Lift Ensemble

Integrate the module at the token-processing position occupied by an S4/Mamba-style SSM, GRU, or the residual stream of a small Transformer. Let q_t be the current token embedding or hidden vector of width d, u_t be the same token projection plus optional positional features, and initialize z_{0,e}=0 for each of E=4 memory experts with state width h=8 or 16. For every time step and expert, execute: (1) qnext_e=A_e q_t+B_e z_e+F_e u_t+b_e; (2) znext_e=C_e q_t+D_e z_e; (3) v_e=output_projection(qnext_e,znext_e); (4) compute pi_t with a two-layer gate and mix y_t=sum_e pi_t,e v_e; (5) add y_t to the residual stream. Parameterize D_e=tanh(r_e)Dhat_e/sigma_max(Dhat_e), with a learned scalar or diagonal r_e constrained below one, and initialize experts with distinct target timescales such as 2, 8, 32, and 128 tokens. Compute M_e=[[A_e,B_e],[C_e,D_e]] and estimate lambda_star,e every 100 optimizer steps using 10 power iterations on M_e^T M_e; use the resulting upper-bound singular value as a conservative stability proxy. Add L_spec=beta sum_e ReLU(sigma_bound,e-0.98)^2, or rescale M_e by 0.98/max(1,sigma_bound,e) before the recurrence. The gate's tau term is the paper-inspired slow-mode signal; stop gradients through tau initially to avoid unstable eigenvalue optimization. The exact recurrence and kernel-equivalent state update come from the paper, while spectral values and gating parameters are estimated numerically or learned. First test on character-level enwik8 or TinyStories with a 4-layer 256-width causal model, comparing a standard GRU, a single learned SSM, and this four-expert lift at equal parameter count and sequence length 2048. Measure validation perplexity, gradient norm spikes, hidden-state norm, and throughput. Success means lower perplexity at long context than the single SSM, no exploding states, and improved loss at sequence lengths where the baseline must increase explicit context or attention cost.