Fractional Memory State-Space Layer

Integrate the layer at the recurrent or SSM block, immediately after the input projection and before the usual gated update or residual output. For a sequence x[1:T] with channel dimension d, choose J=8--32 positive decay rates lambda_j logarithmically spaced between 1/T_context and 1, or between the target longest timescale and the Nyquist-scale decay. Maintain q[j,d]. At timestep t with spacing Delta, update each mode using the exact discretization q_j <- exp(-lambda_j Delta) q_j + (1-exp(-lambda_j Delta))/lambda_j times x_t. Compute r_t = a x_t - sum_j w_j q_j, with a=sum_j w_j/lambda_j. Feed the concatenation of x_t and r_t into a learned projection or gated transition, for example h_t=tanh(W_x x_t + W_r r_t + A h_{t-1}+b). Parameterize lambda_j=softplus(theta_j)+epsilon and w_j=softplus(phi_j) so every mode remains stable and positive. Initialize lambda_j logarithmically and initialize weights proportional to lambda_j^p over logarithmic bins, which approximates k(s)=s^{p-1}; optionally learn p through p=0.05+0.90 sigmoid(p_raw). For a complex-valued SSM, initialize the memory projection phase using C_p=Gamma(p) exp(i pi p/2); for a real network, represent its real and imaginary parts with two channels. The paper provides the fractional exponent and phase structure, while the exponential quadrature weights and rates are learned or fitted empirically. First test a 1-layer 256-hidden-unit model on copy/retrieval and Long Range Arena Pathfinder or ListOps against a GRU and diagonal SSM at equal parameter count and FLOPs. Measure loss versus sequence length, long-context accuracy, and stability under extrapolated context. Success means slower accuracy decay beyond the training length or equal accuracy with fewer recurrent states than the baselines.