Jensen-Gap Regularization for Temporal Cascades

Integrate this at the hidden-state dynamics of a stacked scalar- or channelwise-leaky RNN/SSM, not at the final classifier alone. For each training batch, take an input sequence u[0:L-1], construct a mean-preserving periodic perturbation u_per[t]=u[t]+A sin(2 pi t/P) or use a random zero-mean Fourier perturbation, and run the network on both the original and perturbed sequences. Also run a counterfactual constant-input cascade whose input is the temporal mean of u_per. Use the final-period hidden states after burn-in to estimate m_i and the terminal mean output. Pseudocode: (1) compute x_i[t] with x_i[t+1]=(1-alpha_i)x_i[t]+alpha_i phi_i(x_{i-1}[t]); (2) estimate m_i=mean_t x_i[t] over the last K steps; (3) compute c_0=mean_t u_per[t], then c_i=phi_i(c_{i-1}); (4) compute J_i=mean_t phi_i(x_{i-1}[t])-phi_i(m_{i-1}); (5) compute a_i as the divided difference in the formula above; (6) form delta_decomp=sum_i J_i product_{j>i} a_j; (7) add L_JG=lambda*(mean(y_per)-mean(y_const)-stopgrad(delta_decomp))^2 plus lambda_abs*delta_decomp^2 when invariance is desired. The network output y may be a learned readout of x_n. The simulation supplies m_i and J_i; a_i is computed directly from activations, while the second-order approximation uses empirical variance and automatic differentiation for phi_i''. Start with a 3-layer leaky tanh RNN on a synthetic sequence-regression task where the target depends on the mean of a slowly varying signal but training inputs contain random sinusoidal modulation. Compare a standard RNN, ordinary temporal augmentation, and this regularizer at equal parameter count. Success means a lower terminal mean shift under unseen frequencies and amplitudes, reduced prediction error under modulation, and no more than a small clean-input accuracy loss. A useful diagnostic is whether the measured shift agrees with the weighted sum of local J_i; poor agreement indicates vector interactions or insufficient burn-in.