Intermittent Multi-Mode Memory Gate

1. Integration point: place the gate before an RNN update or MoE expert selection. Maintain a scalar state s_t in [0,1] for each sequence, head, or token, and choose K mode centers xi_k. For the nearest center k, set u=s_t-xi_k and apply s_new=clip(xi_k+u+gamma*b_k*sign(u)*(abs(u)+epsilon)^(1+alpha_k)+sigma*g(h_t),0,1), where g is a bounded projection of the current hidden state, gamma is the integration scale, sigma is input noise, and epsilon prevents zero-gradient numerical issues. Form routing logits ell_j=-q*(s_new-xi_j)^2+r_j and use p=softmax(ell/tau) to interpolate expert or residual updates. 2. Pseudocode: initialize s; at each step find k=argmin_j abs(s-xi_j); compute u=s-xi_k; update s with the intermittent formula; compute p; then update h_(t+1)=sum_j p_j RNN_j(h_t,x_t). Detach s only if backpropagation through long trajectories is unstable. 3. The power-law exponent is designed by alpha_k and b_k; estimate the realized exponent by collecting mode residence times and fitting the survival curve Pr(tau>n) on log-log axes. Estimate pressure branches using blockwise H(p)-beta times the empirical mean preference. 4. First cheap experiment: compare a GRU and this gate-augmented GRU on the adding task, sequential MNIST, and a synthetic two-regime sequence with controllable dwell times. Use K=2 and alpha_1=alpha_2=1 while sweeping gamma. The predicted signature is tau proportional to u_0^(-1), with fitted log-log slope -1 plus or minus 0.2. On the two-regime task, sweep beta or routing temperature and predict a kink in mode occupancy and entropy at beta_c=(H_A-H_B)/(R_A-R_B), with the measured transition within 20 percent of the estimated crossing. Long-context accuracy should improve specifically for power-law dwell-time data, while exponential-dwell controls should show substantially less benefit.