Contractive Rank-Adaptive Residual Layer

(1) Exact integration point: add this as a recurrent propagation block or replace the token-mixing operation in a small Transformer or state-space model. For each example, arrange hidden activations as $U\in\mathbb{R}^{L\times d}$, where rows are tokens or time positions and columns are channels. Use a learned propagation operator $A_\theta$ that is cheap to apply, for example $A_\theta U=-DU+KU$, with $D=\operatorname{diag}(\operatorname{softplus}(d_i))$ and $K$ a depthwise, convolutional, or low-rank mixing operator. The trainable update is applied during both forward training and inference.

(2) Pseudocode: U = input; for n in range(N): Z = U + tau * A_theta(U); estimate the largest singular value of the linear map V -> V + tau*A_theta(V) using 2--5 power iterations; if it exceeds one, divide the residual increment by max(1,s); compute an exact or randomized SVD Z=P diag(sigma) Q.T; choose the smallest r satisfying sum(sigma[r:]2) <= eps_n2; set U = P[:,:r] @ diag(sigma[:r]) @ Q[:r,:]; then apply the output projection or pointwise nonlinearity. Use a rank cap and rank floor. Backpropagate through the retained SVD components, while treating the discrete rank decision as stop-gradient or using a straight-through estimator.

(3) Computed from the paper's mathematics: the forward-Euler residual step, singular-value-tail error criterion, adaptive rank, Frobenius truncation error, and norm-rescaling rule. Estimated empirically: the operator norm, an appropriate step size $\tau$, tolerance schedule $\varepsilon_n$, and the accuracy-memory Pareto curve. Set $\varepsilon_n=\rho\|U\|_F$ with $\rho\in\{10^{-2},10^{-3},10^{-4}\}$, or use a fixed activation-scale-normalized tolerance. Log the maximum norm ratio and the rank at every block.

(4) First cheap experiment: implement a 6-layer sequence classifier or small language model on WikiText-2, comparing a standard residual token-mixing layer, a fixed-rank low-rank layer, and this adaptive step-and-truncate layer. Start with sequence length $L=256$, width $d=512$, four propagation substeps, rank cap 64, and tolerances $\varepsilon\in\{10^{-2},10^{-3},10^{-4}\}\|U\|_F$. Measure validation loss or accuracy, average and peak retained rank, activation memory, wall-clock SVD cost, and the maximum observed ratio $\|U^{n+1}\|_F/\|U^n\|_F$. The method succeeds if it reduces activation memory or rank at matched accuracy, has no exploding hidden norms, and gives a smoother loss curve than an unconstrained low-rank residual baseline. Also report whether randomized SVD makes the compression overhead smaller than the memory savings.