Commutator-Polynomial Residual Adapter

Integrate the construction into a bottleneck residual adapter inside an MLP or transformer feed-forward block. Let the hidden width be \(d\), choose bottleneck rank \(r_b\), and implement the adapter as \(U T V\), with \(U\in\mathbb R^{d\times r_b}\), \(V\in\mathbb R^{r_b\times d}\), and \(T\in\mathbb R^{r_b\times r_b}\). Initialize \(A\) as diagonal with distinct bounded real entries, or learn \(A\) while clipping its spectral norm. Choose \(P(z)=z^2+2\gamma z+\gamma^2\), with \(\gamma>0\), and at every training step compute C = A @ T - T @ A, then R = A @ C - C @ A + 2*gamma*C + gamma**2*T. Add lambda_comm * mean(R**2) to the task loss. Add an optional nilpotency penalty lambda_nil * ||T^k||_F^2, with \(k=\min(4,r_b)\); in a separate exact ablation, parameterize \(T\) as strictly upper triangular, which guarantees \(T^{r_b}=0\). The paper supplies the commutator polynomial and nilpotency implication; the penalty weight, optimization behavior, and useful nilpotency index must be estimated empirically. First compare rank-16 constrained adapters, unconstrained rank-16 LoRA, and strictly triangular rank-16 adapters on a 100M-parameter causal language model or GLUE fine-tuning. Record validation loss, gradient-norm spikes, activation norms after repeatedly applying the adapter 24 times, and parameter count. Success means lower loss variance or fewer instability events at equal parameter count, with no meaningful validation degradation. Also test whether the finite inverse sum((-T)**q for q in range(k)) is cheaper and more stable than iterative inversion.