Dynamic String-Averaged Constraint Layer

Integrate the module between transformer blocks or immediately before a task head, operating on token representations h in R^{B x N x d}. Define M affine constraints, each with a fixed or learned orthogonal projector L_i in R^{d x d} and offset p_i in R^d; for low cost, parameterize L_i=U_i U_i^T with orthonormal low-rank U_i, or use diagonal masks. At training step t, construct R strings s[r,1:L_r] with 1 <= L_r <= L_max. Maintain a usage counter and force every constraint to appear at least once in each W-step window. Compute: outputs=[]; for r in range(R): y=h; for i in string[r]: y=(1-lambda[i])*y + lambda[i]*(y @ L[i].T + p[i]); outputs.append(y); hout=sum(omega[r]*outputs[r] for r in range(R)). Choose omega by softmax over learnable logits or uniformly, and clip lambda_i to [epsilon,2-epsilon]. Add hout=h+alpha*(hout-h) with alpha initialized to zero. The paper supplies the affine projection decomposition and nonexpansive composition principle; the projector parameters, strings, and weights are learned or sampled by the experiment. First test on CIFAR-10 with a residual MLP or tiny transformer, comparing an unconstrained residual block, fixed sequential projections, and dynamic string averaging at equal FLOPs. Measure accuracy, constraint residuals ||h-L_i h-p_i||, activation norms, gradient variance, and robustness to injected projection noise. Success means lower residuals and equal-or-better accuracy without exploding activations.