Concave-Spectral Residual Aggregation

(1) Integrate this at the aggregation point of a multi-branch residual block. Let each branch output a matrix $A_i\in\mathbb R^{B\times D}$; for convolutional features, reshape each sample's $C\times HW$ activation into a matrix and process samples independently. Compute $S=\sum_iA_i$, $G=S^\top S+\varepsilon I$, and the eigendecomposition $G=V\operatorname{diag}(\lambda_j)V^\top$. Define $f_\alpha(t)=(t+\varepsilon)^{\alpha/2}-\varepsilon^{\alpha/2}$ on squared singular values and return $Y=V\operatorname{diag}(f_\alpha(\lambda_j))V^\top$; this applies a concave transform to the singular spectrum. Add a learned residual scale $\beta$ and output $x+\beta Y$, with $\beta$ initialized to $0.1$. (2) Pseudocode: A=[branch_i(x) for i in range(m)]; S=sum(A); G=S.T@S + eps*I; lam,V=eigh(G); Y=V@diag((lam+eps)(alpha/2)-eps(alpha/2))@V.T; out=x+beta*Y. For efficiency, use randomized rank-$r$ SVD or apply the transform only to channel groups of size 32--64. (3) The paper supplies the structural prediction $C^{conc}=C^{lin}$; do not assume a numerical $C^{lin}$ unless a baseline inequality is known. Estimate the empirical ratios $R=\|f(|S|)\|_{S_p}/\|\sum_i f(|A_i|)\|_{S_p}$ and $R_{lin}=\|S\|_{S_p}/\|\sum_i|A_i|\|_{S_p}$ on minibatches, using the same randomized SVD for both. (4) First experiment: train ResNet-20 on CIFAR-10, comparing ordinary branch summation, concave aggregation with $\alpha\in\{0.5,0.75,1\}$, and a parameter-matched scalar RMS-normalized aggregator. Record training loss, test accuracy, gradient norm, largest singular value, and the 99th percentile of $R/R_{lin}$. Success means lower gradient spikes and lower activation spectral tails at equal accuracy, or improved test accuracy under the same spectral-norm budget; reject the method if SVD overhead produces no stability or accuracy benefit.