Schur-Riesz Greedy Adapter Expansion

(1) Integration point: apply the procedure during progressive construction of LoRA adapters, feature banks, or MoE experts, while keeping the base Transformer frozen in the first experiment. For a selected layer, candidate j is a low-rank update Delta W_j=A_j B_j, an extra MLP feature group, or an expert. Collect its response C_j on a fixed calibration batch. Use T=I on logits for a cheap version, or use Jacobian-vector products T C_j to measure the candidate's first-order effect on logits. The incumbent matrix B_m contains responses from accepted blocks. (2) Pseudocode: initialize B as an empty matrix. At each growth round, generate K candidates; evaluate R_j=T C_j; compute the Y-orthogonal projection using P=B(B^T Y B)^dagger B^T Y; form Q_j=(I-P)R_j; calculate g_j=trace(Q_j^T Y Q_j) and the singular values of Y^{1/2}Q_j; discard candidates with sigma_min_plus below alpha or sigma_max above beta; choose the largest remaining g_j; train only that block for s steps; accept it if held-out loss decreases by at least threshold tau; append its response columns to B; recompute the projection and continue. Stop when no candidate passes tau or a parameter budget is reached. (3) Compute P, Q_j, and g_j from the residualization formula. Estimate alpha, beta, Y, s, and tau empirically. Use QR or randomized SVD rather than explicitly constructing I-P. (4) First cheap experiment: compare against fixed-rank LoRA and random or magnitude-based adapter expansion on a 125M-parameter Transformer fine-tuned on a 10k-example instruction dataset or GLUE task. Plot validation loss versus trainable parameters and FLOPs. Success means at least 20% fewer adapter parameters at matched validation loss, or faster loss descent at equal parameter count, with accepted blocks having substantially larger projected response energy than rejected blocks.