Activation-SVD Zeroth-Order Adapters

1. Integration point: modify every selected Transformer linear layer, especially attention q/k/v/o projections and MLP up/down projections. Keep the original weight W frozen, including INT4 weights, and add a trainable parameter A with shape [r,n]. During the forward pass compute Y=XW+X V_r A; the base product can use the existing quantized kernel, while V_r A is stored and computed in fp16 or bf16. 2. Calibration: collect activation rows X from 100-1000 representative sequences for each layer, concatenate or stream a covariance C=X^T X, compute the top-r eigenvectors V_r of C by randomized SVD, and freeze V_r. Initialize A=0. 3. Zeroth-order loop: sample u with the same shape as A and normalize it by its RMS; evaluate the model twice with A_plus=A+mu*u and A_minus=A-mu*u, compute g=(loss_plus-loss_minus)/(2*mu)*u, and update Adam moments m=beta1*m+(1-beta1)*g and v=beta2*v+(1-beta2)*g^2, followed by A=A-lr*m/(sqrt(v)+eps). Use common minibatches and identical dropout/random seeds for the plus and minus evaluations. 4. The paper-derived quantities are V_r and the coefficient dimension rn; estimate sigma_{r+1} and activation energy empirically from the SVD. Start with r in {4,8,16,32}, mu in {1e-3,3e-3,1e-2}, and one or four random directions per update. The first cheap test should use a 125M-350M decoder-only model on WikiText-2 or a small instruction dataset, comparing full-parameter MeZO, random LoRA-subspace ZO, and this adapter at equal forward FLOPs. Measure loss versus wall-clock time, peak memory, and final validation perplexity. Success is lower loss at equal forward evaluations, substantially lower peak memory, and stable Adam moments; a useful target is at least 2x lower estimator variance or faster loss descent than full-space ZO.