{
 "artifacts": null,
 "category": "optimization",
 "description": "Freeze each pretrained linear weight matrix and optimize only a coefficient matrix in the top right-singular subspace of a one-shot activation SVD. Estimate the loss gradient with antithetic finite differences in coefficient space rather than in the full weight space, reducing both the number of forward perturbation directions and the variance of the estimator.",
 "formulas_latex": [
  "$$\\mathbf{X}=\\mathbf{U}\\mathbf{D}\\mathbf{V}^{\\top},\\qquad \\mathbf{g}_{\\mathbf{W}}=\\mathbf{X}^{\\top}\\mathbf{g}_{\\mathbf{Y}}=\\mathbf{V}\\mathbf{D}\\mathbf{U}^{\\top}\\mathbf{g}_{\\mathbf{Y}}.$$",
  "$$\\mathbf{g}_{\\mathbf{W}}\\approx\\mathbf{V}_{r}\\mathbf{D}_{r}\\mathbf{U}_{r}^{\\top}\\mathbf{g}_{\\mathbf{Y}},\\qquad \\left\\|\\mathbf{g}_{\\mathbf{W}}-\\mathbf{V}_{r}\\mathbf{V}_{r}^{\\top}\\mathbf{g}_{\\mathbf{W}}\\right\\|_{F}\\leq \\sigma_{r+1}(\\mathbf{X})\\left\\|\\mathbf{g}_{\\mathbf{Y}}\\right\\|_{F}.$$",
  "$$\\Delta\\mathbf{W}=\\mathbf{V}_{r}\\mathbf{A},\\quad \\mathbf{A}\\in\\mathbb{R}^{r\\times n},\\quad d=rn,$$",
  "$$\\widehat{\\nabla F}(\\mathbf{A})=\\frac{F(\\mathbf{A}+\\mu\\mathbf{u})-F(\\mathbf{A}-\\mu\\mathbf{u})}{2\\mu}\\mathbf{u},\\qquad \\mathbf{u}\\sim\\mathcal{N}(0,\\mathbf{I}_{d}).$$"
 ],
 "id": 322,
 "implementation": "1. Integration point: modify every selected Transformer linear layer, especially attention q/k/v/o projections and MLP up/down projections. Keep the original weight W frozen, including INT4 weights, and add a trainable parameter A with shape [r,n]. During the forward pass compute Y=XW+X V_r A; the base product can use the existing quantized kernel, while V_r A is stored and computed in fp16 or bf16. 2. Calibration: collect activation rows X from 100-1000 representative sequences for each layer, concatenate or stream a covariance C=X^T X, compute the top-r eigenvectors V_r of C by randomized SVD, and freeze V_r. Initialize A=0. 3. Zeroth-order loop: sample u with the same shape as A and normalize it by its RMS; evaluate the model twice with A_plus=A+mu*u and A_minus=A-mu*u, compute g=(loss_plus-loss_minus)/(2*mu)*u, and update Adam moments m=beta1*m+(1-beta1)*g and v=beta2*v+(1-beta2)*g^2, followed by A=A-lr*m/(sqrt(v)+eps). Use common minibatches and identical dropout/random seeds for the plus and minus evaluations. 4. The paper-derived quantities are V_r and the coefficient dimension rn; estimate sigma_{r+1} and activation energy empirically from the SVD. Start with r in {4,8,16,32}, mu in {1e-3,3e-3,1e-2}, and one or four random directions per update. The first cheap test should use a 125M-350M decoder-only model on WikiText-2 or a small instruction dataset, comparing full-parameter MeZO, random LoRA-subspace ZO, and this adapter at equal forward FLOPs. Measure loss versus wall-clock time, peak memory, and final validation perplexity. Success is lower loss at equal forward evaluations, substantially lower peak memory, and stable Adam moments; a useful target is at least 2x lower estimator variance or faster loss descent than full-space ZO.",
 "math_summary": "For a linear layer, X is the calibration activation matrix in R^{b x m}, W is the frozen weight matrix in R^{m x n}, and Y=XW is the layer output. If g_Y in R^{b x n} is the loss gradient with respect to Y, the exact weight gradient is g_W=X^T g_Y. With the thin SVD X=U D V^T, where singular values in D are descending, this becomes g_W=V D U^T g_Y. Retaining U_r=U[:,1:r], V_r=V[:,1:r], and D_r=D[1:r,1:r] gives g_W approximately V_r D_r U_r^T g_Y. The residual satisfies ||g_W-V_r V_r^T g_W||_F \u003c= sigma_{r+1}(X)||g_Y||_F, where sigma_{r+1}(X) is the first discarded singular value. Parameterize the trainable perturbation as Delta W=V_r A with A in R^{r x n}; this makes the zeroth-order search dimension d=r n instead of m n. For a scalar minibatch loss F(A), use the antithetic Gaussian estimator g_hat(A)=(F(A+mu u)-F(A-mu u))/(2 mu) u, with u sampled from N(0,I_d), and update an Adam state using g_hat. The finite-difference bias decreases with mu under smoothness, while the estimator variance scales with the search dimension d, so reducing r directly reduces the dimension-dependent variance term.",
 "math_tags": [
  "linear-algebra",
  "optimization",
  "approximation-theory",
  "statistics"
 ],
 "ml_areas": [
  "optimizer",
  "fine-tuning",
  "quantization",
  "transformer"
 ],
 "paper": {
  "arxiv_id": "2607.01125",
  "arxiv_url": "https://arxiv.org/abs/2607.01125",
  "summary_what_math_gives_to_ml": "The paper provides a concrete way to make zeroth-order fine-tuning practical by replacing full weight perturbations with perturbations in activation-derived low-rank coordinates. The transferable asset is the exact factorization of a linear-layer gradient through the input activation SVD: only the right singular directions of the activation matrix can receive gradient signal, and rapidly decaying singular values provide an explicit approximation-error control. This yields a forward-only adapter whose zeroth-order dimension is reduced from the number of weight parameters to the number of retained activation directions times the output width, while leaving quantized base weights frozen. The most promising implementation is a per-layer activation-SVD ZO adapter combined with Adam-style updates and a rank-selection rule based on the measured singular-value tail.",
  "title": "ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 8
 },
 "solves": [
  "memory",
  "speedup",
  "stability"
 ],
 "title": "Activation-SVD Zeroth-Order Adapters",
 "url": "https://synthcore.org/idea/322/activation-svd-zeroth-order-adapters",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}