# Fisher-Width-Balanced Sparse Fine-Tuning

- ID: 1366
- Canonical URL: https://synthcore.org/idea/1366/fisher-width-balanced-sparse-fine-tuning
- API JSON: https://synthcore.org/api/idea/1366.json
- API Markdown: https://synthcore.org/api/idea/1366.md
- Verification status: unverified
- Source: [arXiv:2607.20578](https://arxiv.org/abs/2607.20578)
- Category: geometry
- Solves: stability, sample-efficiency, memory
- ML areas: fine-tuning, regularization, optimizer
- Math tags: geometry, probability, statistics, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Select the coordinates of a sparse adapter or sparse fine-tuning mask using both Fisher width and inverse-Fisher width. The mask should avoid parameter subsets that are cheap in the Fisher geometry but extremely large in the inverse-Fisher geometry, or vice versa, thereby controlling both prediction sensitivity and estimator-like uncertainty.

## Mathematical statement

For a positive-definite Fisher information matrix G(    heta) and a parameter set T, the paper defines the primal Fisher width w_G(T)=w(G(    heta)^{1/2}T) and inverse-Fisher width w_{G^{-1}}(T)=w(G(    heta)^{-1/2}T). The Gaussian width is w(S)=\mathbb{E}_{z\sim\mathcal{N}(0,I)}[\sup_{u\in S}\langle z,u\rangle]. The paper proves w_G(T)w_{G^{-1}}(T)\geq w(T)^2, where w(T) is ordinary Euclidean Gaussian width. For a diagonal Fisher approximation G=\operatorname{diag}(f_1,\ldots,f_d) and the masked Euclidean ball T_A=\{\delta\in\mathbb{R}^d:\operatorname{supp}(\delta)\subseteq A,\|\delta\|_2\leq r\}, the widths become w_G(T_A)=r\mathbb{E}[\sqrt{\sum_{i\in A}f_i z_i^2}] and w_{G^{-1}}(T_A)=r\mathbb{E}[\sqrt{\sum_{i\in A}f_i^{-1}z_i^2}], with independent z_i\sim\mathcal{N}(0,1). The implementation uses damping f_i+\varepsilon for numerical stability and minimizes a normalized sum of logarithmic widths over masks of fixed cardinality.

## Key formulas

- $$w_G(T)=w\bigl(G(\theta)^{1/2}T\bigr),\qquad w_{G^{-1}}(T)=w\bigl(G(\theta)^{-1/2}T\bigr).$$
- $$w(S)=\mathbb{E}_{z\sim\mathcal{N}(0,I)}\left[\sup_{u\in S}\langle z,u\rangle\right].$$
- $$w_G(T)w_{G^{-1}}(T)\geq w(T)^2.$$
- $$\widehat w_G(A)=\frac{r}{M}\sum_{m=1}^{M}\sqrt{\sum_{i\in A}f_i z_{mi}^2},\qquad \widehat w_{G^{-1}}(A)=\frac{r}{M}\sum_{m=1}^{M}\sqrt{\sum_{i\in A}\frac{z_{mi}^2}{f_i+\varepsilon}}.$$

## Implementation notes

Integrate this into sparse supervised fine-tuning of a pretrained Transformer, using either individual trainable coordinates or, preferably for efficient kernels, whole weight blocks such as attention projections, MLP rows, or LoRA adapter rows. First freeze the pretrained model and estimate a diagonal empirical Fisher on a calibration set: for B examples compute f_i=(1/B)\sum_{b=1}^{B}(\partial\ell_b/\partial    heta_i)^2. Add damping \varepsilon, for example 10^{-8} times the mean positive f_i. For each candidate mask A, draw M fixed Gaussian probes z_m with independent standard-normal entries and compute q_G(A)=M^{-1}\sum_m\sqrt{\sum_{i\in A}f_i z_{mi}^2} and q_I(A)=M^{-1}\sum_m\sqrt{\sum_{i\in A}z_{mi}^2/(f_i+\varepsilon)}. Select exactly k coordinates or blocks by greedy forward selection. At each step, tentatively add each remaining candidate, evaluate J(A)=log(q_G(A)/c_G)+log(q_I(A)/c_I), and add the candidate with the lowest J subject to an optional upper bound on q_G; c_G and c_I are fixed normalization constants computed from a random k-mask. Maintain the sums for every Gaussian probe incrementally so mask search is cheap. The formulas and product inequality come from the paper; diagonal Fisher, damping, block grouping, Monte Carlo, and greedy optimization are empirical adaptations. Compare against random masks, magnitude selection, largest-Fisher selection, and standard LoRA at equal trainable parameter count on a small language-model adaptation task or GLUE. Record validation accuracy, loss, gradient-norm spikes, seed variance, and width estimates. Success requires improved accuracy or stability at fixed trainable memory, with the selected mask having a lower maximum normalized width than baselines.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.