# Cheap Averaged-Gradient Adam

- ID: 920
- Canonical URL: https://synthcore.org/idea/920/cheap-averaged-gradient-adam
- API JSON: https://synthcore.org/api/idea/920.json
- API Markdown: https://synthcore.org/api/idea/920.md
- Verification status: unverified
- Source: [arXiv:2607.14516](https://arxiv.org/abs/2607.14516)
- Category: regularization
- Solves: generalization, stability, speedup
- ML areas: optimizer, regularization, training-dynamics
- Math tags: numerical-analysis, optimization, dynamical-systems
- Ratings: usefulness 5/10; difficulty 4/10; novelty 5/10

## Idea description

Use a two-gradient predictor-corrector average as the gradient supplied to Adam, retaining trajectory smoothing while avoiding the three or four gradient evaluations required by full RK3. Vary the mixing coefficient to test whether the reported regularization comes from gradient averaging itself rather than from high-order integration.

## Mathematical statement

The RK construction evaluates gradients at nearby predicted parameters and combines them as \(g_3=\frac{2}{9}k_1+\frac{1}{3}k_2+\frac{4}{9}k_3\), where \(k_1=\nabla L(\theta)\), \(k_2=\nabla L(\theta-hk_1/2)\), and \(k_3=\nabla L(\theta-3hk_2/4)\). The transferable structure is a convex combination of gradients sampled along a short trajectory. A cheap two-stage adaptation computes \(k_1=\nabla L(\theta)\), predicts \(\widetilde\theta=\theta-hk_1\), computes \(k_2=\nabla L(\widetilde\theta)\), and supplies \(\bar g=(1-\alpha)k_1+\alpha k_2\) to Adam, where \(0\leq\alpha\leq1\). The Adam moments are \(m_t=\beta_1m_{t-1}+(1-\beta_1)\bar g_t\) and \(v_t=\beta_2v_{t-1}+(1-\beta_2)\bar g_t^2\); bias-corrected moments produce the final parameter update. The nearby-gradient average may suppress minibatch noise and alter implicit bias, while costing only two backward passes per optimizer step.

## Key formulas

- $$g_{3}=\tfrac{2}{9}k_{1}+\tfrac{1}{3}k_{2}+\tfrac{4}{9}k_{3},\qquad k_1=\nabla L(\theta),\quad k_2=\nabla L(\theta-\tfrac{h}{2}k_1),\quad k_3=\nabla L(\theta-\tfrac{3h}{4}k_2)$$
- $$\widetilde\theta_t=\theta_t-hk_{1,t},\qquad \bar g_t=(1-\alpha)k_{1,t}+\alpha\nabla L(\widetilde\theta_t),\quad 0\leq\alpha\leq1$$
- $$m_t=\beta_1m_{t-1}+(1-\beta_1)\bar g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)\bar g_t^{\,2},\quad \theta_{t+1}=\theta_t-h\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}$$

## Implementation notes

(1) Modify only the optimizer's gradient input. Retain the current parameter tensor \(\theta\), calculate the ordinary minibatch gradient \(k_1\), create a temporary prediction \(\widetilde\theta=\theta-hk_1\), run a second forward/backward pass on the same minibatch at \(\widetilde\theta\), and set \(\bar g=(1-\alpha)k_1+\alpha k_2\). Do not commit \(\widetilde\theta\); it is only a probe. (2) Pseudocode: \(k_1=\text{backward}(L(\theta))\); copy parameters; set temporary parameters to \(\theta-hk_1\); compute \(k_2=\text{backward}(L(\widetilde\theta))\); restore \(\theta\); form \(\bar g=(1-\alpha)k_1+\alpha k_2\); update Adam moments and parameters using \(\bar g\). Test \(\alpha\in\{0,0.25,0.5,0.75,1\}\). A variant may use the Adam-normalized direction for the temporary prediction while keeping the displayed averaging rule. (3) The combination and moment equations are fixed; learning rate, \(\alpha\), same-batch versus fresh-batch probing, and weight decay are empirical choices. Log gradient cosine similarity, gradient norms, across-batch variance, wall-clock time, and backward evaluations. (4) First run CIFAR-10 with ResNet-18 for a fixed backward-pass budget, comparing Adam, AdamW, RMSprop, NAdam, and the averaged-gradient method at \(\alpha=0.5\), using at least five seeds. Success is higher test accuracy or lower validation loss at equal backward evaluations, with comparable training loss and preferably reduced gradient variance. If it only matches RMSprop or NAdam while costing twice as much, the method should be rejected as noncompetitive.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.