# Memory-Light Differentiable Learned Optimizer

- ID: 2001
- Canonical URL: https://synthcore.org/idea/2001/memory-light-differentiable-learned-optimizer
- API JSON: https://synthcore.org/api/idea/2001.json
- API Markdown: https://synthcore.org/api/idea/2001.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.08559](https://arxiv.org/abs/2608.08559)
- Category: optimizer
- Solves: memory, speedup, accuracy
- ML areas: optimizer, fine-tuning, training, memory
- Math tags: optimization, linear-algebra, dynamical-systems, numerical-analysis
- Ratings: usefulness 7/10; difficulty 6/10; novelty 5/10

## Idea description

Construct a learned optimizer whose update is an ordered sequence of local implicit parameter-block solves, then differentiate the finite optimization trajectory with reverse local adjoints. This enables training optimizer hyperparameters or meta-gradients through many inner steps without storing all intermediate tensor operations or replacing the executed trajectory by an idealized fixed-point gradient.

## Mathematical statement

Let the inner optimizer state be partitioned into blocks \(w=(w_1,\ldots,w_m)\), with momentum or auxiliary state included in each block. At inner step \(k\), block \(i\) is defined by an implicit update residual \(r_{k,i}(w_{k,i};s_{k,i},\eta)=0\), where \(s_{k,i}\) contains the current gradient, neighboring parameter blocks, and auxiliary states, and \(\eta\) are learned optimizer parameters such as step sizes, preconditioners, or damping values. The local differential identity is \(A_{k,i}dw_{k,i}+B_{k,i}ds_{k,i}+C_{k,i}d\eta=0\), with \(A_{k,i}=\partial r_{k,i}/\partial w_{k,i}\), \(B_{k,i}=\partial r_{k,i}/\partial s_{k,i}\), and \(C_{k,i}=\partial r_{k,i}/\partial\eta\). Reverse differentiation of the executed inner trajectory solves \(A_{k,i}^{\top}\lambda_{k,i}=\bar w_{k,i}\) in reverse time and reverse block order, then propagates \(-B_{k,i}^{\top}\lambda_{k,i}\) and \(-C_{k,i}^{\top}\lambda_{k,i}\). The construction is a transpose of the finite solver map, not the adjoint of the converged optimization equation, so it remains correct when the inner optimizer uses only a small fixed number of sweeps.

## Key formulas

- $$r_{k,i}(w_{k,i};s_{k,i},\eta)=0,$$
- $$A_{k,i}\,dw_{k,i}+B_{k,i}\,ds_{k,i}+C_{k,i}\,d\eta=0,\qquad A_{k,i}=\frac{\partial r_{k,i}}{\partial w_{k,i}},\quad B_{k,i}=\frac{\partial r_{k,i}}{\partial s_{k,i}},\quad C_{k,i}=\frac{\partial r_{k,i}}{\partial\eta},$$
- $$A_{k,i}^{\top}\lambda_{k,i}=\bar w_{k,i},\qquad \bar s_{k,i}\mathrel{+}= -B_{k,i}^{\top}\lambda_{k,i},\qquad \bar\eta\mathrel{+}= -C_{k,i}^{\top}\lambda_{k,i}.$$
- $$\frac{\partial L_{\mathrm{outer}}}{\partial\eta}=\sum_{k=K}^{1}\sum_{i=m}^{1}-C_{k,i}^{\top}\lambda_{k,i},\qquad A_{k,i}^{\top}\lambda_{k,i}=\bar w_{k,i}.$$

## Implementation notes

Use this inside a meta-learning or differentiable optimization benchmark. Partition model parameters into layer blocks, for example one block per Transformer layer or one block per convolutional stage, and define an implicit damped update for block \(i\): \(r_{k,i}(w_{k,i};s_{k,i},\eta)=w_{k,i}-w_{k-1,i}+\alpha_i g_{k,i}(w_{k,i})+D_i^{-1}(w_{k,i}-w_{k-1,i})\), where \(g_{k,i}\) is the block gradient, \(\alpha_i>0\) is a learned step size, and \(D_i\) is a positive diagonal damping vector. Solve each block with one or two Newton or fixed-point iterations, while treating the resulting block value as the output of the local implicit solve. During forward, run \(K\) inner steps and save block outputs, gradients, and diagonal or factored local Jacobians. During backward from the outer validation loss, traverse \(k=K\) to 1 and blocks in reverse order. Form the local VJP for \(A_{k,i}\), solve the transposed system; with diagonal damping this is elementwise division, while a small dense layer block can use batched LU. Propagate cotangents into earlier parameters and accumulate meta-gradients for \(\eta=(\alpha,D)\) using the displayed \(-C^{\top}\lambda\) term. Compare against unrolled autodiff through Adam or SGD on few-shot MNIST or CIFAR-10 with a 20-step MAML inner loop. Measure peak memory, wall-clock backward time, meta-gradient cosine similarity to unrolling, and outer validation accuracy. The target is 5--20x lower activation memory at equal inner steps, gradient cosine similarity above 0.99, and no degradation in final validation accuracy.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built and numerically verified a diagonal implicit block solver with reverse local adjoints, plus an autodiff-unrolled reference. The contraction law q=(1+d)/(1+d+alpha*lambda) matched exactly across the sweep, implicit stability persisted beyond the explicit Euler boundary x=2, and finite-horizon gradients matched the predicted K*q^(2K-1) scaling and autodiff to approximately 1.8e-15 relative error. In the vector toy, the local reverse pass was 13.9x faster and used an estimated 10x smaller saved-state footprint, but this is not evidence of a neural-network-level win because the footprint is estimated and the task is diagonal quadratic.

### Mechanism check

- Verdict: Built and numerically verified a diagonal implicit block solver with reverse local adjoints, plus an autodiff-unrolled reference. The contraction law q=(1+d)/(1+d+alpha*lambda) matched exactly across the sweep, implicit stability persisted beyond the explicit Euler boundary x=2, and finite-horizon gradients matched the predicted K*q^(2K-1) scaling and autodiff to approximately 1.8e-15 relative error. In the vector toy, the local reverse pass was 13.9x faster and used an estimated 10x smaller saved-state footprint, but this is not evidence of a neural-network-level win because the footprint is estimated and the task is diagonal quadratic.
- Confidence: 8/10
- Limitations: No MNIST/CIFAR or neural-network meta-learning benchmark was tested; no actual peak-memory profiler was used, and the local-adjoint implementation covers a scalar/diagonal quadratic block system without neighboring blocks, momentum, Newton iterations, CUDA fallback, or learned optimizer training.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/761/bench_report.json)
- [experiment.py](https://synthcore.org/code/761/experiment.py)
- [report.md](https://synthcore.org/code/761/report.md)
- [report_bench_2026-09-01T024152.md](https://synthcore.org/code/761/report_bench_2026-09-01T024152.md)
- [results.json](https://synthcore.org/code/761/results.json)
- [stage2_bench.py](https://synthcore.org/code/761/stage2_bench.py)
- [verify.py](https://synthcore.org/code/761/verify.py)
- [Download all files as ZIP](https://synthcore.org/download/761)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.