# KL-Calibrated Historical Replay

- ID: 2167
- Canonical URL: https://synthcore.org/idea/2167/kl-calibrated-historical-replay
- API JSON: https://synthcore.org/api/idea/2167.json
- API Markdown: https://synthcore.org/api/idea/2167.md
- Verification status: unverified
- Source: [arXiv:2608.12159](https://arxiv.org/abs/2608.12159)
- Category: training
- Solves: accuracy, sample-efficiency, stability
- ML areas: fine-tuning, training-dynamics, curriculum
- Math tags: information-theory, statistics, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Use the paper's optimal power-prior exponent to determine how much source data, old-task data, or replay data should influence neural-network fine-tuning. Estimate the predictive KL divergence between the current and historical distributions on a small target validation stream, then set the replay loss coefficient from the closed-form rule instead of tuning it by grid search.

## Mathematical statement

The power prior is \(\pi(\theta\mid\mathcal D_0,a_0)\propto L(\theta\mid\mathcal D_0)^{a_0}\pi_0(\theta)\), where \(\theta\) is the model parameter, \(\mathcal D_0\) is historical data, \(L\) is its likelihood, and \(a_0\in[0,1]\) is the historical-data discount. For a model with \(d\) parameters, historical sample size \(N_0\), and average KL divergence \(\bar D_0\) between historical and current data-generating distributions, the paper gives \(a_0^*=d/(2N_0\bar D_0+d)\). The equivalent borrowed effective sample size is \(E^*=a_0^*N_0\), satisfying \(1/E^*=1/N_0+2\bar D_0/d\); mismatch therefore caps useful borrowing at \(d/(2\bar D_0)\). For a neural network, use \(d\) as the number of trainable parameters in the adapted block, or test a Fisher effective dimension instead of counting the frozen backbone. Estimate mismatch from predictive distributions: for target inputs \(x_j\), current predictions \(p_t(y\mid x_j)\), and historical-teacher predictions \(p_0(y\mid x_j)\), use \(\widehat{\bar D}_0=M^{-1}\sum_j\sum_y p_t(y\mid x_j)\log[(p_t(y\mid x_j)+\epsilon)/(p_0(y\mid x_j)+\epsilon)]\). The adapted loss is \(\mathcal L=\mathcal L_{\mathrm{current}}+\lambda_0\mathcal L_{\mathrm{historical}}\), with \(\lambda_0=a_0^*\), or replay sampling proportional to \(E^*/N_0\).

## Key formulas

- $$\pi(\theta\mid\mathcal D_0,a_0)\propto L(\theta\mid\mathcal D_0)^{a_0}\,\pi_0(\theta),\qquad a_0\in[0,1].$$
- $$a_0^*=\frac{d}{2N_0\bar D_0+d},\qquad E^*=a_0^*N_0=\frac{dN_0}{2N_0\bar D_0+d}.$$
- $$\frac{1}{E^*}=\frac{1}{N_0}+\frac{2\bar D_0}{d},\qquad E^*\leq\frac{d}{2\bar D_0}\quad\text{for }\bar D_0>0.$$
- $$\widehat{\bar D}_0=\frac{1}{M}\sum_{j=1}^{M}\sum_y p_t(y\mid x_j)\log\frac{p_t(y\mid x_j)+\epsilon}{p_0(y\mid x_j)+\epsilon},\qquad \lambda_0=\frac{d}{2N_0\widehat{\bar D}_0+d}.$$

## Implementation notes

(1) Integration point: modify the loss mixer in supervised continual learning or domain-adaptive fine-tuning. Keep a replay buffer containing historical examples and compute separate minibatch losses for current examples and replay examples. Initially adapt only a linear classification head or a small LoRA block, so that \(d\), the number of trainable parameters, is meaningful and does not collapse the discount for a billion-parameter frozen backbone. (2) Pseudocode: save a frozen historical teacher producing \(p_0(y\mid x)\). At the start of each epoch, draw \(M\) target inputs \(x_j\), compute student probabilities \(p_t\) and teacher probabilities \(p_0\), then calculate \(D=\mathrm{mean}_j\sum_y p_t[y]\log((p_t[y]+\epsilon)/(p_0[y]+\epsilon))\). Set \(a=d/(2N_0D+d)\), clip \(a\) to \([0,1]\), and optionally smooth it using \(a\leftarrow\beta a_{\mathrm{old}}+(1-\beta)a\). On every training step use \(L=L_{\mathrm{current}}+aL_{\mathrm{replay}}\), or choose a replay batch size proportional to \(aN_0\). (3) Computed from the paper: the discount formula and effective sample size. Estimated empirically: \(D\) from target-input predictions; \(d\) is exact for the adapted block, while a Fisher effective dimension can be evaluated as an ablation. If teacher and student distributions are poorly calibrated, estimate mismatch with held-out target cross-entropy differences or temperature-calibrated probabilities. (4) First experiment: use sequential CIFAR-100 task groups or CIFAR-10 versus CIFAR-10-C, with a ResNet-18 whose head or LoRA adapter is trainable. Compare no replay, full replay, fixed coefficients \(0.1,0.5,1\), and the calibrated rule. Measure target accuracy, old-task accuracy, forgetting, replay examples consumed, and coefficient trajectories. Success means the coefficient stays near one for compatible streams, falls under severe shift, and matches or exceeds fixed-weight accuracy with less replay computation.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.