Heavy-Tail Path-Adaptive Optimizer Pool

1. Integration point: apply this to a parameter-efficient fine-tuning module rather than an entire billion-parameter model. Use a frozen backbone plus one LoRA adapter or final classification head, and create K=5 or 6 copies of only that trainable parameter tensor. Set H_k=2^k optimizer steps, for example 16, 32, 64, 128, 256, and 512; each expert resets its AdaGrad accumulator and adapter parameters at the end of its block, either from the current weighted model or from a fresh checkpoint. At each minibatch, compute the shared backbone activations and per-expert adapter outputs, obtain gradient g_t^(k), and evaluate each expert on a small held-out minibatch to obtain scalar loss ellhat_k,t. 2. Pseudocode: initialize q_k=1/K, v_k=0, accumulator A_k=epsilon, and adapter theta_k. For each step t, compute g_k=grad(loss_train(theta_k)); A_k <- A_k + g_k squared; theta_k <- project(theta_k - alpha*g_k/sqrt(A_k)); compute ellhat_k on validation data; bar <- sum_k q_k*ellhat_k; z_k <- ellhat_k-bar; v_k <- v_k+z_k^2; q_k <- normalize(q_k*exp(-beta*z_k/sqrt(epsilon+v_k))); if expert age equals H_k, restart it and reset A_k and v_k. Produce the deployed prediction using the highest-q expert, or q-weighted logits for a cheaper ensemble. 3. Computed from the paper's mathematics: geometric block lengths, AdaGrad normalization, pathwise centered meta-losses, and parameter-free operation with respect to unknown drift and noise. Estimated empirically: the effective path length is not needed by the algorithm; monitor it afterward as sum_t ||theta_bar_t-theta_bar_(t-1)||, and estimate heavy-tail behavior from minibatch gradient norms using quantiles or a log-log tail plot. Use gradient accumulation or one shared backward pass where possible to reduce the K-fold cost. 4. First experiment: fine-tune a 2-layer MLP or ViT-small final head on CIFAR-10 with a distribution shift every 2,000 examples, comparing AdamW, AdaGrad, cosine-scheduled SGD, and the expert pool at equal total backward FLOPs. Add synthetic heavy-tailed noise g <- g + sigma*StudentT_p noise with p=1.3 and p=2.0. Measure cumulative validation loss, worst-window loss after shifts, final accuracy, gradient-spike frequency, and wall-clock overhead. Success is faster recovery after shifts, fewer catastrophic loss spikes, and higher accuracy than AdamW at equal training compute; under stationary Gaussian noise, the pool should not lose more than 1% accuracy.