Residual-Histogram Block Coordinate Fine-Tuning

Integrate this at the optimizer step of a model with separable trainable blocks, such as LoRA adapters grouped by Transformer layer, expert groups in an MoE model, or partitions of a large embedding table. Freeze the base model initially so the scheduling effect is isolated. After each minibatch forward and backward pass, compute per-example losses ell_j and per-block gradient magnitudes g_i. Set delta_j = 1 when ell_j exceeds threshold tau, or alternatively when the example produces a gradient above tau_g. For each trainable block i, aggregate associated active indicators into omega_i, clip omega_i to [0,A_max], quantize it, and form the normalized histogram for each parameter cluster. Feed all histograms, current loss, update age, and recent reward to a two-layer scheduler MLP. Select one cluster with epsilon-greedy exploration. Apply AdamW only to parameters in that cluster; retain parameters and optimizer moments for unselected blocks. Prevent starvation with an age penalty or a hard rule that every cluster must be updated at least once within K minibatches. Define reward as (L_t - L_{t+1}) divided by estimated FLOPs plus epsilon, minus lambda, with L_t an exponential moving average of minibatch loss. Compare against all-block AdamW, round-robin block updates, random selection, and top-gradient greedy selection. The first cheap experiment should fine-tune a 12-layer Transformer with LoRA on GLUE or a small language-model corpus using 4 to 8 adapter clusters. Measure validation loss versus optimizer FLOPs, wall-clock time, final accuracy, and the number of skipped updates per block. A positive result is equal validation accuracy with 25 to 40 percent fewer adapter updates, or faster loss descent at equal optimizer compute without oscillatory starvation.