{
 "artifacts": null,
 "category": "dynamics",
 "description": "Replace a fixed optimizer memory order with a nested family of gradient-integral controllers. Training begins with a first-order update and activates additional accumulated-gradient states only after an exponentially smoothed residual fails to decrease for several decision intervals; newly activated gains are ramped from zero, so the parameter update remains continuous and previously learned states are preserved. The optimizer should use little memory on easy problems and acquire longer memory on ill-conditioned or delayed-gradient problems.",
 "formulas_latex": [
  "$$\\dot z_1(t)=g(\\theta(t)),\\qquad \\dot z_{r+1}(t)=z_r(t),\\quad r=1,\\ldots,p-1$$",
  "$$\\dot\\theta(t)=-a_0g(\\theta(t))-\\sum_{r=1}^{p}a_rz_r(t),\\qquad P_{p,q}(\\lambda)=\\lambda^{p+1}+a_0q\\lambda^p+\\sum_{r=1}^{p}a_rq\\lambda^{p-r}$$",
  "$$a_l^2\u003e4a_{l-1}a_{l+1}\\quad(l=1,\\ldots,p-1),\\qquad a_l\u003e0$$",
  "$$r_k=\\beta r_{k-1}+(1-\\beta)\\|g_k\\|_2,\\qquad p_{k+1}=p_k+1\\ \\text{if}\\ r_{k-j}\u003e\\rho r_{k-j-1}\\ \\forall j=0,\\ldots,K-1$$"
 ],
 "id": 2988,
 "implementation": "1. Integration point: implement this as a PyTorch optimizer around any MLP, CNN, Transformer, or language-model training loop. Start with order \\(p=0\\), meaning ordinary gradient descent or AdamW without changing the network architecture. For each parameter tensor, maintain optional states \\(z_1,z_2,\\ldots,z_{p_{\\max}}\\); use a global order decision initially, then test layerwise order as a follow-up. 2. Pseudocode: compute gradients \\(g\\); update \\(r\\leftarrow\\beta r+(1-\\beta)\\|g\\|\\); if the contraction test fails for \\(K\\) checks, increment \\(p\\); initialize the new state \\(z_{p+1}=0\\), set its activation \\(\\gamma=0\\), and ramp \\(\\gamma\\leftarrow\\min(1,\\gamma+1/H)\\) over \\(H\\) optimizer steps; update \\(z_1\\leftarrow z_1+\\eta g\\), \\(z_{r+1}\\leftarrow z_{r+1}+\\eta z_r\\), and \\(\\theta\\leftarrow\\theta-\\eta[a_0g+\\sum_r\\gamma_ra_rz_r]\\). Apply norm clipping to every state to prevent integral windup. 3. Computed from the paper: nested order expansion, residual-triggered finite decision intervals, continuous transitions, and polynomial gain constraints. Estimated empirically: gradient contraction \\(r_k/r_{k-1}\\), Hessian curvature range, and the discrete-time stability boundary. For a local linear model, estimate dominant Jacobian eigenvalues by Lanczos and compare the measured Euler stability limit \\(\\eta_c\\) with \\(\\eta_c=\\min_j[-2\\operatorname{Re}(\\lambda_j)/|\\lambda_j|^2]\\). 4. First cheap experiment: train a 2-layer MLP on MNIST and a small Transformer on Penn Treebank, comparing SGD, fixed momentum, fixed order-1 integral memory, and adaptive orders \\(0\\rightarrow1\\rightarrow2\\). Use \\(\\beta=0.95\\), \\(\\rho=0.98\\), \\(K=20\\), \\(H=50\\), and coefficients satisfying the displayed inequality. The primary falsifiable signature is a plateau in \\(r_k\\) followed by an order transition, after which residual decay accelerates without a discontinuous loss spike. On quadratic regression, the measured divergence boundary should agree with the Jacobian prediction within 20%; on easy well-conditioned data, the adaptive method should remain at \\(p=0\\) for most training, while on ill-conditioned data it should activate higher order and produce faster late-time residual decay.",
 "math_summary": "For parameters \\(\\theta\\in\\mathbb{R}^d\\), gradient \\(g(\\theta)=\\nabla L(\\theta)\\), and order \\(p\\), maintain integral-chain states \\(z_1,\\ldots,z_p\\in\\mathbb{R}^d\\) with \\(\\dot z_1=g\\) and \\(\\dot z_{r+1}=z_r\\). The continuous optimizer is \\(\\dot\\theta=-a_0g-\\sum_{r=1}^{p}a_rz_r\\). Around a quadratic mode \\(L(\\theta)=\\frac12q\\theta^2\\), where \\(q\u003e0\\), each eigenmode has characteristic polynomial \\(P_{p,q}(\\lambda)=\\lambda^{p+1}+a_0q\\lambda^p+a_1q\\lambda^{p-1}+\\cdots+a_pq\\). Choose positive coefficients satisfying the paper's adjacent log-concavity condition \\(a_l^2\u003e4a_{l-1}a_{l+1}\\); this is a conservative design rule for separated, nonoscillatory roots. Candidate orders are nested because order \\(p+1\\) retains \\(z_1,\\ldots,z_p\\) and adds only \\(z_{p+1}\\). Let \\(r_k\\) be an exponential moving average of gradient norm or prediction residual. If \\(r_k\\) fails to contract below a threshold for \\(K\\) decision intervals, increase \\(p\\). A smooth activation \\(\\gamma_p(t)\\in[0,1]\\) multiplies the newly added gain and prevents discontinuous updates.",
 "math_tags": [
  "control-theory",
  "dynamical-systems",
  "linear-algebra",
  "numerical-analysis"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics",
  "scheduler"
 ],
 "paper": {
  "arxiv_id": "2609.00688",
  "arxiv_url": "https://arxiv.org/abs/2609.00688",
  "summary_what_math_gives_to_ml": "The paper offers a constructive order-adaptation mechanism for distributed integral control: begin with a low-order controller, add integral states only when measurable residual errors demonstrate that the current order is insufficient, and embed all controllers in a nested state/gain structure so transitions preserve existing memory and keep the control signal continuous. Its strongest transferable asset is the combination of residual-triggered memory expansion, gain conditions shared by all candidate orders, and a finite-time rejection guarantee for inadequate orders. A neural-network analogue is an adaptive-order gradient optimizer that starts with plain gradient descent, activates increasingly long-lived integral or momentum states when the loss or gradient residual stops contracting, and uses polynomially stable gains plus smooth gain ramps during activation.",
  "title": "Order-Adaptive Distributed Integral Control",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 7,
  "usefulness": 8
 },
 "solves": [
  "stability",
  "speedup",
  "sample-efficiency"
 ],
 "title": "Order-Adaptive Integral Optimizer",
 "url": "https://synthcore.org/idea/2988/order-adaptive-integral-optimizer",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}