{
 "artifacts": null,
 "category": "regularization",
 "description": "Add a local Jacobian spectral regularizer and an initialization sweep to steer a looped transformer away from uncontrolled near-unit dynamics. The goal is to prevent examples from entering a fold-critical regime with very long relaxation times, or alternatively to deliberately target a controlled critical regime when adaptive test-time compute is useful.",
 "formulas_latex": [
  "$$z_{t+1}=F_{\\theta}(z_t;h)$$",
  "$$\\rho(J)=\\max_i|\\lambda_i(J)|,\\qquad J=\\frac{\\partial F_{\\theta}(z_t;h)}{\\partial z_t}$$",
  "$$\\mathcal{L}_{\\mathrm{total}}=\\mathcal{L}_{\\mathrm{task}}+\\beta\\,\\big[\\max(0,\\hat\\rho_t-\\rho_*)\\big]^2$$",
  "$$\\tau(\\varepsilon)\\,[1-\\lambda_{\\max}(-\\varepsilon)]\\to\\pi$$"
 ],
 "id": 2749,
 "implementation": "Modify training of a weight-tied recurrent transformer by adding a Jacobian-spectrum penalty at randomly selected unroll states. For a minibatch input, unroll the tied block for \\(T\\) steps, select one or two states \\(z_t\\), and estimate the dominant Jacobian magnitude with three power iterations: sample normalized \\(v\\), compute \\(u=J_t v\\) by a Jacobian-vector product, update \\(v\\leftarrow u/(\\|u\\|_2+10^{-8})\\), and record \\(\\hat\\rho_t=\\|J_t v\\|_2\\). Add \\(\\beta[\\max(0,\\hat\\rho_t-\\rho_*)]^2\\) to the task loss, where \\(\\rho_*\\), for example 0.90 or 0.95, is selected from a validation sweep. To avoid over-regularizing useful representation directions, apply the penalty only after a warmup period and average it over tokens or examples. Run at least eight seeds with different block initializations, because the paper predicts phase selection is seed-sensitive; log residual decay, \\(\\hat\\rho_t\\), iteration count, and signs of oscillation. A second condition can use a narrow target penalty \\(\\beta(\\hat\\rho_t-\\rho_*)^2\\) with \\(\\rho_*=0.98\\) to test deliberate near-critical compute scaling. The Jacobian estimate and unit-circle spectral interpretation come from the paper's dynamics; \\(\\rho_*\\), penalty weight, unroll length, and estimator frequency are empirical. First evaluate on a small looped transformer and modular arithmetic or sequence-copying tasks against unregularized training at equal parameter count and maximum depth. The primary success signal is reduced variance of inference iterations and fewer divergence or oscillation failures at equal accuracy; a secondary signal is improved accuracy per average recurrent block evaluation.",
 "math_summary": "The paper's central recurrence is \\(z_{t+1}=F_\\theta(z_t;h)\\), and it distinguishes dynamical phases by bifurcation mechanism, including a saddle-node fold and a Neimark-Sacker-type transition to bounded nonstationary motion. The fold phase exhibits critical slowing down governed by \\(\\tau(\\varepsilon)[1-\\lambda_{\\max}(-\\varepsilon)]\\to\\pi\\), where \\(\\tau\\) is relaxation time, \\(\\varepsilon\\) is distance from the fold, and \\(\\lambda_{\\max}\\) is the dominant linearized eigenvalue. A practical phase-control surrogate is the spectral radius \\(\\rho(J)=\\max_i|\\lambda_i(J)\\) of the local Jacobian \\(J=\\partial F_\\theta/\\partial z_t\\). Penalizing \\(\\rho(J)\\) above a target \\(\\rho_*\u003c1\\) enforces a margin from the unit circle and should suppress excessive relaxation times; a target near one can instead create compute-adaptive but potentially fragile dynamics. The paper also reports that identical objectives and accuracies can reach distinct phases depending strongly on initialization, motivating explicit multi-seed phase screening.",
 "math_tags": [
  "dynamical-systems",
  "bifurcations",
  "linear-algebra",
  "random-matrix"
 ],
 "ml_areas": [
  "transformer",
  "training-dynamics",
  "regularization",
  "initialization"
 ],
 "paper": {
  "arxiv_id": "2608.26556",
  "arxiv_url": "https://arxiv.org/abs/2608.26556",
  "summary_what_math_gives_to_ml": "The paper treats a weight-tied transformer as a discrete dynamical system and argues that inference cost is controlled by the bifurcation phase selected during training, not merely by architecture or final accuracy. The most transferable object is the local relaxation law near a saddle-node fold: the iteration time and spectral gap obey the parameter-free asymptotic relation \\(\\tau(\\varepsilon)[1-\\lambda_{\\max}(-\\varepsilon)]\\to\\pi\\). This suggests measuring local Jacobian criticality during inference to predict required iterations and implementing adaptive halting rather than using a fixed loop depth. A second practical direction is phase-aware training or initialization that penalizes undesirable near-unit Jacobian spectra, reducing unpredictable critical slowing down while preserving the option of deliberately targeting compute-adaptive behavior.",
  "title": "Dynamical phase selection controls compute scaling in looped transformers",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "inference-speedup",
  "generalization"
 ],
 "title": "Phase-controlled Jacobian training",
 "url": "https://synthcore.org/idea/2749/phase-controlled-jacobian-training",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}