# Singular-Mode Phase-Transition Regularization Curriculum

- ID: 1900
- Canonical URL: https://synthcore.org/idea/1900/singular-mode-phase-transition-regularization-curriculum
- API JSON: https://synthcore.org/api/idea/1900.json
- API Markdown: https://synthcore.org/api/idea/1900.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.06597](https://arxiv.org/abs/2608.06597)
- Category: dynamics
- Solves: stability, generalization, sample-efficiency
- ML areas: optimizer, scheduler, training-dynamics, regularization
- Math tags: dynamical-systems, bifurcations, spectral-theory, optimization
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Replace fixed weight decay with a spectrum-aware schedule that intentionally crosses predicted activation thresholds one at a time. The curriculum should first learn strong, well-conditioned input-output modes and only later lower regularization enough to activate weak modes, producing controlled rank growth instead of simultaneous fitting of noisy directions.

## Mathematical statement

Assume a depth-n linear network W = W(n) ... W(1), whitened inputs with covariance equal to the identity, and approximately balanced layers aligned with one singular mode. Let s be the end-to-end singular value for that mode, y be its target singular value, and beta be the per-layer Frobenius penalty coefficient. Minimizing over balanced factors reduces the mode objective to f(s) = (s - y)^2 + n beta s^(2/n). For n greater than 2, a nonzero stationary branch satisfies y = s + beta s^(-(n-2)/n). Comparing this branch with the zero solution gives the transition value s_c = ((n-2) beta)^(n/(2n-2)) and target threshold y_c = s_c + beta s_c^(-(n-2)/n). A mode is predicted to become active when y exceeds y_c. The reduction is exact only for the aligned model; in a general network, y and s are estimated from empirical singular spectra.

## Key formulas

- f(s) = (s - y)^2 + n beta s^(2/n), with s >= 0
- df/ds = 2(s - y) + 2 beta s^(2/n - 1) = 0, hence y = s + beta s^(-(n-2)/n)
- s_c = ((n - 2) beta)^(n/(2n - 2)), and y_c(beta,n) = s_c + beta s_c^(-(n-2)/n)
- A mode k is active when y_k > y_c(beta,n); equivalently beta_k_crit = (y_k / C_n)^((2n - 2)/n), where C_n = (n - 1)(n - 2)^(-(n - 2)/(2n - 2))

## Implementation notes

1. Integration point: use this as a scheduler for weight decay during training of an MLP, linear-probe network, low-rank transformer projection, or state-space input/output map. At initialization and then every K steps, estimate the whitened cross-covariance C = Sigma_yx Sigma_xx^(-1/2), where Sigma_yx is the minibatch output-input cross-covariance and Sigma_xx is the input covariance. Compute its singular values y_k. 2. Pseudocode: choose depth n and an initial beta above the critical value for weak modes; train for T optimizer steps; reduce beta geometrically; after every reduction compute end-to-end singular values s_k of the factorized network and the predicted threshold y_c(beta,n); hold beta until the observed active-mode count, defined as the number of s_k greater than epsilon, agrees with the predicted count of y_k greater than y_c; then continue. Use loss equal to task MSE plus beta times the sum of squared Frobenius norms, with ordinary SGD or Adam. 3. Computed from the paper: the reduced objective and critical formulas. Estimated empirically: covariance singular values, end-to-end singular values, and epsilon. 4. First cheap experiment: train depth-4 linear networks on whitened Gaussian data generated by a diagonal teacher with singular values [4, 2, 1, 0.5, 0.2]. Compare fixed beta with the curriculum. Sweep beta densely and measure active rank, validation MSE, and mode singular values. The falsifiable signature is a sharp rank transition near the predicted y_c(beta,4), with measured critical beta for each mode within 20 percent of beta_k_crit; activation should occur in descending order of teacher singular value.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible depth-4 singular-mode phase-transition MVP with analytic formulas, independent numerical minimization, beta/teacher sweeps, and a small linear-network curriculum comparison. The toy mechanism manifested strongly: observed threshold exponents and critical-beta exponents matched predictions almost exactly, boundary errors were below 0.4%, and rank-count agreement was 100%. The neural-network run did not produce rank growth or a meaningful practical win; both methods remained effectively rank-1, so the positive result is limited to the aligned reduced model.

### Mechanism check

- Verdict: Built a reproducible depth-4 singular-mode phase-transition MVP with analytic formulas, independent numerical minimization, beta/teacher sweeps, and a small linear-network curriculum comparison. The toy mechanism manifested strongly: observed threshold exponents and critical-beta exponents matched predictions almost exactly, boundary errors were below 0.4%, and rank-count agreement was 100%. The neural-network run did not produce rank growth or a meaningful practical win; both methods remained effectively rank-1, so the positive result is limited to the aligned reduced model.
- Confidence: 8/10
- Limitations: The practical network scheduler uses a fixed beta sequence rather than recomputing minibatch whitened cross-covariance and observed end-to-end spectra. Only a tiny noiseless diagonal-teacher linear network was tested; no MLP, noisy data, validation generalization, wall-clock comparison, or broader optimizer/schedule sweep was evaluated. The exact toy verification confirms the aligned mathematical mechanism, not general-network effectiveness.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.96975
- Paired wins: 2/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/705/bench_report.json)
- [experiment.py](https://synthcore.org/code/705/experiment.py)
- [report.md](https://synthcore.org/code/705/report.md)
- [report_bench_2026-09-02T225039.md](https://synthcore.org/code/705/report_bench_2026-09-02T225039.md)
- [results.json](https://synthcore.org/code/705/results.json)
- [stage2_bench.py](https://synthcore.org/code/705/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/705)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.