{
 "artifacts": null,
 "category": "regularization",
 "description": "Replace parameter-count or validation-loss-only model selection with a singular-complexity score based on the evidence scaling of each candidate neural network. Select or prune architectures using \\(n\\widehat L_n+\\widehat\\lambda\\log n-(\\widehat m-1)\\log\\log n\\), which can prefer overparameterized but highly redundant networks when their effective singular complexity is lower.",
 "formulas_latex": [
  "$$Z_n=\\int p(X^n\\mid w)\\,\\varphi(w)\\,dw=\\int \\exp\\{-nL_n(w)\\}\\,\\varphi(w)\\,dw,$$",
  "$$-\\log Z_n=nL_n(w_0)+\\lambda\\log n-(m-1)\\log\\log n+O_{\\mathbb P}(1),$$",
  "$$-\\log M(X^n)\\ \\leq\\ nL_n(w_0)+\\lambda\\log n-(m-1)\\log\\log n+O_{\\mathbb P}(1),$$",
  "$$\\operatorname{Score}(\\mathcal A;n)=n\\widehat L_n(\\mathcal A)+\\widehat\\lambda_{\\mathcal A}\\log n-(\\widehat m_{\\mathcal A}-1)\\log\\log n.$$"
 ],
 "id": 3019,
 "implementation": "(1) Integration point: apply this after training several candidate architectures, or periodically during neural architecture search. Candidates can be Transformer widths, low-rank attention ranks, MLP expansion ratios, MoE expert counts, or structured-pruning masks. Replace the usual selection criterion of validation loss plus parameter-count penalty with the evidence-based score. (2) For each architecture \\(\\mathcal A\\), choose a prior \\(p_{\\mathcal A}(\\theta)\\), train an ensemble of \\(S\\) posterior approximations \\(q_{\\mathcal A}(\\theta)\\) using independent fine-tuning runs, SWAG, or a diagonal Gaussian around trained weights. For dataset prefixes of sizes \\(n_j\\), estimate evidence with self-normalized importance sampling: \\(\\log\\widehat Z_{\\mathcal A}(n_j)=\\log[(1/S)\\sum_s\\exp(\\log p_{\\mathcal A}(X^{n_j}\\mid\\theta_s)+\\log p_{\\mathcal A}(\\theta_s)-\\log q_{\\mathcal A}(\\theta_s))]\\). Fit \\(y_j=-\\log\\widehat Z_{\\mathcal A}(n_j)-n_j\\widehat L_{\\mathcal A}(n_j)=a+\\widehat\\lambda_{\\mathcal A}\\log n_j+b\\log\\log n_j\\), with \\(b=-(\\widehat m_{\\mathcal A}-1)\\), constraining \\(\\widehat\\lambda\\geq0\\) and \\(\\widehat m\\geq1\\), or use a one-parameter slope if the fit is noisy. Compute the displayed score and select the lowest value. (3) Quantities computed from the paper's mathematics are the evidence integral and its \\(\\log n\\), \\(\\log\\log n\\) asymptotic terms. Quantities estimated empirically are the posterior approximation, finite-sample evidence, optimum loss, and regression uncertainty; use bootstrap resampling over posterior particles and data prefixes for confidence intervals. Reject the correction if the fitted slope has poor fit or large uncertainty. (4) First experiment: train 4-layer and 8-layer small Transformers with widths 128, 256, and 512, plus LoRA rank variants, on WikiText-2. Compare validation-NLL selection, parameter-count BIC/MDL, and the proposed score using 4--6 prefix sizes and 32--128 posterior particles. Measure test NLL, parameter count, and inference memory. Success means selecting a model with lower test NLL than parameter-count BIC at equal or lower memory, or identifying a wider redundant model with improved test NLL despite similar or smaller estimated \\(\\widehat\\lambda\\).",
 "math_summary": "For a Bayesian model with parameter \\(w\\), prior density \\(\\varphi(w)\\), data \\(X^n=(X_1,\\ldots,X_n)\\), likelihood \\(p(X^n\\mid w)\\), and empirical loss \\(L_n(w)=-n^{-1}\\log p(X^n\\mid w)\\), the evidence is \\(Z_n=\\int p(X^n\\mid w)\\varphi(w)\\,dw=\\int e^{-nL_n(w)}\\varphi(w)\\,dw\\). The paper's Solomonoff/evidence construction implies that a universal semimeasure can match this evidence within a multiplicative constant, so code lengths differ by only \\(O(1)\\). Under Watanabe-style singular-learning assumptions, if \\(w_0\\) minimizes the population loss, the free-energy asymptotic is \\(-\\log Z_n=nL_n(w_0)+\\lambda\\log n-(m-1)\\log\\log n+O_{\\mathbb P}(1)\\), and the paper states the corresponding universal-code upper bound \\(-\\log M(X^n)\\le nL_n(w_0)+\\lambda\\log n-(m-1)\\log\\log n+O_{\\mathbb P}(1)\\). Here \\(\\lambda\\ge 0\\) is the learning coefficient measuring local singular complexity around the neural model's optimum, and \\(m\\in\\mathbb N\\) is its multiplicity. We estimate \\(\\lambda\\) and \\(m\\) from evidence values at several dataset sizes and use the resulting asymptotic score for architecture selection.",
 "math_tags": [
  "statistics",
  "information-theory",
  "algebraic-geometry"
 ],
 "ml_areas": [
  "architecture",
  "regularization",
  "moe-routing",
  "pruning"
 ],
 "paper": {
  "arxiv_id": "2609.01666",
  "arxiv_url": "https://arxiv.org/abs/2609.01666",
  "summary_what_math_gives_to_ml": "The paper gives a constructive bridge between universal algorithmic priors and Bayesian model evidence: a monotone-machine semimeasure can reproduce the evidence of any computable model up to a constant factor. The transferable asset is the singular-learning asymptotic, in which the complexity penalty is not parameter count but the learning coefficient \\(\\lambda\\) and multiplicity \\(m\\), invariants of the model's local geometry near optimal parameters. This suggests a neural architecture-selection and pruning criterion that measures how quickly each candidate's marginal likelihood grows with dataset size, favoring models with low singular complexity rather than merely few weights. The practical route is to estimate evidence for a small set of candidate networks using posterior samples, fit the \\(\\log n\\) scaling, and use the estimated coefficient to select widths, ranks, or experts.",
  "title": "Solomonoff Induction and Singular Integrals",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 7,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "generalization",
  "accuracy",
  "memory"
 ],
 "title": "Learning-Coefficient Architecture Selection",
 "url": "https://synthcore.org/idea/3019/learning-coefficient-architecture-selection",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}