Learning-Coefficient Architecture Selection

(1) Integration point: apply this after training several candidate architectures, or periodically during neural architecture search. Candidates can be Transformer widths, low-rank attention ranks, MLP expansion ratios, MoE expert counts, or structured-pruning masks. Replace the usual selection criterion of validation loss plus parameter-count penalty with the evidence-based score. (2) For each architecture \(\mathcal A\), choose a prior \(p_{\mathcal A}(\theta)\), train an ensemble of \(S\) posterior approximations \(q_{\mathcal A}(\theta)\) using independent fine-tuning runs, SWAG, or a diagonal Gaussian around trained weights. For dataset prefixes of sizes \(n_j\), estimate evidence with self-normalized importance sampling: \(\log\widehat Z_{\mathcal A}(n_j)=\log[(1/S)\sum_s\exp(\log p_{\mathcal A}(X^{n_j}\mid\theta_s)+\log p_{\mathcal A}(\theta_s)-\log q_{\mathcal A}(\theta_s))]\). Fit \(y_j=-\log\widehat Z_{\mathcal A}(n_j)-n_j\widehat L_{\mathcal A}(n_j)=a+\widehat\lambda_{\mathcal A}\log n_j+b\log\log n_j\), with \(b=-(\widehat m_{\mathcal A}-1)\), constraining \(\widehat\lambda\geq0\) and \(\widehat m\geq1\), or use a one-parameter slope if the fit is noisy. Compute the displayed score and select the lowest value. (3) Quantities computed from the paper's mathematics are the evidence integral and its \(\log n\), \(\log\log n\) asymptotic terms. Quantities estimated empirically are the posterior approximation, finite-sample evidence, optimum loss, and regression uncertainty; use bootstrap resampling over posterior particles and data prefixes for confidence intervals. Reject the correction if the fitted slope has poor fit or large uncertainty. (4) First experiment: train 4-layer and 8-layer small Transformers with widths 128, 256, and 512, plus LoRA rank variants, on WikiText-2. Compare validation-NLL selection, parameter-count BIC/MDL, and the proposed score using 4--6 prefix sizes and 32--128 posterior particles. Measure test NLL, parameter count, and inference memory. Success means selecting a model with lower test NLL than parameter-count BIC at equal or lower memory, or identifying a wider redundant model with improved test NLL despite similar or smaller estimated \(\widehat\lambda\).