✗ Failed on benchmark 2026

Permutation-Symmetric Quadratic Module

Implementation & benchmark of arXiv:2608.13335 — Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

Usefulness7/10

Difficulty5/10

Novelty7/10

Architecture Memory Scalability Stability Mlp Attention Cnn Linear algebra Symmetry Approximation theory

Source paper: Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws arXiv:2608.13335 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a wide collection of interchangeable near-zero branches with a module whose output is explicitly a quadratic form in the branch-weight Gram matrix. The module preserves the paper's leading-order behavior while making the relevant collective variable explicit and allowing low-rank parameterizations.

Formulas

$$f_x(W)-f_x(0)=\mu^{\top}g(x)+\operatorname{Tr}\!\left[WW^{\top}A(x)\right]+O\!\left(\|W\|^{3}\right)$$

$$M=WW^{\top},\qquad \widehat f_x(W)=b(x)+\mu^{\top}g(x)+\operatorname{Tr}\!\left[M A(x)\right]$$

$$\operatorname{Tr}[M A(x)]=\sum_{p,q}M_{pq}A(x)_{qp}=\sum_{i=1}^{d}w_i^{\top}A(x)w_i$$

Mathematical statement

Theorem 1 states that a module f_x(w_1,...,w_d) that is invariant under permutations of its d components, is three-times continuously differentiable, and satisfies the condition that a component's gradient vanishes when that component's own weights are zero, has the expansion f_x(W) - f_x(0) = mu^T g(x) + Tr[W W^T A(x)] + O(||W||^3). Here W is the matrix whose columns are component weights, x is the module input, g(x) is a first-order feature vector, mu is its coefficient vector, M = W W^T is the order parameter, A(x) is the architecture- and input-dependent structure matrix, and O(||W||^3) is the cubic Taylor remainder. Since Tr[M A(x)] depends only on the symmetric part of A(x), A can be symmetrized without changing the output.

Implementation notes

Integrate this as a residual feature-mixing block in a small MLP, CNN, or attention-style module. Let x be an input feature in R^n, let W in R^(n by d) contain d exchangeable branch weights, and let a structure network produce B_theta(x) in R^(n by n). Symmetrize it as A_theta(x) = (B_theta(x) + B_theta(x)^T)/2. Compute M = W W^T and z(x) = b_theta(x) + mu^T g_theta(x) + Tr(M A_theta(x)); add a projection of z to the residual stream. Pseudocode: g,b,B = structure_net(x); A = 0.5*(B + transpose(B)); M = W @ transpose(W); z = b + dot(mu,g) + sum(M*A); y = x + output_projection(z). The paper's mathematical contribution is the trace quadratic form and its cubic approximation guarantee. The functions A_theta, g_theta, and b_theta are learned approximations. To reduce cost, use W = U V^T with rank r and compute M implicitly or factor the trace as Tr(V U^T A U V^T). Initialize W with standard deviation epsilon divided by sqrt(n), then vary epsilon to test the predicted cubic remainder. First compare a width-64 two-layer MLP branch-sum against this module on MNIST or CIFAR-10, matching parameter count and FLOPs. Measure accuracy, peak memory, and training loss at equal optimizer steps and equal FLOPs. Success means equal or higher accuracy with reduced memory or compute, plus approximation error scaling close to epsilon cubed at initialization.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a permutation-symmetric branch module and its Gram-matrix quadratic approximation in experiment.py. The core mechanism manifested: the remainder scaled as epsilon^3 (observed slope 3.010 vs predicted 3), orthogonal branch mixing changed the original only at cubic order (2.992 vs 3), quadratic output was Gram-invariant (error 4.8e-7), and coherent width scaling was linear (1.000 vs 1). In the tiny fitting test, the full branch baseline substantially outperformed the quadratic approximation, so the mechanism was validated but no task-level accuracy win was observed.

Agent confidence: 8/10

Baseline: Branch MLP MSE at steps 0/99/249/499: 0.08170, 9.22e-05, 2.14e-05, 1.16e-05
Idea: Quadratic Gram module MSE at steps 0/99/249/499: 0.08175, 0.01447, 0.01447, 0.01447; parameter count 112 each. Math: epsilon slope 3.010, orthogonal-difference slope 2.992, aligned width slope 1.000, quadratic Gram invariance error 4.8e-7.

Limitations: Only a synthetic scalar branch-sum toy was tested, not MNIST/CIFAR, CNNs, attention, low-rank parameterizations, FLOPs, or peak-memory measurements. The random-width exponent was noisy (0.680 rather than exactly 0.5), and the quadratic model was evaluated against a teacher containing a sizable cubic term.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.13335) Sep 1, 2026

Ahead of the baseline (−0.3%) but not statistically significant (p=0.4109, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: custom_branch_mlp
Paired seeds: 8
Baseline mean: 6.8396
Idea mean: 6.8195
Effect (Δ): -0.02 (−0.3%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.4109 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.7%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched tabular track was tested; vision, sequence, and dynamics were not relevant to this branch-aggregation structure and were not run. The benchmark used 800 training and 400 test examples rather than larger defaults, and measured task MSE but not FLOPs or peak memory. The baseline sweep used four seeds, while the selected baseline and idea were evaluated on eight paired seeds.

How to run: python3 bench_quadratic.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the permutation-symmetric quadratic module as a matched replacement for an exchangeable width-64 branch MLP on the tabular Friedman regression track. The cubic remainder scaling held quantitatively (slope 2.999 in the math check and 3.020 on a trained model), and Gram invariance held to numerical precision, but the task-level result was not significant: idea MSE 6.8195 versus baseline 6.8396, paired delta -0.0200 with permutation p=0.4109. Therefore the mechanism transferred, but no statistically reliable benchmark win was observed.

Baseline: Friedman tabular test MSE: mean 6.839571, std 0.346865, 8 paired seeds; best sweep lr=0.01, weight_decay=0.0
Idea: Quadratic Gram module Friedman tabular test MSE: mean 6.819535, std 0.370936, 8 paired seeds; best sweep lr=0.01, weight_decay=0.0; paired delta=-0.020036, 5/8 wins, permutation p=0.4109
Mechanism signature: Math remainder log-log slope 2.9993 vs predicted 3.0; trained-model slope 3.0196 vs predicted 3.0; trained-model permutation max error 1.91e-6; confirmed=true

Benchmark result