# Permutation-Symmetric Quadratic Module

- ID: 2235
- Canonical URL: https://synthcore.org/idea/2235/permutation-symmetric-quadratic-module
- API JSON: https://synthcore.org/api/idea/2235.json
- API Markdown: https://synthcore.org/api/idea/2235.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.13335](https://arxiv.org/abs/2608.13335)
- Category: architecture
- Solves: memory, scalability, stability
- ML areas: mlp, attention, cnn
- Math tags: linear-algebra, symmetry, approximation-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a wide collection of interchangeable near-zero branches with a module whose output is explicitly a quadratic form in the branch-weight Gram matrix. The module preserves the paper's leading-order behavior while making the relevant collective variable explicit and allowing low-rank parameterizations.

## Mathematical statement

Theorem 1 states that a module f_x(w_1,...,w_d) that is invariant under permutations of its d components, is three-times continuously differentiable, and satisfies the condition that a component's gradient vanishes when that component's own weights are zero, has the expansion f_x(W) - f_x(0) = mu^T g(x) + Tr[W W^T A(x)] + O(||W||^3). Here W is the matrix whose columns are component weights, x is the module input, g(x) is a first-order feature vector, mu is its coefficient vector, M = W W^T is the order parameter, A(x) is the architecture- and input-dependent structure matrix, and O(||W||^3) is the cubic Taylor remainder. Since Tr[M A(x)] depends only on the symmetric part of A(x), A can be symmetrized without changing the output.

## Key formulas

- $$f_x(W)-f_x(0)=\mu^{\top}g(x)+\operatorname{Tr}\!\left[WW^{\top}A(x)\right]+O\!\left(\|W\|^{3}\right)$$
- $$M=WW^{\top},\qquad \widehat f_x(W)=b(x)+\mu^{\top}g(x)+\operatorname{Tr}\!\left[M A(x)\right]$$
- $$\operatorname{Tr}[M A(x)]=\sum_{p,q}M_{pq}A(x)_{qp}=\sum_{i=1}^{d}w_i^{\top}A(x)w_i$$

## Implementation notes

Integrate this as a residual feature-mixing block in a small MLP, CNN, or attention-style module. Let x be an input feature in R^n, let W in R^(n by d) contain d exchangeable branch weights, and let a structure network produce B_theta(x) in R^(n by n). Symmetrize it as A_theta(x) = (B_theta(x) + B_theta(x)^T)/2. Compute M = W W^T and z(x) = b_theta(x) + mu^T g_theta(x) + Tr(M A_theta(x)); add a projection of z to the residual stream. Pseudocode: `g,b,B = structure_net(x); A = 0.5*(B + transpose(B)); M = W @ transpose(W); z = b + dot(mu,g) + sum(M*A); y = x + output_projection(z)`. The paper's mathematical contribution is the trace quadratic form and its cubic approximation guarantee. The functions A_theta, g_theta, and b_theta are learned approximations. To reduce cost, use W = U V^T with rank r and compute M implicitly or factor the trace as Tr(V U^T A U V^T). Initialize W with standard deviation epsilon divided by sqrt(n), then vary epsilon to test the predicted cubic remainder. First compare a width-64 two-layer MLP branch-sum against this module on MNIST or CIFAR-10, matching parameter count and FLOPs. Measure accuracy, peak memory, and training loss at equal optimizer steps and equal FLOPs. Success means equal or higher accuracy with reduced memory or compute, plus approximation error scaling close to epsilon cubed at initialization.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a permutation-symmetric branch module and its Gram-matrix quadratic approximation in experiment.py. The core mechanism manifested: the remainder scaled as epsilon^3 (observed slope 3.010 vs predicted 3), orthogonal branch mixing changed the original only at cubic order (2.992 vs 3), quadratic output was Gram-invariant (error 4.8e-7), and coherent width scaling was linear (1.000 vs 1). In the tiny fitting test, the full branch baseline substantially outperformed the quadratic approximation, so the mechanism was validated but no task-level accuracy win was observed.

### Mechanism check

- Verdict: Built a permutation-symmetric branch module and its Gram-matrix quadratic approximation in experiment.py. The core mechanism manifested: the remainder scaled as epsilon^3 (observed slope 3.010 vs predicted 3), orthogonal branch mixing changed the original only at cubic order (2.992 vs 3), quadratic output was Gram-invariant (error 4.8e-7), and coherent width scaling was linear (1.000 vs 1). In the tiny fitting test, the full branch baseline substantially outperformed the quadratic approximation, so the mechanism was validated but no task-level accuracy win was observed.
- Confidence: 8/10
- Limitations: Only a synthetic scalar branch-sum toy was tested, not MNIST/CIFAR, CNNs, attention, low-rank parameterizations, FLOPs, or peak-memory measurements. The random-width exponent was noisy (0.680 rather than exactly 0.5), and the quadratic model was evaluated against a teacher containing a sizable cubic term.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.4109
- Paired wins: 5/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_quadratic.py](https://synthcore.org/code/887/bench_quadratic.py)
- [bench_report.json](https://synthcore.org/code/887/bench_report.json)
- [experiment.py](https://synthcore.org/code/887/experiment.py)
- [report.md](https://synthcore.org/code/887/report.md)
- [report_bench_2026-09-01T133647.md](https://synthcore.org/code/887/report_bench_2026-09-01T133647.md)
- [results.json](https://synthcore.org/code/887/results.json)
- [Download all files as ZIP](https://synthcore.org/download/887)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.