SynthCore
✗ Mechanism failed 2026

Intrinsic-Capacity Feature Bottleneck

Implementation & benchmark of arXiv:2607.01010 — Function-Counting Theory for Low-Dimensional Data Structures

Usefulness6/10
Difficulty6/10
Novelty6/10
Architecture GeneralizationMemoryAccuracyEmbeddingArchitectureRegularizationGeometryLinear algebraCombinatorics

Source paper: Function-Counting Theory for Low-Dimensional Data Structures arXiv:2607.01010 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Regularize an intermediate neural representation according to its estimated low-dimensional separability capacity instead of its ambient feature width. Learn feature gates or subspace assignments, estimate the union of active supports, and penalize representations whose Cover capacity exceeds a task-dependent target.

Formulas

$$C(N,d)=2\sum_{k=0}^{d-1}\binom{N-1}{k},\quad d\leq N;\qquad C(N,d)=2^N,\quad d\geq N.$$
$$C(N,s)<C_F<C(N,r)\qquad\text{when }0<s<r.$$
$$\dim_{\mathbb R}\left(\sum_{j\in\mathfrak{s}^{c}}E_j\right)=\left|\bigcup_{j\in\mathfrak{s}^{c}}S_j\right|.$$
$$d_{\mathrm{eff}}=\sum_{k=1}^{D}\left[1-\prod_{j=1}^{J}(1-a_{jk})\right],\qquad \mathcal L=\mathcal L_{\mathrm{task}}+\lambda\max\left(0,\log C(N,d_{\mathrm{eff}})-\tau\right).$$

Mathematical statement

For N points and a homogeneous linear classifier in dimension d, Cover's dichotomy count is C(N,d) = 2 times the sum from k = 0 to d - 1 of binomial(N - 1,k), when d is at most N, and equals 2 to the power N when d is at least N. The paper proves that if the refined structural dimensions satisfy 0 < s < r, then C(N,s) < C_F < C(N,r), where C_F is the number of separable dichotomies of the data set F. For a union of coordinate subspaces E_j = span of the basis vectors xi_k indexed by S_j, Proposition 3.2 gives dim(sum over j in s-complement of E_j) = cardinality of the union of S_j. We use d_eff = sum over coordinates k of one minus the product over groups j of (1 - a_jk), where a_jk is a soft gate, as a differentiable estimate of the active support union. The regularizer targets log C(N,d_eff), with N the effective sample count and tau a desired capacity budget.

Implementation notes

Integrate this at the output of an encoder, before the classifier or before a transformer block whose width is being selected. Let the representation have D coordinates and introduce J soft group gates a_jk in [0,1], implemented with sigmoid parameters or hard-concrete gates. For each minibatch, compute d_eff = sum over k of one minus the product over j of (1 - a_jk). Evaluate log C(N,d_eff) using log-gamma binomial coefficients and a log-sum-exp calculation; use the continuous d_eff during backpropagation and round it only for reporting. Optimize the task loss plus lambda times max(0, log C(N,d_eff) - tau), where N is the batch size or an exponential moving average of the number of distinct training examples represented, tau is the target capacity, and lambda is the penalty coefficient. The exact mathematics supplies the Cover count and the support-union dimension identity; gates, group assignments, and the target tau are estimated or selected experimentally. Start with a two-layer MLP on MNIST using a 256-dimensional bottleneck, comparing a dense baseline, ordinary L1 gating, and this capacity penalty. Measure test accuracy, active dimensions, classifier parameters, activation memory, and optimization speed. The first success signal is a substantial reduction in active dimensions and memory at matched accuracy; a stronger result is improved test accuracy or calibration at equal parameter count.

Verification

Mechanism failed

Stage 1 · Toy mechanism gate: Failed ✗

Stage 2 · Mechanism transferred to benchmark: Not tested

Stage 2 · Practical benchmark result: Not run

Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built the intrinsic-capacity bottleneck with soft union-support gates, a differentiable piecewise Cover penalty, exact Cover-count verification, and a fixed-seed digits MLP comparison. The mathematical checks passed: Cover counts were strictly increasing for positive dimensions, autodiff matched finite differences to 1.2e-11, and the union-support dimension matched span rank. However, the capacity penalty saturated at d_eff > N, producing no meaningful sparsity or accuracy improvement over L1; both gated methods reduced accuracy from 97.56% to 96.44%, so the promised win was not observed.

Agent confidence
9/10
Baseline
Dense: 97.56% test accuracy, 256 active dimensions, 1.00 relative activation size, 7.47 s
Idea
Capacity: 96.44% test accuracy, 71.79 soft active dimensions, 0.280 relative activation size, 9.48 s; nearly identical sparsity and accuracy to L1.

Limitations: Only sklearn digits and one fixed seed were tested. The continuous Cover surrogate clamps at batch size N, so its gradient vanishes when d_eff exceeds N; the capacity run remained in this saturated regime. Classifier parameters were not physically reduced after gating, and activation-memory savings were estimated rather than measured from an exported compact model.

How to run: python3 capacity_bottleneck_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Artifacts

Implementation overview ⬇ Download all as ZIP 3 files · code, reports and structured results

Related ideas