△ Mechanism confirmed, baseline not beaten 2026

Positive-Measure Span Regularizer

Implementation & benchmark of arXiv:2607.06048 — Separation Capacity of Scattering Networks on Low-Dimensional Datasets

Usefulness6/10

Difficulty4/10

Novelty6/10

Regularization Generalization Accuracy Stability Cnn Embedding Regularization Geometry Measure theory Linear algebra

Source paper: Separation Capacity of Scattering Networks on Low-Dimensional Datasets arXiv:2607.06048 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Regularize an encoder so that feature vectors from every substantial local data region occupy a well-conditioned, high-dimensional linear span. Instead of only maximizing global covariance rank, penalize low effective rank in many local batches or neighborhoods, approximating the paper's worst-positive-measure-set definition of separation capacity.

Formulas

$$\mathcal{SC}^{s}(\Phi)=2\cdot\min\left\{\dim_{\mathbb{R}}(V):V\subseteq\mathbb{R}^{M'},\left(\Phi_{\sharp}\left(\mathcal{H}^{s}\llcorner E\right)\right)(V)>0\right\}.$$

$$\mathcal{SC}^{s}(\Phi)=2\min_{\substack{A\subseteq E\\\mathcal{H}^{s}(A)>0}}\dim_{\mathbb{R}}\left(\operatorname{span}_{\mathbb{R}}(\Phi(A))\right).$$

$$r_{\tau}(S)=\sum_{j=1}^{q}\frac{\sigma_j(S)^2}{\sigma_j(S)^2+\tau^2},\qquad q=\min(|S|,M').$$

$$\mathcal{L}_{\mathrm{span}}=-\operatorname{softmin}_{S\in\mathcal{N}}r_{\tau}(S)=\frac{1}{\beta}\log\left(\sum_{S\in\mathcal{N}}e^{-\beta r_{\tau}(S)}\right).$$

Mathematical statement

Lemma 2.2 states that for a measurable feature map $\Phi:E\to\mathbb{R}^{M'}$, the $s$-separation capacity is $\mathcal{SC}^{s}(\Phi)=2\min\{\dim_{\mathbb{R}}(V):V\subseteq\mathbb{R}^{M'},\ (\Phi_{\sharp}(\mathcal{H}^{s}\llcorner E))(V)>0\}$. Here $E$ is the data set, $\mathcal{H}^{s}\llcorner E$ is its $s$-dimensional Hausdorff measure, $\Phi_{\sharp}$ is the pushforward measure, and $V$ is a linear subspace containing the feature image of a positive-measure subset. The equivalent formula in (2.2) is $\mathcal{SC}^{s}(\Phi)=2\min_{A\subseteq E,\ \mathcal{H}^{s}(A)>0}\dim_{\mathbb{R}}(\operatorname{span}_{\mathbb{R}}(\Phi(A)))$. We approximate positive-measure subsets $A$ by local minibatch neighborhoods $S$, and approximate span dimension using singular values $\sigma_j(S)$ of the feature matrix. A smooth effective-rank proxy is $r_{\tau}(S)=\sum_{j=1}^{q}\frac{\sigma_j(S)^2}{\sigma_j(S)^2+\tau^2}$, where $q=\min(|S|,M')$ and $\tau>0$ is a numerical threshold. Maximizing the soft minimum of $r_{\tau}(S)$ over neighborhoods raises the empirical analogue of the paper's worst-subset span dimension.

Implementation notes

(1) Integration point: attach this regularizer to the output of a CNN, scattering network, or encoder before the classifier head. Let $z_i=\Phi(x_i)\in\mathbb{R}^{M'}$ be the flattened or projected representation for each sample. Do not apply it to logits, because the desired property is diversity of the representation rather than class-score separation. For computational control, first project $z_i$ to a fixed dimension of 64 or 128 with a learned linear probe whose output is used only by the regularizer. (2) Pseudocode: z = encoder(x); z_reg = normalize(proj(z)); neighborhoods = build_knn_groups(z_reg.detach(), k=32). For every group $S$, form the feature matrix $Z_S\in\mathbb{R}^{|S|\times M'}$ with rows $z_i^T$, compute singular values $\sigma_j(S)$ using torch.linalg.svdvals, evaluate r = sum(sigma2/(sigma2 + tau**2)), and combine groups with L_span = logsumexp(-beta*r)/beta. Optimize L_total = L_task + lambda_span*L_span. Recompute nearest-neighbor groups every few hundred steps, or use labels, augmentations, or a frozen embedding to define groups without backpropagating through group membership. (3) The exact mathematical quantity supplied by the paper is the minimum span dimension over all positive-measure subsets; the neighborhood collection $\mathcal{N}$, threshold $\tau$, and soft-min temperature $\beta$ are empirical approximations. Normalize each group by its Frobenius norm before the SVD so the loss cannot be improved merely by feature scaling. Start with $\tau=0.05$, $\beta=5$, $\lambda_{span}\in\{10^{-4},10^{-3},10^{-2}\}$, and groups of 32 samples. (4) First experiment: train a small ResNet-18 and a fixed Morlet-scattering classifier on CIFAR-10, comparing cross-entropy alone, global variance regularization, and this local span loss at equal FLOPs. Measure test accuracy, nearest-neighbor robustness under augmentations, per-group singular-value spectra, and the worst 10th-percentile effective rank. The method is successful if it increases the worst-group rank and improves accuracy or corruption robustness without increasing training instability; a negative result is indicated if only global feature norms increase while local rank and accuracy remain unchanged.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a differentiable local effective-rank regularizer with Frobenius normalization and label-defined local groups, plus a numerical sanity check. The toy check confirmed that collapsed features have low rank, optimization increases their rank, and uniform feature rescaling leaves the proxy unchanged. In the digits experiment, the regularizer improved worst-group rank from 2.6055 to 2.7393 and mean rank from 3.1458 to 3.2542, but accuracy slightly fell from 95.78% to 95.33%; thus the rank effect is real, while an accuracy win was not demonstrated.

Agent confidence: 7/10

Baseline: accuracy=0.9578, test 10th-percentile rank=2.6055, mean rank=3.1458
Idea: accuracy=0.9533, test 10th-percentile rank=2.7393, mean rank=3.2542; toy collapsed rank 0.9780 -> 6.7793 after optimization; scale invariance 6.2335 vs 6.2335

Limitations: Only one small digits dataset and one random seed were tested; no CIFAR-10, ResNet, scattering classifier, augmentation robustness, FLOP-matched analysis, or hyperparameter sweep was performed. Local groups use class labels rather than geometric neighborhoods, and the initial soft-min sign error was corrected before the reported run.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06048) Sep 4, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.6923, wins 4 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 8.009
Idea mean: 8.0075
Effect (Δ): -0.0015 (−0.0%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.6923 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched built-in tabular track was tested; vision, sequence, and dynamics tracks were not applicable to this encoder regularizer. Local neighborhoods were approximated by shuffled minibatches rather than true k-nearest-neighbor regions, and the experiment used a compact 400/200 tabular split and 20 epochs. Representation diagnostics were stored separately from task metrics.

How to run: python3 run_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the positive-measure span regularizer as an auxiliary loss on the shared mlp_tiny encoder and ran the full 8-seed tabular benchmark with a matched three-point learning-rate sweep. The idea produced a negligible MSE improvement (delta=-0.001512, permutation p=0.69225) and only a tiny effective-rank increase (1.014278 to 1.014433), so there is no significant task-metric win; the mechanism direction was observed but not substantively strong.

Baseline: best lr=0.01; mean test MSE=8.0089843
Idea: best shared lr=0.01; mean test MSE=8.0074723; paired delta=-0.0015120; permutation p=0.69225; mean held-out effective rank 1.014278 -> 1.014433

Benchmark result