# Mass-Covering Dimension Regularizer

- ID: 1861
- Canonical URL: https://synthcore.org/idea/1861/mass-covering-dimension-regularizer
- API JSON: https://synthcore.org/api/idea/1861.json
- API Markdown: https://synthcore.org/api/idea/1861.md
- Verification status: unverified
- Source: [arXiv:2608.05882](https://arxiv.org/abs/2608.05882)
- Category: regularization
- Solves: generalization, memory, stability
- ML areas: embedding, regularization, quantization
- Math tags: metric-geometry, measure-theory, approximation-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Regularize hidden representations using the number of metric balls required to cover at least a fixed fraction of minibatch probability mass. The outlier tolerance ignores a controlled fraction of atypical samples, while the resolution parameter makes the penalty explicitly scale-dependent. Combine the penalty with a variance floor or reconstruction term so that reducing geometric dimension does not produce a constant representation.

## Mathematical statement

Definition 1 defines the covering number of a set S in metric space X as N_epsilon(S) = min{N : S is contained in the union of N closed balls B(x_i, epsilon), with centers x_i in X}. Definition 2 makes this robust to outliers: for a probability measure mu on X and tolerance tau in (0,1), N_epsilon(mu,tau) = inf{N_epsilon(S) : mu(S) >= 1-tau}. Definition 3 defines the measure-based intrinsic dimension d_epsilon(mu,tau) = log N_epsilon(mu,tau) / (-log epsilon). In the neural adaptation, X is the hidden representation space, mu is the empirical embedding distribution, epsilon is the target resolution, and tau is the tolerated outlier mass. For normalized embeddings z_1,...,z_B and M candidate centers c_1,...,c_M, use a differentiable soft cover a_ij = exp(-||z_j-c_i||_2^2/(2 sigma^2)). Define N_soft = sum_i [1-exp(-sum_j a_ij/B)] and d_soft = log(N_soft+delta)/(-log(epsilon+delta)), where sigma is a softness bandwidth and delta prevents numerical singularities. Minimize d_soft together with the task objective and an anti-collapse penalty.

## Key formulas

- $$\mathcal{N}_{\varepsilon}(S):=\min\left\{N:S\subseteq\bigcup_{i=1}^{N}B(x_{i},\varepsilon),\ x_{i}\in X\right\}$$
- $$\mathcal{N}_{\varepsilon}(\mu,\tau):=\inf\left\{\mathcal{N}_{\varepsilon}(S):\mu(S)\geq 1-\tau\right\}$$
- $$d_{\varepsilon}(\mu,\tau):=\frac{\log\mathcal{N}_{\varepsilon}(\mu,\tau)}{-\log\varepsilon}$$
- $$a_{ij}=\exp\left(-\frac{\lVert z_j-c_i\rVert_2^2}{2\sigma^2}\right),\quad N_{\mathrm{soft}}=\sum_{i=1}^{M}\left[1-\exp\left(-\frac{1}{B}\sum_{j=1}^{B}a_{ij}\right)\right],\quad d_{\mathrm{soft}}=\frac{\log(N_{\mathrm{soft}}+\delta)}{-\log(\varepsilon+\delta)}$$

## Implementation notes

1. Integration point: attach the regularizer to a hidden representation tensor, such as the CLS token of a ViT, the projection output of a contrastive encoder, or the penultimate feature vector of an MLP. Flatten each sample to z_j in R^D and L2-normalize it before calculating distances. Add lambda_dim*d_soft to the ordinary task loss. 2. Pseudocode: take minibatch z[1:B]; normalize z; choose M centers c[1:M] as random minibatch points or use a detached k-means update; compute A[i,j] = exp(-||z[j]-c[i]||^2/(2*sigma^2)); compute Nsoft = sum_i(1-exp(-sum_j A[i,j]/B)); compute dsoft = log(Nsoft+delta)/(-log(epsilon+delta)); compute variance_penalty = mean(relu(gamma-std(z,dim=0))^2); optimize task_loss + lambda_dim*dsoft + lambda_var*variance_penalty. 3. The paper supplies the exact covering-number and intrinsic-dimension definitions; Nsoft is the differentiable estimator. For evaluation, estimate the empirical robust covering number using a greedy epsilon-net: repeatedly choose an uncovered point as a center, mark all points within epsilon as covered, and optionally ignore the tau fraction with largest nearest-center distances. Sweep epsilon and tau, and compare dsoft with this nondifferentiable estimate. 4. First experiment: train a small CIFAR-10 MLP or ViT-Tiny against an unregularized baseline and a covariance-rank penalty. Test lambda_dim values {1e-4, 1e-3, 1e-2}, three epsilon values, and tau in {0, 0.05, 0.1}. Measure test accuracy, held-out robust cover dimension, activation quantization error at fixed bit width, and per-layer feature variance. Success is equal or better accuracy with lower held-out covering dimension and reduced activation storage or quantization error, without feature collapse.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.