# Accessibility regularizer for power-law experts

- ID: 2469
- Canonical URL: https://synthcore.org/idea/2469/accessibility-regularizer-for-power-law-experts
- API JSON: https://synthcore.org/api/idea/2469.json
- API Markdown: https://synthcore.org/api/idea/2469.md
- Verification status: unverified
- Source: [arXiv:2608.17714](https://arxiv.org/abs/2608.17714)
- Category: regularization
- Solves: stability, generalization
- ML areas: mlp, regularization, training-dynamics
- Math tags: metric-geometry, convex-analysis, geometry
- Ratings: usefulness 5/10; difficulty 3/10; novelty 8/10

## Idea description

When a neural field learns power-law exponents, penalize exponent configurations whose Newton support violates the paper's finite-distance accessibility condition. This discourages combinations of exponents that create excessively strong joint singularities while preserving anisotropic scaling when it is supported by the data.

## Mathematical statement

For exponent pairs \((a_j,b_j)\), the paper's support function is \(\mathcal{H}(p,q)=\max_j(pa_j+qb_j)\). A projective direction \([(p,q)]\), with \(p,q>0\), is accessible exactly when \(\mathcal{H}(p,q)<2\min\{p,q\}\). For nonnegative exponent data, the paper further states that the corner is at finite metric distance exactly when \(\max_j(a_j+b_j)<2\). The latter follows by evaluating the balanced direction \(p=q\) and is convenient for implementation. We use a soft violation penalty rather than a hard constraint: \(R_{\mathrm{acc}}=\operatorname{softplus}(\tau(\max_j(a_j+b_j)-2))/\tau\), where \(\tau\) controls sharpness. To test anisotropic rather than only balanced behavior, sample positive directions \((p_k,q_k)\) and add \(\frac{1}{K}\sum_k\operatorname{softplus}(\tau[\mathcal{H}(p_k,q_k)-2\min(p_k,q_k)])/\tau\). This is a transferable stability prior, not a theorem about arbitrary neural networks: it is justified when the network explicitly represents singular coordinate scalings.

## Key formulas

- $$[(p,q)]\ \text{is accessible}\quad\Longleftrightarrow\quad\mathcal{H}(p,q)<2\min\{p,q\}$$
- $$\mathcal{H}(p,q)=\max_{1\leq j\leq N}(pa_j+qb_j)$$
- $$\max_j(a_j+b_j)<2$$
- $$R_{\mathrm{acc}}=\frac{1}{K}\sum_{k=1}^{K}\frac{\operatorname{softplus}\!\left(\tau\left[\max_j(p_k a_j+q_k b_j)-2\min(p_k,q_k)\right]\right)}{\tau}$$

## Implementation notes

Apply the regularizer to the exponent parameters of the Newton-envelope layer, not to ordinary unconstrained MLP weights. Parameterize a_j=softplus(alpha_j) and b_j=softplus(beta_j) so the intended nonnegative-exponent assumption holds. During each training step, draw K fixed or freshly sampled positive directions p_k,q_k from log-uniform values in [0.25,4], optionally normalize each pair by p_k+q_k because the condition is homogeneous. Compute h_k=max_j(p_k*a_j+q_k*b_j), v_k=h_k-2*min(p_k,q_k), and add lambda_acc/K times softplus(tau*v_k)/tau to the task loss. Include the balanced shortcut v_bal=max_j(a_j+b_j)-2 for a cheap always-on penalty. Start with lambda_acc in {1e-4,1e-3,1e-2} and tau=10; anneal lambda from zero over the first 10% of training so the model first discovers useful exponents. The exact support maximum and accessibility threshold come from the paper; K, direction sampling, lambda, and softplus smoothing are engineering choices. First test on a synthetic near-corner regression task with train points at moderate radii and test points closer to x=y=0, comparing no regularization, balanced-only regularization, and directional regularization. Track extrapolation error, maximum predicted magnitude, gradient norm, and learned exponents. Success means improved near-corner extrapolation and fewer exploding gradients at equal in-domain error; reject the method if it simply biases the model toward underfitting or materially worsens ordinary-domain accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.