# Sparse Learnable Power-Law Head

- ID: 1634
- Canonical URL: https://synthcore.org/idea/1634/sparse-learnable-power-law-head
- API JSON: https://synthcore.org/api/idea/1634.json
- API Markdown: https://synthcore.org/api/idea/1634.md
- Verification status: unverified
- Source: [arXiv:2608.00379](https://arxiv.org/abs/2608.00379)
- Category: architecture
- Solves: accuracy, generalization, scalability
- ML areas: mlp, optimizer, loss, regularization, training-dynamics
- Math tags: optimization, approximation-theory, statistics, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Attach a symbolic sparse head to a neural encoder instead of using a dense final MLP. The head evaluates a library of learnable power-law and interaction terms on nonnegative learned features, jointly optimizes linear coefficients and exponents, and removes inactive terms with coefficient sparsity. This should provide a compact model with better relative-error behavior on positive targets spanning several orders of magnitude.

## Mathematical statement

Let x_i be an input, h_phi(x_i) in R_+^d the nonnegative representation produced by encoder parameters phi, and theta_j(h;lambda_j) the jth candidate symbolic feature with learnable structural parameters lambda_j. The prediction is yhat_i=sum_j xi_j theta_j(h_phi(x_i);lambda_j), where xi_j is a linear coefficient and sparsity is imposed on the coefficient vector. A practical library is theta_j(h;lambda_j)=product_k(h_k+epsilon_h)^(a_jk), with learnable exponents a_jk, optionally including single-feature powers and pairwise products. For positive targets y_i, optimize a hybrid relative-scale objective consisting of squared logarithmic error, MAPE, and an L1 penalty on xi. The logarithmic term equalizes multiplicative errors, while the L1 term drives redundant symbolic terms to zero.

## Key formulas

- $$\hat y_i=\boldsymbol{\Theta}(h_{\phi}(x_i);\boldsymbol{\Lambda})\cdot\boldsymbol{\Xi}=\sum_{j=1}^{M}\xi_j\theta_j(h_{\phi}(x_i);\lambda_j)$$
- $$\theta_j(h;\lambda_j)=\prod_{k=1}^{d}(h_k+\epsilon_h)^{a_{jk}},\qquad \lambda_j=(a_{j1},\ldots,a_{jd})$$
- $$\mathcal{L}=\frac{w_{\log}}{N}\sum_{i=1}^{N}\left[\ln(\hat y_i+\epsilon_y)-\ln(y_i+\epsilon_y)\right]^2+\frac{w_{\mathrm{mape}}}{N}\sum_{i=1}^{N}\left|\frac{\hat y_i-y_i}{y_i+\epsilon_y}\right|+\lambda_{\mathrm{s}}\sum_{j=1}^{M}|\xi_j|$$
- $$\mathrm{active\mbox{-}term\ fraction}=\frac{1}{M}\sum_{j=1}^{M}\mathbf{1}\{|\xi_j|>\tau\},\qquad \hat y_i\leftarrow\max(\hat y_i,0)$$

## Implementation notes

1. Integration point: use a small encoder h_phi(x)=softplus(g_phi(x)) followed by a symbolic head. Start with M=32 candidates: h_k^a for exponents initialized from {-2,-1,-0.5,0.5,1,2}, pairwise products h_k^a h_l^b, and optionally log(h_k+epsilon_h). Parameterize exponents as a_max tanh(u) to avoid numerical explosions, and use yhat=softplus(sum_j xi_j theta_j+b) for positive outputs. 2. Pseudocode: compute h=softplus(g_phi(x)); construct every theta_j(h;lambda_j); compute yhat; backpropagate the hybrid log-MAPE loss plus lambda_s times the L1 norm of xi; every 10 to 50 epochs apply soft-thresholding xi_j <- sign(xi_j) max(|xi_j|-alpha lambda_s,0); prune terms with |xi_j|<tau and fine-tune. 3. Taken from the paper: the sparse linear library form and hybrid logarithmic/MAPE objective. Estimated empirically: encoder features, exponent values, sparsity strength, pruning threshold, and whether the chosen library spans the task. 4. First cheap experiment: train on a synthetic positive regression problem with targets spanning 10^4, comparing a same-width dense MLP head, a fixed-power library, and this learnable sparse head. Sweep lambda_s and report test MAPE, parameter count, and active terms. The falsifiable prediction is a sparsity transition: as lambda_s increases, the active-term fraction should collapse sharply while MAPE remains within 5 percent of the dense head over an intermediate interval. On y=2 x_1^1.7 x_2^-0.6+0.3 x_3^0.5, recovered exponents should approach 1.7, -0.6, and 0.5, with approximately three retained terms.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.