# Distinct-kink complexity regularizer and merger

- ID: 559
- Canonical URL: https://synthcore.org/idea/559/distinct-kink-complexity-regularizer-and-merger
- API JSON: https://synthcore.org/api/idea/559.json
- API Markdown: https://synthcore.org/api/idea/559.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.07778](https://arxiv.org/abs/2607.07778)
- Category: memory
- Solves: memory, generalization, stability
- ML areas: mlp, pruning, regularization, inference-speedup
- Math tags: geometry, approximation-theory, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace nominal hidden-unit count by the number of distinct realized ReLU kink hyperplanes, then regularize or prune this effective count. Neurons whose normalized affine boundaries coincide can be exactly merged by summing their canonical coefficients, reducing memory without changing the represented function and aligning the model's complexity measure with the theorem.

## Mathematical statement

A ReLU neuron $a\,\operatorname{ReLU}(\langle w,x\rangle+b)$ has a kink on the hyperplane $\langle w,x\rangle+b=0$. For $w\neq0$, define $r=\|w\|_2$, $u=w/r$, $\beta=b/r$, and the canonical coefficient $\alpha=ar$; then $a\operatorname{ReLU}(\langle w,x\rangle+b)=\alpha\operatorname{ReLU}(\langle u,x\rangle+\beta)$. If several neurons have the same canonical hyperplane $(u,\beta)$, their coefficients add exactly. Let $k(f)$ be the number of distinct realized kink hyperplanes after merging. The paper's realized-kink theorem replaces $\bar m$ by $k(f)+1$: $\operatorname{Lip}(f)\geq c\varepsilon\sqrt{n/[(k(f)+1)\log(C(k(f)+1)nd/\varepsilon)]}$ for $k(f)\leq n$. Thus redundant parameterization cannot evade the robustness-capacity tradeoff.

## Key formulas

- $$a\,\operatorname{ReLU}(\langle w,x\rangle+b)=\alpha\,\operatorname{ReLU}(\langle u,x\rangle+\beta),\qquad r=\|w\|_2,\quad u=\frac{w}{r},\quad \beta=\frac{b}{r},\quad \alpha=ar.$$
- $$f(x)=\sum_{j=1}^{k(f)}\alpha_j\operatorname{ReLU}(\langle u_j,x\rangle+\beta_j)+\langle v,x\rangle+c,\qquad \|u_j\|_2=1,$$
- $$\operatorname{Lip}(f)\geq c\,\varepsilon\sqrt{\frac{n}{(k(f)+1)\log(C(k(f)+1)nd/\varepsilon)}}.$$

## Implementation notes

Apply this at optimizer checkpoints or before inference for a two-layer ReLU network. For every nonzero hidden weight, compute $u=w/\|w\|_2$, $\beta=b/\|w\|_2$, and $\alpha=a\|w\|_2$. Canonicalization has a sign subtlety: retain a fixed orientation convention for $(u,\beta)$, and only merge entries whose full normalized pair agrees within tolerance; do not merge $(u,\beta)$ with $(-u,-\beta)$ unless the corresponding affine correction is explicitly derived. Hash rounded $(u,\beta)$ vectors or use locality-sensitive nearest-neighbor search with tolerance $\tau$. For each cluster, replace all members by one neuron with coefficient $\alpha_{\mathrm{cluster}}=\sum_i\alpha_i$, preserving the shared $(u,\beta)$; discard coefficients below a threshold only in approximate compression mode. During training, add $\lambda_k\tilde{k}$ to the loss, where $\tilde{k}=\sum_{p\in\text{clusters}}\operatorname{softplus}((\|\alpha_p\|-\alpha_0)/s)$ is a differentiable proxy, and periodically perform exact discrete merging. Re-estimate $k(f)$ after every merge and log the theorem-inspired lower bound using $k(f)+1$, not raw width. The exact algebraic rescaling and coefficient summation come from the canonical kink representation; clustering tolerance, penalty weights, and pruning thresholds are empirical. First test on noisy 2D regression and CIFAR-10 with a width-1024 shallow ReLU model, comparing raw-$L_0$ pruning and magnitude pruning. Measure exact function discrepancy on a dense probe set, parameter reduction, validation accuracy, Jacobian norm, and robustness to input perturbations. Success means at least 2x fewer hidden units at near-identical predictions, with lower or unchanged Jacobian norm and no worse validation error.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Implemented canonical ReLU kink normalization and exact same-hyperplane coefficient merging. The algebra check and dense-probe comparison showed only float32-level discrepancies (maximum about 1.7e-6 and 1.4e-6 respectively), while merging reduced width from 48 to 24 and parameters from 147 to 75 without changing validation MSE or mean Jacobian norm. This verifies the claimed exact merger phenomenon, but does not demonstrate that a differentiable regularizer discovers duplicates spontaneously during training.

### Mechanism check

- Verdict: Implemented canonical ReLU kink normalization and exact same-hyperplane coefficient merging. The algebra check and dense-probe comparison showed only float32-level discrepancies (maximum about 1.7e-6 and 1.4e-6 respectively), while merging reduced width from 48 to 24 and parameters from 147 to 75 without changing validation MSE or mean Jacobian norm. This verifies the claimed exact merger phenomenon, but does not demonstrate that a differentiable regularizer discovers duplicates spontaneously during training.
- Confidence: 8/10
- Limitations: The experiment uses deliberately constructed duplicate neurons, so it isolates exact merger correctness rather than testing duplicate formation under regularization. It does not test CIFAR-10, magnitude/L0 pruning baselines, approximate clustering tolerance, inference latency, or robustness under adversarial perturbations.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.75635
- Paired wins: 6/8
- Benchmark verdict: no measurable effect

## Artifacts

- [kink_experiment.py](https://synthcore.org/code/1387/kink_experiment.py)
- [report.md](https://synthcore.org/code/1387/report.md)
- [report_bench_2026-09-04T181008.md](https://synthcore.org/code/1387/report_bench_2026-09-04T181008.md)
- [results/bench_report.json](https://synthcore.org/code/1387/results/bench_report.json)
- [results/metrics.json](https://synthcore.org/code/1387/results/metrics.json)
- [stage2_bench.py](https://synthcore.org/code/1387/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1387)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.