✗ Failed on benchmark 2025

Log-Scale Self-Similar Activation

Implementation & benchmark of arXiv:2512.02291 — From two-dimensional continuous maps to one-dimensional discontinuous maps: a novel reduction explaining complex bifurcation structures in piecewise-linear families of maps

Usefulness6/10

Difficulty4/10

Novelty7/10

Architecture Accuracy Generalization Scalability Mlp Transformer Initialization Dynamical systems Linear algebra Approximation theory

Source paper: From two-dimensional continuous maps to one-dimensional discontinuous maps: a novel reduction explaining complex bifurcation structures in piecewise-linear families of maps arXiv:2512.02291 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a conventional scalar activation by a geometrically indexed family of affine pieces whose slope changes with the logarithmic magnitude of the input. The same two endpoint parameters are reused across all scales, giving a compact, explicitly scale-aware activation that can represent different responses for exponentially separated activation magnitudes.

Formulas

$$I_k=[\sigma^{-k},\sigma^{-(k-1)}),\qquad k(z)=\left\lceil-\frac{\ln z}{\ln\sigma}\right\rceil.$$

$$h_k(z;\eta,\nu,\sigma)=\frac{\eta-\nu}{\sigma-1}\,\sigma^k z+\frac{-\eta+\sigma\nu}{\sigma-1},\qquad z\in I_k.$$

$$h\!\left(\frac{z}{\sigma};\frac{\eta}{\sigma},\frac{\nu}{\sigma},\sigma\right)=\frac{1}{\sigma}h(z;\eta,\nu,\sigma).$$

$$\widetilde h(x)=\operatorname{sign}(x)\,h(|x|+\epsilon;\eta,\nu,\sigma)$$

Mathematical statement

The paper defines intervals I_k=[sigma^(-k), sigma^(-(k-1))) for sigma>1 and uses the affine branch h_k(z;eta,nu,sigma)=((eta-nu)/(sigma-1)) sigma^k z + (-eta+sigma nu)/(sigma-1) when z belongs to I_k. Here z>0 is the input magnitude, k is the unique logarithmic scale index, and eta and nu are the two endpoint values. The branch slope is s_k=((eta-nu)/(sigma-1)) sigma^k, so the function has exponentially changing local gain. Its key structural guarantee is the scale-equivariance identity h(z/sigma; eta/sigma, nu/sigma, sigma)=h(z;eta,nu,sigma)/sigma. For a neural activation, use a shared normalized pair (eta_0,nu_0) and scale them as eta_k=eta_0 sigma^(-k) and nu_k=nu_0 sigma^(-k), preserving the same dimensionless shape across magnitude bins. The implementation should use a finite clipped range of k because floating-point activations have bounded practical dynamic range.

Implementation notes

Integrate this at the scalar activation point of an MLP or transformer feed-forward block, immediately after the linear projection and before the next projection. Use a signed version so positive and negative channels share the same magnitude response; optionally give positive and negative branches separate eta and nu parameters. Pseudocode: input tensor X; set z=abs(X)+1e-6; compute k=clip(ceil(-log(z)/log(sigma)), k_min, k_max); compute u=sigma^k z; compute y=((eta-nu)/(sigma-1))*u+(-eta+sigma*nu)/(sigma-1); return sign(X)*y. Parameterize sigma=1+softplus(r), and initialize sigma between 2 and 4, eta>0, and nu>0. To avoid unstable discontinuous gradients at bin boundaries, use a straight-through estimator for the integer k, or linearly blend the two neighboring branch outputs within a narrow band around each boundary. The exact computation from the paper is the logarithmic partition, branch slope, and scaling law; activation bounds and blending width are empirical design choices. First test on a small four-layer MLP and CIFAR-10 or Fashion-MNIST against GELU, ReLU, and piecewise-linear Hard-Swish at equal parameter count. Measure validation accuracy, activation saturation, robustness to multiplying inputs by 2^j, and training stability. Success means equal-or-better accuracy with substantially lower error under input rescaling, without increasing wall-clock cost by more than 10 percent.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built the signed log-scale activation and verified its mechanism across sigma=1.5, 2, 3, 4. All predicted boundary locations were detected, adjacent-branch slope ratios matched sigma within 6e-10, and the scale-equivariance identity held with maximum relative error below 2.2e-15. The small digits MLP comparison showed no practical accuracy win: log-scale achieved 0.880 validation accuracy versus ReLU’s 0.978, with mean rescaled accuracy 0.890 versus 0.973 and roughly 1.9x runtime.

Agent confidence: 8/10

Baseline: ReLU: validation accuracy 0.9778; mean accuracy over input scales 0.9733; runtime 1.09 s
Idea: Log-scale: validation accuracy 0.8800; mean accuracy over input scales 0.8902; runtime 2.06 s; slope-ratio error <=5.7e-10; equivariance error <=2.2e-15

Limitations: Only a small MLP on sklearn digits was tested, with fixed sigma=2 and fixed eta/nu rather than learned parameters. No CIFAR-10, Fashion-MNIST, blending, separate signed parameters, multi-seed statistics, or broader hyperparameter tuning was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2512.02291) Sep 2, 2026

Significantly WORSE than the tuned baseline (+222.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny_shared_activation
Paired seeds: 8
Baseline mean: 7.9319
Idea mean: 25.5462
Effect (Δ): 17.6143 (+222.1%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched tabular MLP track was tested. The activation used fixed sigma=2, eta=1, nu=0 and hard integer binning; no learned endpoint parameters, boundary blending, vision, sequence, or dynamics tracks were tested.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Built and ran an end-to-end paired benchmark on the structurally matched tabular Friedman regression track using the same MLP architecture, optimizer, learning rates, epochs, and seeds for ReLU and the signed log-scale activation. ReLU achieved mean test MSE 7.93194 versus 25.54624 for the idea; the paired delta was +17.61430 with permutation p=0.0081, so the idea was significantly worse. The trained-model mechanism probe was not confirmed because its increasing-magnitude probe measured the reciprocal slope ratio 0.5 rather than the expected ordered ratio 2.0.

Baseline: mean test MSE 7.931941092014313, std 0.8101161551333251, best lr=0.01, epochs=20
Idea: mean test MSE 25.546241760253906, std 1.397921331795095, best lr=0.01, epochs=20
Paired delta: +17.614300668239594; p=0.0081; idea wins 0/8

Benchmark result