# Residual-Contraction log1p Kernel

- ID: 1308
- Canonical URL: https://synthcore.org/idea/1308/residual-contraction-log1p-kernel
- API JSON: https://synthcore.org/api/idea/1308.json
- API Markdown: https://synthcore.org/api/idea/1308.md
- Verification status: unverified
- Source: [arXiv:2607.23196](https://arxiv.org/abs/2607.23196)
- Category: other
- Solves: speedup, stability
- ML areas: transformer, loss, inference-speedup, training
- Math tags: fixed-point, dynamical-systems, numerical-analysis, approximation-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Implement \(\log(1+u)\) through the bounded residual \(\rho(u)\), refining the argument by repeated square roots and propagating the residual backward with a rational recurrence. Use this table-free approximation in softplus and log-sum-exp kernels, where millions of logarithms are evaluated and predictable arithmetic can outperform general-purpose library calls on vectorized hardware.

## Mathematical statement

The paper defines the residual \(\rho(u)=[\log(1+u)-u+\tfrac12u^2]/u^3\), equivalently \(\log(1+u)=u-\tfrac12u^2+u^3\rho(u)\). For \(v=\sqrt{1+u}-1\), the duplication identity \(\log(1+u)=2\log(1+v)\) gives \(\rho(u)=[2\rho(v)+2+v/2]/(2+v)^3\). Starting from \(u_0=u\), define \(u_{k+1}=\sqrt{1+u_k}-1\); then \(u_k\to0\), while \(\rho(u_k)\to1/3\) because the Taylor expansion gives \(\rho(u)=1/3+O(u)\). The backward map \(R(v,r)=[2r+2+v/2]/(2+v)^3\) has local derivative \(2/(2+v)^3\) with respect to \(r\), equal to \(1/4\) at zero, so residual errors are geometrically damped in the refinement direction rather than amplified by reconstructing \(\log(1+u)\) at each level. For numerical use, restrict the direct residual kernel to a safe interval such as \(u\in[-1/2,1]\); for general positive \(x\), range-reduce \(x=2^m(1+u)\) and return \(m\log 2+\log(1+u)\).

## Key formulas

- $$\rho(u)=\frac{\log(1+u)-u+\frac{1}{2}u^{2}}{u^{3}},\qquad \log(1+u)=u-\frac{1}{2}u^{2}+u^{3}\rho(u).$$
- $$v:=\sqrt{1+u}-1,\qquad \log(1+u)=2\log(1+v).$$
- $$\rho(u)=\frac{2\rho(v)+2+\frac{1}{2}v}{(2+v)^{3}}.$$
- $$u_{k+1}=\sqrt{1+u_k}-1,\quad r_K=\frac13,\quad r_k=\frac{2r_{k+1}+2+\frac12u_{k+1}}{(2+u_{k+1})^3},\quad \widehat{\log(1+u)}=u-\frac12u^2+u^3r_0.$$

## Implementation notes

Integrate this at the scalar-elementwise primitive level, initially replacing `log1p` inside softplus and stable log-sum-exp rather than modifying transformer weights. Inputs are floating-point tensors `u`; choose a safe range such as `[-0.5, 1]`, and use ordinary range reduction for values outside that interval. For each element, run the following fixed-depth procedure: (1) set `a[0]=u`; (2) for `k=0,...,K-1`, compute `a[k+1]=sqrt(1+a[k])-1`; (3) initialize `r[K]=1/3`; (4) for `k=K-1,...,0`, compute `r[k]=(2*r[k+1]+2+0.5*a[k+1])/(2+a[k+1])^3`; (5) return `u-0.5*u*u+u^3*r[0]`. In code, fuse the square-root, residual-backward loop, polynomial reconstruction, and surrounding softplus or log-sum-exp operations into one CUDA or Triton kernel to avoid intermediate memory traffic. The formulas determine the recurrence and base residual; empirically estimate the smallest K meeting the target error by comparing against high-precision `log1p` over a dense grid and random samples, and add a fallback for inputs outside the safe interval or for subnormal and exceptional values. First benchmark a fused softplus and log-sum-exp kernel on an RTX-class GPU using tensors of sizes 1M, 16M, and transformer vocabulary logits, against PyTorch and CUDA implementations. Measure maximum and percentile absolute error, gradient error, kernel latency, achieved bandwidth, and end-to-end training throughput. A successful result is at least 1.2x primitive or fused-kernel speedup at comparable FP16/BF16 accuracy, with no overflow or visible loss increase in a small Transformer language-model run. Test stability especially for logits near zero, where cancellation affects direct polynomial approximations.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.