# Stieltjes-Newton Feature Head

- ID: 148
- Canonical URL: https://synthcore.org/idea/148/stieltjes-newton-feature-head
- API JSON: https://synthcore.org/api/idea/148.json
- API Markdown: https://synthcore.org/api/idea/148.md
- Verification status: unverified
- Source: [arXiv:2608.24884](https://arxiv.org/abs/2608.24884)
- Category: architecture
- Solves: accuracy, scalability, sample-efficiency
- ML areas: mlp, embedding, initialization
- Math tags: approximation-theory, measure-theory, linear-algebra
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Replace raw polynomial or Fourier features for a scalar coordinate x with iterated Stieltjes-integral features generated by a monotone data-adaptive coordinate g. Use a Newton-form basis so the model can add higher-order terms or new interpolation knots without recomputing all previous coefficients.

## Mathematical statement

The paper defines the pseudometric d_g(x,y)=|g(x)-g(y)|, so inputs are close when their g-coordinates are close. Define Stieltjes monomials recursively by g_{y0,0}(x)=1 and g_{y0,k+1}(x)=∫_{y0}^{x} g_{y0,k}(t) dg(t), where y0 is a base point and dg is the Stieltjes measure induced by g. The generalized polynomial space is P_n=span{1,g_{y0,1},...,g_{y0,n}}. The paper states that {1,N[x1],N[x1,x2],...,N[x1,...,xn]} is a basis of P_n, where N denotes generalized Newton basis functions, and that this basis induces a lower-triangular interpolation matrix. Thus a new node can be added without recomputing all previous coefficients. The interpolation matrix uses rows (1,g_{y0,1}(x_i),...,g_{y0,n}(x_i)); for g-ordered distinct nodes its determinant is positive. In the neural adaptation, g is a nondecreasing scalar function, z=g(x), and the continuous special case gives g_{y0,k}(x)=(z-z0)^k/k!. Jumps can be retained by numerically evaluating the Stieltjes recurrence.

## Key formulas

- $$d_g(x,y)=\left\lvert g(x)-g(y)\right\rvert$$
- $$g_{y_0,0}(x)=1,\qquad g_{y_0,k+1}(x)=\int_{y_0}^{x}g_{y_0,k}(t)\,dg(t),\qquad \operatorname{P}_n=\operatorname{span}\{1,g_{y_0,1},\ldots,g_{y_0,n}\}$$
- $$\{1,N[x_1],N[x_1,x_2],\ldots,N[x_1,\ldots,x_n]\}\text{ is a basis of }\operatorname{P}_n$$
- $$h_{m+1}(x)=h_m(x)+c_{m+1}N[x_1,\ldots,x_m,x],\qquad c_{m+1}=f[x_1,\ldots,x_{m+1}]$$

## Implementation notes

Integrate this as a small feature head before an MLP, or as the scalar-coordinate branch of a coordinate network. For each scalar input x, first fit or learn a monotone coordinate g. A cheap fixed choice is the clipped empirical CDF g(x)=rank(x)/(N+1); a learnable choice is a monotone rational-quadratic spline with positive bin widths and heights, initialized to the identity. Let z=g(x), choose ordered knots z_1<...<z_K, and compute generalized Newton features recursively. In the continuous case use B_0(z)=1 and B_{j+1}(z)=(z-z_{j+1})B_j(z). For a genuinely Stieltjes version, approximate the integral recurrence by quadrature over sorted training values and include jump increments Δg(t). Concatenate B_0,...,B_K with the original hidden representation and apply a learned linear projection. Initialize the constant coefficient and add one feature every few epochs using the Newton update h ← h+c_j B_j, freezing old coefficients during the first validation interval. Normalize every feature by its training-set RMS and regularize the spline derivative toward a positive bounded range. The paper supplies the basis construction and lower-triangular incremental update; estimate g, knot locations, feature scales, and coefficients empirically. First test on one-dimensional Friedman regression and a UCI tabular regression dataset with a two-layer MLP baseline at equal parameter count. Compare the raw MLP, ordinary polynomial features, Fourier features, and Stieltjes-Newton features. Success means lower validation MSE at equal parameters, faster loss reduction at equal FLOPs, and an advantage when samples are highly nonuniform or contain sharp distributional jumps.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.