# Hyperbola-Tangent Quadratic Features

- ID: 207
- Canonical URL: https://synthcore.org/idea/207/hyperbola-tangent-quadratic-features
- API JSON: https://synthcore.org/api/idea/207.json
- API Markdown: https://synthcore.org/api/idea/207.md
- Verification status: unverified
- Source: [arXiv:2608.26639](https://arxiv.org/abs/2608.26639)
- Category: architecture
- Solves: accuracy, stability, generalization
- ML areas: mlp, attention, regularization
- Math tags: convex-analysis, geometry, optimization
- Ratings: usefulness 5/10; difficulty 3/10; novelty 7/10

## Idea description

Add a bank of quadratic features encoding tangent contact with the reciprocal manifold x1 x2 = 1, rather than forcing a generic MLP to discover this interaction from arbitrary monomials. For positive bounded feature pairs, each feature is nonnegative and becomes exactly zero at a selected reciprocal operating point. The module can be used either as an input feature expansion or as a regularizer encouraging learned gates and scales to follow a reciprocal geometry.

## Mathematical statement

The paper studies G = { (x1,x2) : x1 >= 1/2, x2 >= 1/2, x1 x2 <= 1 } and its curved boundary Gamma = { (u,u^(-1)) : u in [1/2,2] }. At the boundary point (v,v^(-1)), the tangent line is tau_v(x) = x1 + v^2 x2 - 2v. The square q_v(x) = tau_v(x)^2 is nonnegative everywhere and satisfies q_v(v,v^(-1)) = 0. The paper identifies such quadratic squares as extreme rays of the cone of quadratic polynomials nonnegative on G. Here x1 and x2 are bounded positive neural features, v is a selected operating point in [1/2,2], and q_v is either concatenated to a representation or included in a penalty. The additional identity g^1_{v,(2+3v)/4}(x) = (x1 + (v/2)x2 - v - 1/2)^2 shows that some parameterized rays reduce to affine squares, so duplicate feature directions should be removed.

## Key formulas

- $$\mathcal{G}=\left[\tfrac{1}{2},2\right]^2\cap\{(x_1,x_2):x_1x_2\leq 1\}=\{(x_1,x_2):x_1\ge\tfrac{1}{2},\ x_2\ge\tfrac{1}{2},\ x_1x_2\leq 1\}.$$
- $$\Gamma=\{(u,u^{-1}):\tfrac{1}{2}\leq u\leq 2\},\qquad \tau_v(x)=x_1+v^2x_2-2v=0.$$
- $$q_v(x)=\tau_v(x)^2=\left(x_1+v^2x_2-2v\right)^2\geq 0,\qquad q_v(v,v^{-1})=0.$$
- $$g^{1}_{v,(2+3v)/4}(\mathbf{x})=\left(x_{1}+\frac{v}{2}x_{2}-v-\frac{1}{2}\right)^{2}.$$

## Implementation notes

Integrate the module at a multiplicative interaction point. Let z1 and z2 be two learned feature tensors with identical shape, such as an attention gate and a scale feature or two outputs of an MLP projection. Map them into the bounded coordinates of the paper using x1 = 0.5 + 1.5 sigmoid(z1) and x2 = 0.5 + 1.5 sigmoid(z2). Select K values v_k logarithmically spaced in [0.5, 2], optionally including reciprocal pairs. Compute q_k = (x1 + v_k^2 x2 - 2 v_k)^2 for every k. Either concatenate the K channels to the local representation and apply a learned linear projection, or add L_tan = lambda E[min_k q_k] to the training loss when the task expects each pair to approach some reciprocal operating point. Compact pseudocode is: `x1 = .5 + 1.5 * sigmoid(z1); x2 = .5 + 1.5 * sigmoid(z2); q = []; for v in grid: q.append((x1 + v*v*x2 - 2*v)**2); h_new = Linear(concat(h, q))`. The paper supplies the tangent construction and nonnegativity property; K, lambda, the grid, and feature-versus-penalty usage must be selected empirically. Normalize each q channel by its batch standard deviation and remove channels whose normalized outputs correlate above 0.999, avoiding redundant affine-square directions. First test a parameter-matched two-layer MLP on synthetic reciprocal regression and a small attention-gating task. Compare against a standard MLP and a bilinear feature baseline at equal parameter count. Measure validation error, loss smoothness, gradient-norm spikes, and robustness when inputs move toward the boundary x1 x2 = 1. Success means lower error or fewer unstable runs at comparable FLOPs, particularly on reciprocal or scale-invariant data.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.