Hyperbola-Tangent Quadratic Features

Integrate the module at a multiplicative interaction point. Let z1 and z2 be two learned feature tensors with identical shape, such as an attention gate and a scale feature or two outputs of an MLP projection. Map them into the bounded coordinates of the paper using x1 = 0.5 + 1.5 sigmoid(z1) and x2 = 0.5 + 1.5 sigmoid(z2). Select K values v_k logarithmically spaced in [0.5, 2], optionally including reciprocal pairs. Compute q_k = (x1 + v_k^2 x2 - 2 v_k)^2 for every k. Either concatenate the K channels to the local representation and apply a learned linear projection, or add L_tan = lambda E[min_k q_k] to the training loss when the task expects each pair to approach some reciprocal operating point. Compact pseudocode is: x1 = .5 + 1.5 * sigmoid(z1); x2 = .5 + 1.5 * sigmoid(z2); q = []; for v in grid: q.append((x1 + v*v*x2 - 2*v)**2); h_new = Linear(concat(h, q)). The paper supplies the tangent construction and nonnegativity property; K, lambda, the grid, and feature-versus-penalty usage must be selected empirically. Normalize each q channel by its batch standard deviation and remove channels whose normalized outputs correlate above 0.999, avoiding redundant affine-square directions. First test a parameter-matched two-layer MLP on synthetic reciprocal regression and a small attention-gating task. Compare against a standard MLP and a bilinear feature baseline at equal parameter count. Measure validation error, loss smoothness, gradient-norm spikes, and robustness when inputs move toward the boundary x1 x2 = 1. Success means lower error or fewer unstable runs at comparable FLOPs, particularly on reciprocal or scale-invariant data.