# Lorentz-Gram-preserving hyperbolic attention

- ID: 65
- Canonical URL: https://synthcore.org/idea/65/lorentz-gram-preserving-hyperbolic-attention
- API JSON: https://synthcore.org/api/idea/65.json
- API Markdown: https://synthcore.org/api/idea/65.md
- Verification status: mechanism_works
- Source: [arXiv:2608.22568](https://arxiv.org/abs/2608.22568)
- Category: geometry
- Solves: accuracy, stability, generalization
- ML areas: attention, graph-nn, embedding
- Math tags: geometry, metric-geometry, convex-analysis, probability
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace an unconstrained nonlinearity on hyperbolic pairwise similarities with a function from the paper's exact Lorentz–Gram preserver family. The transformed similarity matrix remains realizable as Lorentz inner products of future-directed unit timelike vectors, allowing a network to sharpen or smooth hyperbolic neighborhoods without introducing geometrically impossible pairwise relations.

## Mathematical statement

For a future-directed unit timelike vector x=(x_0,x_1,...,x_d) with Lorentz product \(\langle x,y\rangle_L=x_0y_0-\sum_{k=1}^d x_ky_k\), one has \(\langle x_i,x_j\rangle_L\ge1\). A Lorentz–Gram matrix is \(G_{ij}=\langle x_i,x_j\rangle_L\), with diagonal one and off-diagonal entries at least one, together with the appropriate Lorentz signature. The paper states that entrywise application preserves the class of Lorentz–Gram matrices exactly when \(f(t)=1+\gamma 1_{(1,\infty)}(t)+\beta(t^\alpha-1)/\alpha+\int_{(0,\infty)}(1-t^{-s})\,d\nu(s)\), where \(\alpha\in[0,1]\), \(\beta,\gamma\ge0\), and \(\nu\) is a positive Radon measure. At \(\alpha=0\), the power term is \(\log t\). For \(\alpha>0\), \(\beta\ge\alpha(1+\gamma+\nu((0,\infty)))\); for \(\alpha=0\), \(\int\min\{s,1\}\,d\nu(s)<\infty\). Use a finite positive mixture \(\nu=\sum_{m=1}^M q_m\delta_{s_m}\), with \(q_m,s_m\ge0\), as the trainable implementation.

## Key formulas

- $$f(t)=1+\gamma 1_{(1,\infty)}(t)+\beta\frac{t^{\alpha}-1}{\alpha}+\int_{(0,\infty)}(1-t^{-s})\,\mathrm{d}\nu(s),\qquad t\ge1,$$
- $$\frac{t^{\alpha}-1}{\alpha}=\log t\quad\text{if }\alpha=0,$$
- $$\beta\geq\alpha\bigl(1+\gamma+\nu((0,\infty))\bigr)\quad(\alpha\in(0,1]),\qquad \int_{(0,\infty)}\min\{s,1\}\,\mathrm{d}\nu(s)<\infty\quad(\alpha=0),$$
- $$\widetilde G_{ij}=f(G_{ij}),\qquad G_{ij}=\langle x_i,x_j\rangle_L=x_{i0}x_{j0}-\sum_{k=1}^{d}x_{ik}x_{jk}.$$

## Implementation notes

Integrate this as a hyperbolic similarity layer before attention or graph aggregation. Inputs are a batch of Lorentz points X with shape B×N×(d+1), normalized so every row satisfies x_0^2-||x_{1:d}||_2^2=1 and x_0>0. First compute G=X[:,:,0]X[:,:,0]^T-X[:,:,1:]X[:,:,1:]^T. Clamp off-diagonal values below 1 to 1+10^-6 for numerical stability and set the diagonal to one. Parameterize alpha=sigmoid(a) in (0,1), s_m=softplus(u_m), and q_m=softplus(v_m). With gamma=0, set beta=alpha*(1+sum_m q_m+epsilon), which enforces the theorem's sufficient constraint. Evaluate f(G)=1+beta*expm1(alpha*log(G))/alpha+sum_m q_m*(1-exp(-s_m*log(G))); use beta*log(G) when alpha is near zero. Subtract the rowwise maximum from f(G) and apply softmax to obtain attention weights, or pass f(G) directly as a message-passing kernel. If explicit coordinates are required, factorize the Lorentz matrix using the eigendecomposition of J*f(G), where J=diag(1,-1,...,-1), retaining one positive and the required negative eigenvalues. The mathematical guarantee comes from the preserver theorem; empirically monitor diagonal error, minimum off-diagonal value, and the number of incorrect-signature eigenvalues. Start with a two-dimensional synthetic hyperbolic classification problem and Cora link prediction using a hyperbolic GNN. Compare Euclidean attention, raw hyperbolic attention, and the constrained transform at equal hidden dimension and FLOPs. Success is higher validation accuracy or MRR, fewer invalid factorizations, and lower seed-to-seed variance.

## Verification

- Status: mechanism_works
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a trainable finite-mixture Lorentz–Gram-preserving attention layer with positive softplus mixture parameters and theorem-constrained beta. Across 12 numerical trials, the constrained transform preserved one-positive Lorentz signature in 100% of cases, while an unconstrained exponential transform did so in 0%; the trainable layer also passed a differentiability and shape smoke test. In the five-seed synthetic classification experiment, constrained attention achieved 1.000 accuracy with zero seed variance, matching Euclidean attention and improving over raw hyperbolic attention at 0.971 accuracy with 0.028 standard deviation.

### Mechanism check

- Verdict: Built a trainable finite-mixture Lorentz–Gram-preserving attention layer with positive softplus mixture parameters and theorem-constrained beta. Across 12 numerical trials, the constrained transform preserved one-positive Lorentz signature in 100% of cases, while an unconstrained exponential transform did so in 0%; the trainable layer also passed a differentiability and shape smoke test. In the five-seed synthetic classification experiment, constrained attention achieved 1.000 accuracy with zero seed variance, matching Euclidean attention and improving over raw hyperbolic attention at 0.971 accuracy with 0.028 standard deviation.
- Confidence: 8/10
- Limitations: Only a small synthetic node-classification task was tested; no Cora link prediction, explicit coordinate factorization, GPU benchmark, FLOP/speed comparison, or end-to-end training of the attention parameters was evaluated.

## Artifacts

- [experiment.py](https://synthcore.org/code/62/experiment.py)
- [report.md](https://synthcore.org/code/62/report.md)
- [results.json](https://synthcore.org/code/62/results.json)
- [Download all files as ZIP](https://synthcore.org/download/62)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.