{
 "artifacts": null,
 "category": "architecture",
 "description": "Parameterize a learned token metric as a nonnegative sum of sparse integral rank-one projections with unimodular support, rather than learning an unconstrained dense positive-semidefinite matrix. Graph-incidence covectors give an immediately implementable support family, while nonnegative coefficients guarantee positive semidefiniteness by construction.",
 "formulas_latex": [
  "$$Q_{\\Gamma}=\\sum_{f\\in E(\\Gamma)}\\ell_f a_f^{2},$$",
  "$$Q=\\sum_{e\\in E}\\ell_e a_e^{2},\\qquad \\ell_e\\ge 0,$$",
  "$$M=A\\operatorname{diag}(\\ell)A^T+\\varepsilon I_d,\\qquad x^TMx=\\sum_{e=1}^{m}\\ell_e(a_e^Tx)^2+\\varepsilon\\|x\\|_2^2,$$",
  "$$s_{ij}=-\\frac{1}{2\\tau}(z_i-z_j)^TM(z_i-z_j),\\qquad \\alpha_{ij}=\\operatorname{softmax}_{j}(s_{ij})$$"
 ],
 "id": 2789,
 "implementation": "(1) Integration point: replace the usual dot-product attention score, or a dense Mahalanobis distance in a metric-learning head, with the quadratic distance defined above. Let $z_i\\in\\mathbb{R}^d$ be the projected representation of token $i$. Fix a sparse covector matrix $A\\in\\mathbb{R}^{d\\times m}$, and learn only the coefficient vector $\\theta\\in\\mathbb{R}^m$ with $\\ell_e=\\operatorname{softplus}(\\theta_e)$; use a small fixed $\\varepsilon$, such as $10^{-4}$. (2) Pseudocode: compute `u = einsum('dm,bnd-\u003ebnm', A, z)`, so `u[b,n,e] = a_e^T z[b,n]`; compute `w = softplus(theta)`; for every query-key pair compute `delta = u[:, :, None, :] - u[:, None, :, :]`, `dist = (delta**2 * w).sum(-1) + eps * ((z[:, :, None, :] - z[:, None, :, :])**2).sum(-1)`; set `logits = -dist/(2*tau)` and apply the attention softmax. This is exactly $x^TMx=\\sum_e\\ell_e(a_e^Tx)^2+\\varepsilon\\|x\\|^2$ applied to pairwise differences. (3) Computed from the paper: coefficient nonnegativity, the rank-one PSD decomposition, and the unimodular support. Estimated empirically: $\\tau$, $\\varepsilon$, the number of support columns $m$, and whether coefficient pruning is beneficial. Initially restrict $A$ to graph-incidence columns, for which total unimodularity is guaranteed; optionally prune columns with $\\ell_e\u003c\\delta$ and compare the resulting sparse model. (4) First experiment: train a small 2-layer, 8-head Transformer on character-level Penn Treebank or CIFAR-10 represented as patch tokens. Compare standard dot-product attention, a dense learned Mahalanobis metric, and this matroidal metric at matched hidden dimension and approximately matched parameter count. Sweep $m\\in\\{d,2d,4d\\}$ and measure validation loss or accuracy, gradient-norm variance, attention-logit numerical failures, parameter count, and measured attention FLOPs. The hypothesis is that the structured metric is most useful in low-data or low-parameter settings: success means equal or better validation loss with fewer metric parameters, smoother training, and no loss of stability relative to the dense baseline.",
 "math_summary": "The paper defines a unimodular family of integral covectors $(a_e)_{e\\in E}\\subseteq L^*$ by requiring that, after choosing an integral basis of the rank-$g$ lattice $L$, every maximal minor of the matrix with columns $a_e$ is $0$ or $\\pm1$. A quadratic form is matroidal when it has the decomposition $Q=\\sum_{e\\in E}\\ell_e a_e^2$, where $\\ell_e\\ge0$ and $a_e^2(x)=\\langle a_e,x\\rangle^2$. In coordinates, let $A=[a_1,\\ldots,a_m]\\in\\mathbb{R}^{d\\times m}$ and let $\\ell=(\\ell_1,\\ldots,\\ell_m)$ with nonnegative entries. Then $Q(x)=x^TA\\operatorname{diag}(\\ell)A^Tx$ and the matrix $M=A\\operatorname{diag}(\\ell)A^T$ is positive semidefinite because $x^TMx=\\sum_e\\ell_e(a_e^Tx)^2\\ge0$. Add $\\varepsilon I_d$, with $\\varepsilon\u003e0$, when strict positive definiteness is required. The paper's characterization says that nonnegative conorms are equivalent to matroidality and that positive-conorm support determines a regular matroid; the implementation uses the constructive decomposition directly. For a graph support, use oriented incidence covectors $a_{(u,v)}=e_u-e_v$ after deleting one redundant vertex coordinate. Incidence matrices are totally unimodular, so all maximal minors satisfy the required $0,\\pm1$ condition.",
 "math_tags": [
  "linear-algebra",
  "combinatorics",
  "graph-theory",
  "geometry"
 ],
 "ml_areas": [
  "attention",
  "embedding",
  "graph-nn",
  "regularization"
 ],
 "paper": {
  "arxiv_id": "2608.28783",
  "arxiv_url": "https://arxiv.org/abs/2608.28783",
  "summary_what_math_gives_to_ml": "The paper gives a constructive characterization of a class of positive-definite quadratic forms as nonnegative sums of integral rank-one forms whose covectors have unimodular, regular-matroid support. This suggests replacing an unconstrained dense Mahalanobis metric with a structured PSD metric whose positivity is automatic and whose support can be sparse and interpretable. The most direct neural-network test is to use such a metric for attention distances or metric-learning embeddings, with graph-incidence covectors providing a simple totally unimodular support. The likely benefits are reduced metric parameters, stable optimization, and useful inductive bias on small or graph-structured data.",
  "title": "Nonnegative conorms, regular matroids, and the tropical Schottky problem",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 5
 },
 "solves": [
  "stability",
  "memory",
  "generalization"
 ],
 "title": "Matroidal Mahalanobis Attention",
 "url": "https://synthcore.org/idea/2789/matroidal-mahalanobis-attention",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}