Unverified 2026

TT Cubic Sign Router

Implementation & benchmark of arXiv:2608.17135 — Iterative tensor network transformations for element-wise evaluation of elementary and filtering functions

Usefulness6/10

Difficulty6/10

Novelty7/10

Architecture Memory Inference speedup Scalability Moe routing Quantization Memory Tensor decomposition Approximation theory Algebra Numerical analysis

Source paper: Iterative tensor network transformations for element-wise evaluation of elementary and filtering functions arXiv:2608.17135 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace explicit thresholding of a large structured routing-logit tensor with the paper's cubic sign iteration applied directly to its TT cores. The resulting approximate sign field can be converted into binary expert gates or binary activation masks without materializing all tensor entries, enabling routing over exponentially large Cartesian index spaces. Use TT rounding after each Hadamard product to prevent rank explosion, and compare the method against dense thresholding on smaller instances where exact outputs are available.

Formulas

$$R^{(k+1)}=F\left(R^{(k)}\right),\qquad R^{(0)}=\Psi,$$

$$F(R^{(k)})=-\frac{1}{2}\left((R^{(k)})^{\odot 3}-3R^{(k)}\right),$$

$$|\Psi^{[j]}|<\sqrt{3}\ \ (\forall j),\qquad \mathrm{Sign}(x)=\begin{cases}1,&x>0\\0,&x=0\\-1,&x<0\end{cases},$$

$$R^{(0)}=\frac{L}{a},\qquad R^{(k+1)}=-\frac12\left((R^{(k)})^{\odot3}-3R^{(k)}\right),\qquad G=\frac{1+R^{(K)}}{2}.$$

Mathematical statement

The paper defines the element-wise sign function by comparing a TT tensor $\Psi$ with the zero tensor, where $\mathrm{Sign}(x)=1$ for $x>0$, $0$ for $x=0$, and $-1$ for $x<0$. It computes this using $R^{(0)}=\Psi$ and $R^{(k+1)}=F(R^{(k)})$, with $F(R)=-\frac{1}{2}(R^{\odot 3}-3R)$; $R^{\odot 3}$ means element-wise cubing, not a matrix or tensor contraction. The iteration converges element-wise to $\mathrm{Sign}(\Psi)$ when every entry satisfies $|\Psi^{[j]}|<\sqrt{3}$, where $j$ indexes tensor entries. For a neural router, let $L$ be a TT-compressed tensor of routing logits, let $a>\max_j|L^{[j]}|/\sqrt{3}$ be a safety scale, and initialize $R^{(0)}=L/a$. Then $R^{(K)}$ approximates $\mathrm{Sign}(L)$, and a binary gate is $G=(1+R^{(K)})/2$. Every addition and scalar multiplication is performed core-wise; every Hadamard cube is implemented by TT core products followed by TT-rounding to tolerance $\varepsilon$.

Implementation notes

1. Integration point: use this in a TT-compressed MoE router or binary activation layer. Let the router produce a structured logit tensor $L$ over a Cartesian product of token, spatial, feature, or expert-index modes, stored as TT cores rather than a dense tensor. The gate should be computed at inference time first; training can use a straight-through estimator or backpropagate through the polynomial iterations. 2. Pseudocode: estimate $a=(1+\delta)\max_j|L^{[j]}|/\sqrt{3}$, using exact dense max only for small validation tensors or a TT max/upper bound for production; set $R=L/a$. For $k=1,\ldots,K$, compute $Q=R\odot R$, round TT$(Q)$ to relative tolerance $\varepsilon$, compute $C=Q\odot R$, round TT$(C)$, and set $R=1.5R-0.5C$. Return $G=(1+R)/2$, optionally thresholding $G>0.5$ to obtain a binary route. 3. Mathematical versus empirical quantities: the cubic map and convergence domain come from the paper; $K$, TT-rounding tolerance $\varepsilon$, safety factor $\delta$, and whether to use straight-through gradients must be tuned. Track TT ranks after each product, maximum gate error, and the fraction of entries near zero. If logits are too close to zero, add a learned or fixed margin $m$ and apply the iteration to $L-m$, or use a soft surrogate during early training. 4. First experiment: construct a small TT router for a 4D or 6D synthetic activation tensor and a 2-layer MLP-MoE on MNIST or CIFAR-10. Compare dense sign/top-1 routing, sigmoid routing, and TT cubic routing for $K\in\{2,4,6,8\}$, several $\varepsilon$, and equal routing decisions. Success means at least 4x lower peak activation memory or substantially larger addressable routing tensors at similar accuracy, with TT gate error below 1% on non-margin entries and lower inference cost than densifying the tensor.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.