SynthCore
Unverified 2026

Discrepancy-bounded signed fast memory

Implementation & benchmark of arXiv:2608.12490 — Online balancing of vectors with small coordinates

Usefulness5/10
Difficulty5/10
Novelty9/10
Architecture StabilityMemoryScalabilityRnnSsmAttentionEmbeddingCombinatoricsProbabilityOptimizationDynamical systems

Source paper: Online balancing of vectors with small coordinates arXiv:2608.12490 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an unrestricted additive recurrent or fast-weight memory with a sign-selectable update: for each incoming update vector, choose between adding and subtracting it so that a smooth compact potential of the memory state is minimized. This is appropriate when the memory representation has sign symmetry, such as signed random features or a learned linear sketch; it is not a drop-in replacement for ordinary gradient updates where the sign carries semantic information.

Formulas

$$\mathbb{P}\left\{\max_{k\leqslant T}\left\lVert\sum_{t=1}^{k}\varepsilon_{t}v_{t}\right\rVert_{\infty}>6L\right\}\leqslant CT\exp\!\left[-\frac{cd}{\ln^{2}(ed)}\right],\qquad \|v_t\|_2\le 1,\quad \|v_t\|_\infty\le d^{-1/2}.$$
$$d\ge C\ln\frac{3T}{\varepsilon}\left[\ln\left(e+\ln\frac{3T}{\varepsilon}\right)\right]^2\quad\Longrightarrow\quad \mathbb P\left\{\max_{k\le T}\left\|\sum_{t=1}^k\varepsilon_tv_t\right\|_\infty\le 6L\right\}\ge1-\varepsilon.$$
$$\Phi_\lambda(s)=\sum_{i=1}^{m}\cosh(\lambda s_i),\qquad \varepsilon_t=\operatorname*{argmin}_{\sigma\in\{-1,+1\}}\Phi_\lambda(s_{t-1}+\sigma v_t),\qquad s_t=s_{t-1}+\varepsilon_t v_t.$$
$$v_t=\frac{u_t}{\max\{1,\|u_t\|_2,\sqrt d\,\|u_t\|_\infty\}},\qquad \|v_t\|_2\le1,\quad \|v_t\|_\infty\le d^{-1/2}.$$

Mathematical statement

The paper proves that for fixed vectors $v_1,\ldots,v_T\in B_2^m$ revealed online, with $\|v_t\|_\infty\le d^{-1/2}$, there is an online choice $\varepsilon_t\in\{-1,1\}$ such that $\Pr\{\max_{k\le T}\|\sum_{t=1}^k\varepsilon_t v_t\|_\infty>6L\}\le CT\exp[-cd/\ln^2(ed)]$, for absolute constants $L,C,c>0$. Here $m$ is the state dimension, $T$ is the number of sequential updates, $B_2^m=\{x\in\mathbb R^m:\|x\|_2\le1\}$, and $d$ is the inverse squared maximum coordinate size. The consequence is constant prefix discrepancy with failure probability at most $\varepsilon$ when $d\ge C\ln(3T/\varepsilon)[\ln(e+\ln(3T/\varepsilon))]^2$. The extracted text does not specify the paper's full randomized signing algorithm, so the neural adaptation uses the stated compact-potential method with $\Phi_\lambda(s)=\sum_i\cosh(\lambda s_i)$ and chooses the lower-potential sign. For candidate update $u_t$, define $v_t=u_t/\max\{1,\|u_t\|_2,\sqrt d\|u_t\|_\infty\}$, giving $\|v_t\|_2\le1$ and $\|v_t\|_\infty\le d^{-1/2}$. The controlled state is $s_t=s_{t-1}+\varepsilon_t v_t$.

Implementation notes

(1) Integration point: add this as an optional signed fast-memory branch inside a recurrent layer, state-space model, linear-attention state, or fast-weight module. Let $u_t\in\mathbb R^m$ be the candidate additive memory update produced from token $x_t$, for example $u_t=W_vx_t\odot\sigma(W_kx_t)$, and let $s_{t-1}$ be the persistent state. This module is intended for representations where either sign of an update is admissible; retain an ordinary unsigned branch as a baseline.

(2) Pseudocode:

s = zeros(m)
for t in range(T):
    u = update_network(x[t])
    v = u / max(1, l2_norm(u), sqrt(d)*max_abs(u))
    phi_plus  = sum(cosh(lam*(s + v)))
    phi_minus = sum(cosh(lam*(s - v)))
    eps = +1 if phi_plus <= phi_minus else -1
    s = s + eps*v
    y[t] = readout_network(s, x[t])

Use $\lambda\in\{0.1,0.25,0.5\}$ initially and $d\in\{16,64,256\}$. For numerical stability, compare potentials using $\cosh(a+b)-\cosh(a-b)=2\sinh(a)\sinh(b)$ instead of evaluating large absolute values.

(3) The implementation computes normalization, candidate potential changes, and the sign directly. The theorem supplies the falsifiable target: under diffuse normalized updates, $D=\max_{k\le T}\|s_k\|_\infty$ should remain near a constant, with failure probability decreasing roughly as $T\exp[-cd/\ln^2(ed)]$. Estimate this curve over many sequences. Also compare deterministic argmin signing with a randomized rule whose probabilities are proportional to $\exp(-\tau\Phi_+)$ and $\exp(-\tau\Phi_-)$.

(4) First experiment: use a 128-dimensional fast-memory model on sequential-MNIST or permuted sequential-MNIST, with sequence lengths 1k, 4k, and 16k. Compare ordinary additive state, norm-clipped additive state, and discrepancy-bounded signed state at equal parameter count and approximately equal FLOPs. Measure validation accuracy, maximum coordinate magnitude, saturation frequency, gradient norm, and loss degradation as sequence length grows. Success means smaller $D$, fewer saturated or exploding states, and improved long-context accuracy without more than 10% runtime overhead. Separately inject diffuse random updates with controlled $d$ to verify the predicted dependence on $d/\ln^2(ed)$.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.

Related ideas