✗ Failed on benchmark 2026

Error-budgeted local log-signature tokens

Implementation & benchmark of arXiv:2607.26281 — Concise $(\varepsilon,r)$-representations of a path

Usefulness7/10

Difficulty5/10

Novelty6/10

Memory Memory Scalability Accuracy Transformer Ssm Embedding Memory Tensor decomposition Approximation theory Dynamical systems Combinatorics

Source paper: Concise $(\varepsilon,r)$-representations of a path arXiv:2607.26281 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Encode a continuous or sampled multivariate time series as degree-N log-signatures on m local intervals, choosing (m,N) to minimize the number of stored scalars under an explicit linear-CDE approximation budget. Unlike fixed-size temporal patches, the representation increases temporal resolution when the path is oscillatory and increases algebraic degree when the path is smooth but strongly nonlinear.

Formulas

$$S^n(X)=\int_{0<u_1<\cdots<u_n<T} dX_{u_1}\otimes\cdots\otimes dX_{u_n}$$

$$dY=A Y\,dX,\qquad \lVert A\rVert\le r$$

$$R_N(z)=e^z-\sum_{k=0}^{N}\frac{z^k}{k!},\qquad E(m,N)=\sum_{j=1}^{m}R_N(r\ell_j)$$

$$C(m,N)=m\sum_{k=1}^{N}\ell_k(d),\qquad \ell_k(d)=\frac{1}{k}\sum_{q\mid k}\mu(q)d^{k/q},\qquad (m^*,N^*)=\arg\min_{m,N}C(m,N)\ \text{s.t.}\ E(m,N)\le\varepsilon$$

Mathematical statement

The paper represents a path X:[0,T]->R^d using iterated integrals S^n(X)=integral over 0<u_1<...<u_n<T of dX_{u_1} tensor-product ... tensor-product dX_{u_n}, and approximates the linear controlled differential equation dY=A Y dX, where Y is the hidden state and A is a linear control operator with operator norm |A|<=r. For a segment j with 1-variation length ell_j and degree N, the omitted Taylor-signature tail is bounded by R_N(r ell_j)=exp(r ell_j)-sum_{k=0}^N (r ell_j)^k/k!. A practical global error proxy is E(m,N)=sum_{j=1}^m R_N(r ell_j), which conservatively sums local propagation errors. If all m segments have equal variation ell=L/m, then E(m,N)<=m[exp(rL/m)-sum_{k=0}^N (rL/m)^k/k!], where L is total path variation. The storage cost of local truncated log-signatures is C(m,N)=m sum_{k=1}^N l_k(d), where l_k(d) is the dimension of the degree-k free-Lie component for d input channels; the Witt formula is l_k(d)=(1/k) sum_{q|k} mu(q)d^(k/q), with mu the Moebius function. Select the smallest C(m,N) subject to E(m,N)<=epsilon, where epsilon is the desired approximation budget.

Implementation notes

Integrate this at the sequence-to-token interface before a transformer, SSM, or neural CDE. Given a sampled sequence x[0:T] with d channels, first compute cumulative path variation v_t=sum_{s<t} ||x[s+1]-x[s]||_2. For each candidate interval count m in a small grid such as {1,2,4,8,16,32} and degree N in {1,...,5}, partition the sequence at equal-variation quantiles rather than equal time. On every segment j, compute the truncated log-signature z_j=(log S_j)_{1:N}; concatenate z_j with segment duration and endpoint increment, then project it to the model width with one shared linear layer. Estimate ell_j from observed variation in segment j and set r to a tunable sensitivity scale, initially r=1/(median segment norm) or a validation parameter. Compute R_N(r ell_j) using a recurrence for Taylor terms rather than direct factorials. Enumerate candidate (m,N), retain the lowest C(m,N) satisfying E(m,N)<=epsilon, and cache the selected representation during preprocessing. Pseudocode: v=variation(x); cuts=variation_quantiles(v,m); for j: ell[j]=segment_variation; z[j]=logsignature(x[cuts[j]:cuts[j+1]],N); E=sum(exp(r*ell[j])-taylor_exp(r*ell[j],N)); choose lowest_cost(E<=eps); tokens=Linear(concat(z,duration,delta_x)). The mathematical quantities are ell_j, R_N, C, and the feasibility test; log-signatures and r are computed numerically or estimated from data. First test on Speech Commands or a UEA multivariate time-series dataset with a 2-layer transformer, comparing fixed raw patches, fixed degree-2 signatures, and adaptive tokens at equal input-memory budgets. Measure validation accuracy, token count, training throughput, and accuracy at fixed GPU memory. Success means at least 20% fewer tokens or 1.5x longer sequences at no accuracy loss, with tighter epsilon producing a monotone accuracy improvement.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a readable MVP for recurrence-based Taylor tails, Witt/free-Lie storage cost, variation-quantile segmentation, local degree-1/2 log-signature-like features, budget selection, and a toy classifier. The mechanism manifested: tails decreased strictly with degree, equal-variation subdivision decreased E monotonically, and tightening epsilon monotonically increased selected cost. The recurrence matched factorial-reference tails to 4.5e-13 absolute error. The classification signal was inconclusive because fixed degree-2 signatures and adaptive tokens both reached 100% while raw patches reached 50.5%, so this is evidence for the mathematical mechanism, not a demonstrated ML win.

Agent confidence: 8/10

Baseline: Raw 8 local patches: accuracy 0.5048; fixed degree-2 signatures with 4 segments: accuracy 1.0000
Idea: Adaptive 8 local tokens: accuracy 1.0000; all 700 examples selected degree 2 at epsilon=0.08; runtime 4.14 s
Math predictions: Tail monotonicity: true over z in [0.05,8], N=1..5. Equal-partition E monotonicity: true over L in [0.1,10]. Tightening epsilon cost monotonicity: true for L={1,2,4,8}. Unequal/equal E ratio for ell={0.1,0.2,1.7,2.0}: 4.26.

Limitations: The toy data is small and easily separable; no transformer, Speech Commands, UEA dataset, throughput-at-equal-memory, or GPU benchmark was tested. The feature implementation includes exact degree-1 and antisymmetric degree-2 log-signature coordinates only; degrees 3-5 are used in the mathematical selector but not fully encoded. The unequal-segment proxy can be substantially larger than the equal-variation expression, as expected from the conservative local sum.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.26281) Sep 2, 2026

Significantly WORSE than the tuned baseline (+21.2%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.7327
Idea mean: 0.8882
Effect (Δ): 0.1555 (+21.2%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±12.6%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Built an end-to-end local log-signature token interface for the sequence transformer benchmark, using variation-quantile segmentation, degree-2 increments/area features, and a shared token transformer. The Taylor-tail monotonicity and equal-variation subdivision predictions passed numerically, but the idea lost significantly on test MSE: 0.8882 versus 0.7327 for baseline, paired delta +0.1555 with permutation p=0.0081. The trained-model mechanism probe had only moderate seed-level correlation (0.324) and was correctly marked unconfirmed; therefore there is no demonstrated ML benefit.

Baseline: Tuned fixed raw patch tokens, best cfg lr=0.003 m=8; full 8-seed test MSE mean 0.7327230275, std 0.0837788024, per-seed [0.6893054, 0.6570264, 0.6207273, 0.8268932, 0.7129261, 0.7505295, 0.8937446, 0.7106316]
Idea: Adaptive variation-quantile local log-signature tokens, selected cfg lr=0.001 m=4; full 8-seed test MSE mean 0.8882134855, std 0.0530509465, per-seed [0.9650674, 0.8446773, 0.9014533, 0.8312017, 0.9647781, 0.8639495, 0.9117945, 0.8227863]
Math sanity: Tail decreases with degree: true; equal-variation error decreases with m: true; degree-2 free-Lie dimensions for d=2: [2,1,2,3,6]
Mechanism signature: Predicted local Taylor-tail proxy mean 572838.6955; observed trained-system test MSE mean 0.9076; seed-level correlation 0.3244; confirmed=false
Paired delta: idea - baseline = +0.1554904580 MSE; idea wins 0/8; permutation p=0.0081; verdict: idea worse (significant)

Benchmark result