✓✓ Beats tuned baseline 2026

Reversible Mealy Token Mixer

Implementation & benchmark of arXiv:2607.06942 — Invariant Measures for Soliton Systems Generated by Mealy Automata

Usefulness6/10

Difficulty5/10

Novelty8/10

Architecture Stability Memory Scalability Ssm Transformer Memory Training Dynamical systems Probability Combinatorics Algebra

Source paper: Invariant Measures for Soliton Systems Generated by Mealy Automata arXiv:2607.06942 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a recurrent token-mixing operation with a finite-state carrier scan over binary or quantized token features. The local transition table is constrained to conserve a scalar token weight and to be bijective, making the mixer reversible, constant-memory, and less prone to activation drift than a generic recurrent layer.

Formulas

$$w(q)+w(s)=w(\varphi(q,s))+w(\psi(q,s)),$$

$$w(q)+s=w(\varphi(q,s))+\psi(q,s)\qquad (S=\{0,1\}),$$

$$Q_n=\lim_{k\to\infty}\varphi\bigl(q;\eta_{n-k}\eta_{n-k+1}\cdots\eta_n\bigr),$$

$$\bigl(T_{\mathscr A}(\eta)\bigr)_n=\psi(Q_{n-1},\eta_n).$$

Mathematical statement

The paper defines a Mealy automaton $\mathscr A=(Q,S,\varphi,\psi)$, where $Q$ is a finite carrier-state set, $S$ is the input/output alphabet, $\varphi:Q\times S\to Q$ updates the carrier, and $\psi:Q\times S\to S$ emits the transformed symbol. Particle preservation is the exact local conservation law from Definition II.1, $w(q)+w(s)=w(\varphi(q,s))+w(\psi(q,s))$, where $w$ assigns a scalar weight to carrier states and symbols and is nonconstant on $S$. For binary symbols, the paper notes that one may normalize $w(s)=s$, so the constraint becomes $w(q)+s=w(\varphi(q,s))+\psi(q,s)$. On a bi-infinite sequence $\eta\in\{0,1\}^{\mathbb Z}$, the carrier state entering position $n$ is defined by the stabilized left scan $Q_n=\lim_{k\to\infty}\varphi(q;\eta_{n-k}\eta_{n-k+1}\cdots\eta_n)$, provided the limit exists and is independent of the initial carrier $q$. The induced sequence map is $(T_{\mathscr A}(\eta))_n=\psi(Q_{n-1},\eta_n)$. We adapt this to finite sequences by initializing $Q_0=q_0$, scanning left to right, and enforcing bijectivity of the pair map $(q,s)\mapsto(\varphi(q,s),\psi(q,s))$ over $Q\times S$. The conservation law guarantees exact preservation of total symbol-plus-carrier weight, while bijectivity supplies an inverse scan for reversible training or checkpoint-free backpropagation. As an optional invariant-statistics diagnostic, estimate input pair frequencies $\hat P_{ab}$ and output pair frequencies $\hat P'_{ab}$, and penalize $\|\hat P'-\hat P\|_2^2$.

Implementation notes

1. Integration point: insert the mixer after a token projection and before the feed-forward sublayer in a small Transformer, or use it as a standalone SSM-like residual branch. Quantize one feature channel per token to $s_n\in\{0,1\}$ using a straight-through threshold, or use several independent binary channels. Use $Q=\{0,\ldots,K-1\}$, initially $K=3$, matching the paper's automata. 2. Construct a lookup table containing $(\varphi(q,s),\psi(q,s))$ for every $(q,s)$. Enumerate candidate tables offline and retain only tables for which the pair map is bijective and there exist carrier weights $c_q$ satisfying $c_q+s=c_{\varphi(q,s)}+\psi(q,s)$. During an initial experiment, select one valid table randomly or learn a soft mixture over valid tables with Gumbel-softmax, then project to the nearest valid table after each epoch. 3. Forward pseudocode: set $q=q_0$; for $n=1,\ldots,L$, read $s=s_n$, set $(q_{new},y_n)=table[q,s]$, optionally add a learned embedding $e_{q,s}$ to the continuous token vector, then set $q=q_{new}$. Return the residual output $x_n+W_o e_{q_{n-1},s_n}$. For reverse-mode memory savings, reconstruct the previous pair by applying the inverse lookup table while scanning from the final carrier state. 4. Compute from the paper's mathematics: exact local conservation and inverse-table validity. Estimate empirically: output mean, pair-frequency drift, gradient norm, and language-model loss. Add $\lambda\|\hat P'-\hat P\|_2^2$ only if statistics drift significantly; $\hat P_{ab}$ is the minibatch frequency of adjacent binary pairs $(a,b)$. 5. First experiment: compare a one-layer 3-state mixer against a GRU with matched hidden-state size and against a depthwise-convolution token mixer on TinyStories or CIFAR-10 sequence patches. Measure validation loss or accuracy, peak activation memory, gradient explosions, and throughput for sequence lengths 512, 2K, and 8K. A successful result is stable training without gradient clipping, lower memory than the GRU, and equal or better loss at matched parameters; for language modeling, also test whether the reversible scan permits at least 1.5x longer sequences at fixed memory.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reversible 3-state Mealy mixer using the paper’s BBS-C(2) transition table. All six local transitions satisfy exact particle conservation and form a bijection; exhaustive inverse testing over 8,191 binary words up to length 12 passed, as did a 4,096-token random scan. The claimed constant-carrier-state property is demonstrated, but no language-model or optimization win was established.

Agent confidence: 8/10

Baseline: GRU hidden-state output: 384, 1,536, 6,144, and 24,576 activation elements for lengths 128, 512, 2,048, and 8,192; CPU forward times 0.0205, 0.0831, 0.3121, and 1.4279 seconds.
Idea: Exact local conservation errors all zero; local pair map bijective; exhaustive inverse check passed for 8,191 words; 4,096-token inverse reconstruction exact; total conservation error 0; one final carrier state is sufficient for reverse scanning; pair-frequency L2 drift 0.000345.

Limitations: The test uses a fixed discrete BBS-C(2) table rather than learned continuous token features, and does not measure language-model loss, gradient behavior, checkpoint-free backpropagation, GPU throughput, peak allocator memory, or matched-parameter training. The NumPy scan is a correctness prototype, not an optimized production mixer; the GRU activation comparison is a simple per-token output proxy rather than instrumented autograd memory.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06942) Sep 4, 2026

Beats the tuned baseline by 54.2% (p=0.0163, wins 7 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0014
Idea mean: 0.0007
Effect (Δ): -0.0008 (−54.2%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.0163 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±26.0%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested using 400 training and 200 test examples with an 8-step window. Activation memory, reversible backpropagation, sequence lengths 512-8192, language modeling, GPU throughput, and broader architecture validation were not measured. The implementation uses hard binary thresholding and is an MVP.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Worked ✓

Agent confidence: 8/10

The reversible 3-state Mealy mixer was evaluated on the structurally matched controlled-pendulum dynamics track against the standard rnn_small GRU under paired seeds and equal training budgets. It significantly improved test MSE: 0.0006506 versus 0.0014198, delta_mean=-0.0007692 with p=0.0163 and 7/8 wins. The trained-model conservation signature was also confirmed exactly, although memory savings and long-sequence scaling were not tested.

Baseline: Mean test MSE 0.0014198302 ± 0.0003983911; tuned lr=0.01, epochs=12.
Idea: Mean test MSE 0.0006506168 ± 0.0003999347; lr=0.01, scale=0.5, epochs=12; paired delta=-0.0007692134, p=0.0163.

Benchmark result