Replace repeated substrings in long sequences with nonterminal symbols from an acyclic straight-line grammar, then run the transformer on the compressed sequence. Unlike ordinary fixed tokenization, the compression objective explicitly minimizes the number of reusable binary productions, allowing repeated document-specific or corpus-level motifs to become single units. An expansion map lets the model recover token-level outputs for selected positions.
Replace a uniformly sampled trajectory sequence by a binary temporal partition whose intervals are split only when the observed trajectory makes an excursion larger than a threshold. Encode one summary token per retained leaf, optionally including duration and endpoint displacement, so smooth trajectory regions receive fewer tokens while rapidly changing regions retain resolution.
Replace a learned embedding for each integer with a compositional embedding of its variable-length Gray codeword. Add an auxiliary constraint that numerically adjacent values have nearby representations, while preserving the ordinary task loss so that the model can learn when numerical adjacency matters.
Replace an independently generated discrete latent sequence or redundancy stream with a symmetric two-state Markov source whose transition probability is tuned or learned. Train the downstream transformer to reconstruct the semantic target after random insertions, while using the paper's insertion-capacity expression to select the latent rate and redundancy budget. The representation should preserve information under timing drift, repeated tokens, and inserted distractors better than iid token…
Replace raw braid-generator sequences by sequences of positive simple Garside factors obtained from the left-greedy normal form. Because powers of \(\Delta\) lie in the Hilden subgroup, they can be removed while preserving the relevant double-coset presentation, reducing non-uniqueness and often shortening the sequence. Feed the resulting factor tokens to a Transformer or sequence classifier, and train it to be invariant to inserted removable \(\Delta\)-powers.
Add a multiscale Hamming-ball discrepancy penalty to a learned discrete codebook or tokenizer. The penalty forces the selected codewords to distribute their mass so that every center and radius sees approximately the global expected fraction of codewords, discouraging collapsed or highly clustered codebooks and potentially improving robustness to symbol substitutions.
Add a finite-resolution geometric code to a 3D neural encoder: quantized lattice occupancy, local barycenters, and tangent directions are converted into structural tokens alongside ordinary point or mesh features. Choose lattice spacing from estimated local reach so that small perturbations do not change the code, and train the continuous encoder to agree with this discrete structural representation.
Train a sequence encoder-decoder with an explicit list-consistency objective: after insertion or deletion corruption, require the correct prediction to remain among the top $L$ hypotheses compatible with the clean latent sequence. Instead of optimizing only one alignment, retain multiple low-cost monotone alignments or candidate latent decodings and penalize the model when the clean target falls outside this list.