ML: Tokenizer

Machine-learning ideas tagged Tokenizer in the ML taxonomy of the Math2NN corpus.

2 ideas found

ML: Tokenizer Math: Stochastic processes Clear all

Unverified 2026

Excursion-Adaptive Temporal Tokenization

Replace a uniformly sampled trajectory sequence by a binary temporal partition whose intervals are split only when the observed trajectory makes an excursion larger than a threshold. Encode one summary token per retained leaf, optionally including duration and endpoint displacement, so smooth trajectory regions receive fewer tokens while rapidly changing regions retain resolution.

Useful6/10

Difficulty4/10

Novelty6/10

Architecture Memory Speedup Accuracy

Paper: Sharp Wasserstein Convergence Rates for Empirical Path Laws of Itô Processes arXiv:2608.07879

△ Mechanism confirmed, baseline not beaten 2026

Markov-coded synchronization-robust latent streams

Replace an independently generated discrete latent sequence or redundancy stream with a symmetric two-state Markov source whose transition probability is tuned or learned. Train the downstream transformer to reconstruct the semantic target after random insertions, while using the paper's insertion-capacity expression to select the latent rate and redundancy budget. The representation should preserve information under timing drift, repeated tokens, and inserted distractors better than iid token…

Useful6/10

Difficulty5/10

Novelty7/10

Architecture Accuracy Sample efficiency Stability

Paper: The Insertion List-Decoding Capacity and an Improved Bound on the Deletion List-Decoding Capacity arXiv:2607.03989