# Excursion-Adaptive Temporal Tokenization

- ID: 1966
- Canonical URL: https://synthcore.org/idea/1966/excursion-adaptive-temporal-tokenization
- API JSON: https://synthcore.org/api/idea/1966.json
- API Markdown: https://synthcore.org/api/idea/1966.md
- Verification status: unverified
- Source: [arXiv:2608.07879](https://arxiv.org/abs/2608.07879)
- Category: architecture
- Solves: memory, speedup, accuracy
- ML areas: transformer, attention, tokenizer, memory
- Math tags: stochastic-processes, probability, approximation-theory, metric-geometry
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace a uniformly sampled trajectory sequence by a binary temporal partition whose intervals are split only when the observed trajectory makes an excursion larger than a threshold. Encode one summary token per retained leaf, optionally including duration and endpoint displacement, so smooth trajectory regions receive fewer tokens while rapidly changing regions retain resolution.

## Mathematical statement

The paper constructs a forest by recursively splitting a dyadic interval I=[u,v] with left endpoint u whenever sup_{t in I} ||Y_t-Y_u||>delta, and retaining I as a leaf otherwise. Here Y is a continuous path, delta>0 is the excursion threshold, and ||.|| is the Euclidean norm. For an Ito decomposition Y_t=Y_u+(drift increment)+(M_t-M_u), with drift bounded by G, martingale part M, and dyadic interval length h_l at level l, the paper uses ||Y_t-Y_s|| <= G|t-s|+||M_t-M_s|| and chooses a base level n_0 so that G h_l <= delta/4, yielding {sup_{t in I}||Y_t-Y_u||>delta} subset {sup_{t in I}||M_t-M_u||>3delta/4}. The quadratic-variation bound <M^i>_t-<M^i>_u <= G^2(t-u) <= G^2 h_l explains why large excursions become rare on fine intervals. The paper's broader path quantization scale is (log N)^(-1/2) under the supremum metric; the neural adaptation uses the forest as a variable-rate path approximation rather than estimating Wasserstein distance.

## Key formulas

- $$\mathsf{split}(I)\Longleftrightarrow\sup_{t\in I}\lVert Y_t-Y_u\rVert>\delta,\qquad I=[u,v].$$
- $$\lVert Y_t-Y_s\rVert\leq G|t-s|+\lVert M_t-M_s\rVert,\qquad\left\{\sup_{t\in I}\lVert Y_t-Y_u\rVert>\delta\right\}\subseteq\left\{\sup_{t\in I}\lVert M_t-M_u\rVert>\frac{3\delta}{4}\right\}.$$
- $$\langle M^i\rangle_t-\langle M^i\rangle_u\leq G^2(t-u)\leq G^2h_\ell,\qquad Gh_\ell\leq\frac{\delta}{4}.$$
- $$\widehat{Y}(t)=Y_u+\frac{t-u}{v-u}(Y_v-Y_u),\qquad\lVert Y-\widehat{Y}\rVert_\infty\leq\delta\ \text{on every retained leaf}.$$

## Implementation notes

Integrate this at the temporal tokenizer before a trajectory Transformer, temporal U-Net, or diffusion model. Inputs are a sampled trajectory y[0:T], timestamps, an excursion threshold delta, and a maximum tree depth L. Start with queue=[(0,T,0)]. For each interval (a,b,l), compute e=max_{a<=k<=b} ||y[k]-y[a]||. If e>delta, l<L, and b-a>1, split at m=floor((a+b)/2) and push (a,m,l+1) and (m,b,l+1); otherwise retain the interval as a leaf. For every leaf create a token containing y[a], y[b], normalized duration (b-a)/T, displacement y[b]-y[a], excursion e, and a learned depth embedding. Add relative-time attention biases using interval centers and durations. The paper's split rule is used directly; the drift and martingale inequalities motivate why fine intervals should rarely split but are not required by the implementation. Choose delta by a validation sweep targeting a fixed token budget, or use delta equal to a percentile of training-set one-step excursions. Enforce both a minimum interval length and a hard leaf cap. Train with random threshold jitter to reduce boundary sensitivity. First test on Lorenz-63, human-motion trajectories, and short Navier-Stokes rollouts using a 6-layer Transformer, comparing against uniform subsampling at matched token count and FLOPs. Measure token count, wall-clock attention cost, peak memory, one-step error, and long-horizon rollout error. Success is at least 1.5x fewer tokens at equal error, or lower rollout error at equal attention FLOPs.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.