Følner Block-Replacement Curriculum

1. Integration point: use this as an offline augmentation and curriculum stage for causal Transformers, RNNs, or state-space models. Begin with sequences from a training corpus and choose a block length L. Extract length-L blocks and characterize them by empirical frequencies of short patterns w, such as token n-grams, discretized time-series windows, or vector-quantized symbol patterns. Build a library whose pattern frequencies are within epsilon of the target corpus statistics. 2. Pseudocode: sample a long context F_n g; partition its interior into F_L blocks; for every block, retrieve or sample a replacement block from the library conditioned on its coarse frequency class; copy replacement tokens only on Int_(D_w)(F_L h); preserve the original tokens on the boundary; concatenate the tiles; train on both original and augmented contexts. Increase L during the curriculum. 3. Estimate q_(L,w), library mismatch eta_lib, and boundary fractions directly from data. For ordinary sequences use G = Z, F_L = {0,...,L-1}; the boundary has width approximately equal to the pattern diameter. The paper supplies the tiling and error-propagation mechanism, while retrieval and conditioning are empirical choices. 4. First experiment: train a small two-layer causal Transformer on WikiText-2 and on binary Markov sequences, comparing contiguous minibatches, random token permutation, and this augmentation. Measure n-gram frequency error and validation loss versus L. The quantitative prediction is an approximately 1/L decrease in local-statistic distortion for one-dimensional sequences, with discrepancy bounded by eta_lib plus approximately 2 times the pattern diameter divided by L. Test whether the measured error follows this slope within 20 percent and whether training becomes unstable when the bound is violated.