# Correlation-Length Scaling Benchmark

- ID: 1295
- Canonical URL: https://synthcore.org/idea/1295/correlation-length-scaling-benchmark
- API JSON: https://synthcore.org/api/idea/1295.json
- API Markdown: https://synthcore.org/api/idea/1295.md
- Verification status: unverified
- Source: [arXiv:2607.22863](https://arxiv.org/abs/2607.22863)
- Category: architecture
- Solves: accuracy, scalability
- ML areas: transformer, ssm, architecture, training-dynamics
- Math tags: scaling, statistics, dynamical-systems
- Ratings: usefulness 5/10; difficulty 3/10; novelty 6/10

## Idea description

Treat the maximum dependency distance faithfully modeled by a finite neural architecture as an emergent correlation length, and estimate how it grows with depth, state size, or attention span. Fit the exponent \(\kappa\) and use it as an architecture-selection signal: a model with larger \(\kappa\) should acquire long-range competence more efficiently at equal parameter or FLOP budget.

## Mathematical statement

The paper defines a refinement resource \(D\) (circuit depth in the original setting), an emergent correlation length \(\xi_D\) (the longest distance over which the finite-resource ansatz faithfully captures critical correlations), and the scaling law \(\xi_D\propto D^\kappa\), where \(\kappa\) is the resource-efficiency exponent. In the neural adaptation, let \(R(r;D)\) be a distance-resolved error or retained-correlation score at dependency distance \(r\) for a model using resource \(D\). Choose a fixed failure threshold \(\tau\), and define \(\xi_D=\max\{r:R(r;D)\ge\tau\}\). Fit \(\log \xi_D=a+\kappa\log D\) by least squares over several resource values; \(a\) is an intercept and \(\kappa\) measures how efficiently the architecture converts resources into long-range structure. This is an empirical finite-resource scaling law, not a convergence theorem, so the threshold and task distribution must be held fixed across architectures.

## Key formulas

- $$\xi_D\propto D^{\kappa}$$
- $$\xi_D=\max\{r:R(r;D)\ge \tau\},\qquad \log \xi_D=a+\kappa\log D$$
- $$R(r;D)=1-\frac{\mathbb{E}_{(x,y)}\left[\ell_r(f_D(x),y)\right]}{\mathbb{E}_{(x,y)}\left[\ell_r(f_{\mathrm{chance}}(x),y)\right]}$$

## Implementation notes

Integrate this as an architecture-profiling and selection tool for a Transformer versus an SSM, not initially as a differentiable training loss. Construct synthetic sequences containing an information-bearing token at position \(t-r\) and a target at position \(t\), with distractors between them; vary \(r\) over powers of two. Train each model family at several resource values \(D\): for a Transformer use number of layers or attention span, and for an SSM use number of layers or hidden-state size. For each trained model, evaluate the distance-resolved loss \(\ell_r\) and compute \(R(r;D)\) using the normalized excess-error formula above, where \(f_D\) is the model at resource \(D\), \(f_{\mathrm{chance}}\) is a majority or random predictor, and \(\tau\), for example 0.5, is fixed before comparison. Estimate \(\xi_D\) by scanning distances and linearly interpolating the first crossing of \(R=\tau\). Fit \(\kappa\) using robust linear regression on \((\log D,\log \xi_D)\), with bootstrap confidence intervals over random seeds. The paper supplies the power-law scaling principle; the distance metric, threshold, and regression are empirical engineering choices. The first cheap experiment is a 2-layer/4-layer/8-layer comparison on copy-with-distractors or associative recall using a small Transformer and S4-like SSM. Success means higher \(\xi_D\) at equal FLOPs and a statistically larger fitted \(\kappa\), followed by better performance on held-out longer dependencies than aggregate validation loss would predict.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.