Null-normalized temporal prediction score
Implementation & benchmark of arXiv:2608.10887 — Modeling and Interpreting Correlations, Null Distributions and Significance Levels in Neural Tracking of Natural Stimuli
Source paper: Modeling and Interpreting Correlations, Null Distributions and Significance Levels in Neural Tracking of Natural Stimuli arXiv:2608.10887 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace raw Pearson correlation when evaluating a temporal neural predictor with a score measuring how many null standard deviations its Fisher-transformed correlation exceeds. Estimate the null scale from a small set of time-misaligned predictions, then reuse it across context lengths or checkpoints. This prevents models from being rewarded for predicting statistically easy, low-information features and gives a more comparable validation signal across datasets and targets.
Formulas
Mathematical statement
For paired target values y_t and predictions \hat{y}_t in an evaluation window, let r=\operatorname{corr}(y,\hat{y}) and apply Fisher's transformation z=\operatorname{artanh}(r)=\frac{1}{2}\ln((1+r)/(1-r)). Under a null hypothesis of no temporal correspondence, generate n misaligned pairs by shifting predictions or targets by more than the empirical autocorrelation length. Their correlations r_i^{(\mathrm{null})} should have mean approximately zero, and their Fisher values z_i^{(\mathrm{null})}=\operatorname{artanh}(r_i^{(\mathrm{null})}) are modeled as \mathcal{N}(0,\sigma_z^{(\mathrm{null})2}). Estimate \hat{\sigma}_{z}^{(\mathrm{null})2}=n^{-1}\sum_i(z_i^{(\mathrm{null})})^2. The exact finite-sample variance proxy from the paper is V(N), where N=wf_s is the number of samples in a window of length w seconds sampled at f_s Hz: V(N)=\frac{\pi^2}{12}-\frac{1}{2}\sum_{k=1}^{(N-4)/2}k^{-2} for even N, and V(N)=\frac{\pi^2}{4}-2\sum_{k=1}^{(N-3)/2}(2k-1)^{-2} for odd N. To transfer an empirically estimated base-window variance to target window length, use \hat{\sigma}_{z,\mathrm{target}}^{(\mathrm{null})2}=\hat{\sigma}_{z,\mathrm{base}}^{(\mathrm{null})2}V(N_{\mathrm{target}})/V(N_{\mathrm{base}}). Define the adapted null-normalized score NNTS=(z_{\mathrm{obs}}-0)/\hat{\sigma}_{z,\mathrm{target}}^{(\mathrm{null})}; a one-sided significance estimate is p=1-\Phi(\mathrm{NNTS}), where \Phi is the standard normal CDF. For averaging m approximately independent evaluation windows, the null variance of the raw mean correlation is \hat{\sigma}_{r}^{(\mathrm{null})2}/m, with \hat{\sigma}_{r}^{(\mathrm{null})2}=n^{-1}\sum_i(r_i^{(\mathrm{null})})^2.
Implementation notes
Integrate this at the validation and model-selection layer of a temporal neural network, such as a Transformer, SSM, TCN, or world model that predicts a continuous future signal. Do not alter the forward pass initially. For each held-out sequence, compute the prediction-target Pearson correlation r_obs over a fixed window of w_base seconds, and cache the resulting Fisher-transformed score. Build the null using n=1000 cheap circular shifts: for each shift k, choose k larger than the maximum estimated autocorrelation length of either target or prediction, compute r_null[k] between the target and shifted prediction, and verify that abs(mean(r_null)) is small. Use this pseudocode: inputs=(y,pred,w_base,w_target,fs); r_obs=corr(y,pred); for k in valid_shifts: r_null[k]=corr(y,roll(pred,k)); z_null=atanh(clip(r_null,-1+eps,1-eps)); sigma2=mean(z_null**2); N_base=round(w_base*fs); N_target=round(w_target*fs); sigma_target=sqrt(sigma2*V(N_target)/V(N_base)); score=atanh(r_obs)/sigma_target. Aggregate scores over examples using the median or mean, and use score rather than raw correlation for early stopping and checkpoint selection. The paper supplies the Fisher transform, zero-mean null assumption, empirical variance, and V(N) rescaling; estimate autocorrelation lengths, valid shifts, and any dependence correction empirically. The first experiment should use a small GRU or 2-layer temporal Transformer on Speech Commands or LibriSpeech acoustic-feature prediction, comparing raw-correlation selection against NNTS selection with only 1000 shifts and several window lengths. Success means better held-out correlation on difficult features at equal parameter count, more stable ranking across window lengths, and fewer cases where a low-information target wins solely by having a high raw correlation.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.