# Null-normalized temporal prediction score

- ID: 2101
- Canonical URL: https://synthcore.org/idea/2101/null-normalized-temporal-prediction-score
- API JSON: https://synthcore.org/api/idea/2101.json
- API Markdown: https://synthcore.org/api/idea/2101.md
- Verification status: unverified
- Source: [arXiv:2608.10887](https://arxiv.org/abs/2608.10887)
- Category: theory
- Solves: accuracy, generalization, sample-efficiency
- ML areas: loss, training-dynamics, transformer, ssm
- Math tags: statistics, probability, numerical-analysis
- Ratings: usefulness 6/10; difficulty 3/10; novelty 8/10

## Idea description

Replace raw Pearson correlation when evaluating a temporal neural predictor with a score measuring how many null standard deviations its Fisher-transformed correlation exceeds. Estimate the null scale from a small set of time-misaligned predictions, then reuse it across context lengths or checkpoints. This prevents models from being rewarded for predicting statistically easy, low-information features and gives a more comparable validation signal across datasets and targets.

## Mathematical statement

For paired target values y_t and predictions \hat{y}_t in an evaluation window, let r=\operatorname{corr}(y,\hat{y}) and apply Fisher's transformation z=\operatorname{artanh}(r)=\frac{1}{2}\ln((1+r)/(1-r)). Under a null hypothesis of no temporal correspondence, generate n misaligned pairs by shifting predictions or targets by more than the empirical autocorrelation length. Their correlations r_i^{(\mathrm{null})} should have mean approximately zero, and their Fisher values z_i^{(\mathrm{null})}=\operatorname{artanh}(r_i^{(\mathrm{null})}) are modeled as \mathcal{N}(0,\sigma_z^{(\mathrm{null})2}). Estimate \hat{\sigma}_{z}^{(\mathrm{null})2}=n^{-1}\sum_i(z_i^{(\mathrm{null})})^2. The exact finite-sample variance proxy from the paper is V(N), where N=wf_s is the number of samples in a window of length w seconds sampled at f_s Hz: V(N)=\frac{\pi^2}{12}-\frac{1}{2}\sum_{k=1}^{(N-4)/2}k^{-2} for even N, and V(N)=\frac{\pi^2}{4}-2\sum_{k=1}^{(N-3)/2}(2k-1)^{-2} for odd N. To transfer an empirically estimated base-window variance to target window length, use \hat{\sigma}_{z,\mathrm{target}}^{(\mathrm{null})2}=\hat{\sigma}_{z,\mathrm{base}}^{(\mathrm{null})2}V(N_{\mathrm{target}})/V(N_{\mathrm{base}}). Define the adapted null-normalized score NNTS=(z_{\mathrm{obs}}-0)/\hat{\sigma}_{z,\mathrm{target}}^{(\mathrm{null})}; a one-sided significance estimate is p=1-\Phi(\mathrm{NNTS}), where \Phi is the standard normal CDF. For averaging m approximately independent evaluation windows, the null variance of the raw mean correlation is \hat{\sigma}_{r}^{(\mathrm{null})2}/m, with \hat{\sigma}_{r}^{(\mathrm{null})2}=n^{-1}\sum_i(r_i^{(\mathrm{null})})^2.

## Key formulas

- $$z=\frac{1}{2}\ln\!\left(\frac{1+r}{1-r}\right)=\operatorname{artanh}(r)$$
- $$\hat{\sigma}_{z}^{(\mathrm{null})2}=\frac{1}{n}\sum_{i=1}^{n}\left(z_i^{(\mathrm{null})}\right)^2,\qquad z_i^{(\mathrm{null})}=\operatorname{artanh}\!\left(r_i^{(\mathrm{null})}\right)$$
- $$V(N)=\begin{cases}\dfrac{\pi^{2}}{12}-\dfrac{1}{2}\displaystyle\sum_{k=1}^{\frac{N-4}{2}}k^{-2},&N\ \mathrm{even}\\[4pt]\dfrac{\pi^{2}}{4}-2\displaystyle\sum_{k=1}^{\frac{N-3}{2}}(2k-1)^{-2},&N\ \mathrm{odd},\end{cases}\qquad N=wf_s$$
- $$\mathrm{NNTS}=\frac{\operatorname{artanh}(r_{\mathrm{obs}})}{\hat{\sigma}_{z,\mathrm{base}}^{(\mathrm{null})}\sqrt{V(N_{\mathrm{target}})/V(N_{\mathrm{base}})}},\qquad p_{\mathrm{null}}=1-\Phi(\mathrm{NNTS})$$

## Implementation notes

Integrate this at the validation and model-selection layer of a temporal neural network, such as a Transformer, SSM, TCN, or world model that predicts a continuous future signal. Do not alter the forward pass initially. For each held-out sequence, compute the prediction-target Pearson correlation r_obs over a fixed window of w_base seconds, and cache the resulting Fisher-transformed score. Build the null using n=1000 cheap circular shifts: for each shift k, choose k larger than the maximum estimated autocorrelation length of either target or prediction, compute r_null[k] between the target and shifted prediction, and verify that abs(mean(r_null)) is small. Use this pseudocode: inputs=(y,pred,w_base,w_target,fs); r_obs=corr(y,pred); for k in valid_shifts: r_null[k]=corr(y,roll(pred,k)); z_null=atanh(clip(r_null,-1+eps,1-eps)); sigma2=mean(z_null**2); N_base=round(w_base*fs); N_target=round(w_target*fs); sigma_target=sqrt(sigma2*V(N_target)/V(N_base)); score=atanh(r_obs)/sigma_target. Aggregate scores over examples using the median or mean, and use score rather than raw correlation for early stopping and checkpoint selection. The paper supplies the Fisher transform, zero-mean null assumption, empirical variance, and V(N) rescaling; estimate autocorrelation lengths, valid shifts, and any dependence correction empirically. The first experiment should use a small GRU or 2-layer temporal Transformer on Speech Commands or LibriSpeech acoustic-feature prediction, comparing raw-correlation selection against NNTS selection with only 1000 shifts and several window lengths. Success means better held-out correlation on difficult features at equal parameter count, more stable ranking across window lengths, and fewer cases where a low-information target wins solely by having a high raw correlation.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.