✗ Failed on benchmark 2026

Anchored Whitening Layer

Implementation & benchmark of arXiv:2608.15319 — CORAL: Constrained Oblique Rotation with Anchored Loadings for Fidelity-Constrained Decorrelation

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Stability Accuracy Generalization Cnn Transformer Regularization Initialization Linear algebra Geometry Optimization Statistics

Source paper: CORAL: Constrained Oblique Rotation with Anchored Loadings for Fidelity-Constrained Decorrelation arXiv:2608.15319 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a conventional whitening transform with a constrained whitening layer that minimizes cross-channel covariance while requiring every output channel to remain correlated with its designated input channel by at least a threshold $\rho_{\min}$. The layer exploits the orthogonal freedom in whitening to find a rotation that preserves channel identity instead of arbitrarily mixing features. It can be inserted before an MLP, convolution, or attention projection and compared directly against BatchNorm, LayerNorm, and unconstrained ZCA whitening.

Formulas

$$\widetilde{X}=XT,\qquad \operatorname{Cov}(\widetilde{X})=T^{\top}RT,$$

$$T=R^{-1/2}Q,\qquad Q^{\top}Q=I\quad\Longrightarrow\quad T^{\top}RT=I,$$

$$\operatorname{Cor}(\widetilde{X}_j,X_j)=e_j^{\top}Rt_j\geq\rho_{\min},\qquad j=1,\ldots,p,$$

$$\mathcal{L}_{\mathrm{CORAL}}(T)=\left\|T^{\top}RT-I\right\|_F^2+\lambda\sum_{j=1}^{p}\left[\max\left(0,\rho_{\min}-e_j^{\top}Rt_j\right)\right]^2,\qquad T=R^{-1/2}Q.$$

Mathematical statement

For a standardized activation matrix $X\in\mathbb{R}^{n\times p}$ with symmetric positive-definite covariance or correlation matrix $R\in\mathbb{R}^{p\times p}$, the transformed data are $\widetilde X=XT$. The transformed covariance is $T^{\top}RT$, and unit output variance requires $t_j^{\top}Rt_j=1$ for every column $t_j$ of $T$. Exact decorrelation is obtained by $T=R^{-1/2}Q$, where $Q^{\top}Q=I$ is an orthogonal matrix, because $T^{\top}RT=Q^{\top}Q=I$. The source fidelity of output channel $j$ with input channel $j$ is $e_j^{\top}Rt_j$, so impose $e_j^{\top}Rt_j\geq\rho_{\min}$ for all $j$, with $e_j$ the $j$-th standard basis vector and $\rho_{\min}\in[0,1]$. Optimize only over $Q$, using an orthogonal retraction or Cayley update. In finite minibatches, use the practical penalty objective $\|T^{\top}RT-I\|_F^2+\lambda\sum_j[\max(0,\rho_{\min}-e_j^{\top}Rt_j)]^2$, optionally with a barrier for stricter constraints. The key structural property is that the whitening factor removes covariance while $Q$ controls the fidelity-versus-rotation tradeoff.

Implementation notes

(1) Integration point: insert the layer immediately before a channel-mixing operation, such as a Transformer MLP, a QKV projection, or a convolutional block. Treat the feature dimension as $p$; flatten batch and token or spatial dimensions into $n$ rows. Maintain an exponential moving average covariance $R$ during training, and use the frozen EMA covariance at evaluation. Center and standardize activations before estimating $R$. Compute $R^{-1/2}$ by eigendecomposition $R=U\operatorname{diag}(\lambda_i)U^{\top}$, replacing each $\lambda_i$ by $\lambda_i+\epsilon$.

(2) Pseudocode: Z = center_and_scale(X); R = ema_covariance(Z); A = eig_inverse_sqrt(R, eps); Q = orthogonal_parameter(); T = A @ Q; Y = Z @ T; fidelity = diag(R @ T); decor = T.T @ R @ T - I; loss_aux = ||decor||_F^2 + lambda * sum(relu(rho_min - fidelity)^2); Add loss_aux to the task loss. Parameterize $Q$ as a free matrix followed by QR, or update it on the orthogonal group with $Q\leftarrow\operatorname{qf}(Q+\eta G)$, where qf is the Q factor of QR. A more faithful manifold update uses $Q\leftarrow\operatorname{Cayley}(\eta S)Q$ with skew-symmetric $S=GQ^{\top}-QG^{\top}$.

(3) Computed from the mathematics: the inverse square root, whitening factor, fidelity vector $\operatorname{diag}(RT)$, and orthogonality constraint. Estimated empirically: minibatch covariance, EMA covariance, penalty weight $\lambda$, stabilization $\epsilon$, and the largest feasible $\rho_{\min}$. Start with $Q=I$, which is an identity-preserving initialization, then learn $Q$ jointly or optimize it for several warm-up steps while the rest of the network is frozen.

(4) First experiment: use a 6- or 12-layer CIFAR-10 ResNet with a whitening layer before each channel-mixing block. Compare BatchNorm, ZCA whitening with $Q=I$, unconstrained learned orthogonal $Q$, and anchored whitening at $\rho_{\min}\in\{0.8,0.9,0.95\}$. Measure training loss at equal FLOPs, validation accuracy, off-diagonal covariance, gradient noise, and minimum diagonal fidelity. Success means lower covariance and gradient noise than normalization baselines without accuracy loss, while maintaining the declared fidelity threshold; a useful signal is more stable or faster convergence at matched compute.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an anchored whitening MVP using T=R^{-1/2}Q with identity-preserving Q=I, plus deterministic mathematical sweeps and a toy classification comparison. Whitening residuals stayed between 0 and 1.2e-14, measured fidelity matched the analytic curve within 1.5e-15, and threshold feasibility agreed with prediction in all 10 tested cases. The toy classifier reduced off-diagonal covariance substantially without changing accuracy, so no accuracy win was observed.

Agent confidence: 9/10

Baseline: Per-channel standardization: accuracy=0.889714; off-diagonal covariance Frobenius norm=3.31448
Idea: Q=I ZCA/anchored whitening: accuracy=0.889714; off-diagonal covariance Frobenius norm=1.899e-15; maximum covariance error=1.055e-15; minimum sample fidelity=0.8523
Math predictions: Exact decorrelation error was 0 to 1.18e-14; fidelity-curve absolute error was 0 to 1.44e-15 over correlations {0, 0.1, 0.3, 0.6, 0.9}; threshold transition agreed in 10/10 cases

Limitations: The MVP tests the exact whitening family and identity anchor, not a learned orthogonal Q optimized with a penalty or Cayley update. The empirical comparison is a small synthetic logistic-regression task, not CIFAR/ResNet, and does not test EMA covariance, training dynamics, gradient noise, FLOPs, or BatchNorm/LayerNorm neural-network baselines.

How to run: python3 experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.15319) Sep 2, 2026

No significant difference vs. the tuned baseline (+2.0%, p=0.5077, wins 2 / 8 paired seeds).

Benchmark: Vision (image classification)
Model: cnn_small
Paired seeds: 8
Baseline mean: 0.8537
Idea mean: 0.8706
Effect (Δ): 0.0169 (+2.0%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.5077 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±7.2%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used the standard 400/200 CIFAR-10 subset and 4 epochs rather than the README's example 30 epochs, due to runtime constraints. The implementation used Q=I and did not optimize an orthogonal Q or enforce the rho_min constraint with a penalty; the signature probe was collected from one trained idea model and no baseline covariance signature was recorded. The observed signature failure may therefore reflect both the simplified implementation and short training, but it does not provide evidence for a win.

How to run: python3 bench_experiment.py > bench_results.json

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 9/10

Built a matched cnn_small vision comparison on the CIFAR-10 subset: the baseline was standard channel-mixing CNN behavior, while the idea inserted an EMA ZCA whitening layer after the first convolution. Across 8 paired seeds, the idea had higher test error (0.870625 vs 0.853750), with paired delta +0.016875 and permutation p=0.50765; therefore there is no significant win. The behavior-based signature also failed to confirm the claimed mechanism: observed off-diagonal covariance remained 1.9904, minimum fidelity was -0.0010 versus rho_min=0.8, and unit-variance error was 0.99997.

Baseline: vision/cnn_small; baseline sweep lr={0.001,0.003,0.01}; best lr=0.003; full 8-seed test error mean=0.853750, std=0.032476; per_seed=[0.890000,0.840000,0.870000,0.785000,0.870000,0.850000,0.890000,0.835000]
Idea: EMA Q=I anchored whitening after conv1; shared lr grid; selected lr=0.003; full 8-seed test error mean=0.870625, std=0.046465; per_seed=[0.760000,0.855000,0.885000,0.910000,0.855000,0.910000,0.900000,0.890000]
Mechanism signature: map[confirmed:false idea_max_unit_variance_error_observed:0.9999727606773376 idea_min_fidelity_observed:-0.0010492571163922548 idea_offdiag_cov_observed:1.9903746843338013 prediction:A trained anchored whitening layer should reduce channel covariance while preserving designated-channel fidelity. rho_min:0.8]
Paired comparison: delta_mean=+0.016875 (idea-baseline; lower is better), idea_wins=2/8, permutation_p=0.50765, verdict=no significant win

Benchmark result