Incoherent Frame Averaging for Tensor Layers

(1) Integration point: apply this to a tensorized MLP or tensor-contraction layer F whose input x is flattened to dimension N. During training, create R=2--8 fixed random orthogonal or signed-permutation frames R_r, transform the layer input as x_r=R_r x, evaluate F(x_r; heta), inverse-transform the output, and average. Also register a gradient hook on x to obtain g_r=\partial\mathcal L_{task}/\partial x_r. The regularizer is applied to activations and optionally gradients; it is not needed at inference if the frame ensemble is folded into a fixed deterministic average.

(2) Pseudocode:

for x, y in batch:
    task = 0; reg = 0
    for r in range(R):
        xr = R[r] @ x
        pred_r = R[r].T @ F(xr, theta)
        task += cross_entropy(pred_r, y) / R
        gr = grad(task, xr, create_graph=use_second_order)
        topx = sum(top_b(xr**2)) / (sum(xr**2) + eps)
        topg = sum(top_b(gr**2)) / (sum(gr**2) + eps)
        reg += topx + lambda_g * topg
    loss = task + lambda_inc * reg / R
    theta = optimizer_step(theta, grad(loss, theta))

Here top_b selects the b largest entries, b is tested near N^(1/p), and epsilon prevents division by zero. Stop gradients through gr if second-order differentiation is too expensive.

(3) Taken from the paper: the tensor normalization N^(-(p-1)/2), the bulk-error scale q_N, and the coherent-block prediction b approximately N^(1/p). Estimated during training: top-b energy, gradient coherence, activation norms, and frame-to-frame output variance. Use q_N to set a schedule: start with lambda_inc=0, then increase it only if q_N is not already small and coherence exceeds a target.

(4) First experiment: use a tensorized two-layer MLP on MNIST or CIFAR-10 with matched parameter count. Compare an ordinary tensor layer, weight decay, random frame averaging without the penalty, and the full method. Measure validation accuracy, gradient-norm 99th percentiles, seed-to-seed loss variance, frame output variance, and top-b activation energy. Add a controlled b-coordinate spike to inputs or weights. Success is clean accuracy no worse than baseline, substantially lower coherence and gradient tails, and smaller degradation under the spike; a concrete target is at least 2x lower top-b energy or a measurable reduction in worst-seed validation loss.