# Spherical-design directional heads

- ID: 2138
- Canonical URL: https://synthcore.org/idea/2138/spherical-design-directional-heads
- API JSON: https://synthcore.org/api/idea/2138.json
- API Markdown: https://synthcore.org/api/idea/2138.md
- Verification status: unverified
- Source: [arXiv:2608.11570](https://arxiv.org/abs/2608.11570)
- Category: architecture
- Solves: accuracy, sample-efficiency, stability
- ML areas: attention, embedding, transformer, initialization
- Math tags: geometry, approximation-theory, convex-analysis, harmonic-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Use a fixed spherical t-design as the direction codebook for a directional attention or feature-aggregation module instead of independently sampled random directions. Equal weights provide exact zero mean and isotropic second moments, while exactness for spherical polynomials up to degree t reduces directional aliasing and seed-dependent anisotropy.

## Mathematical statement

A spherical t-design is a set X_N={x_1,...,x_N} subset of S^2 whose empirical measure μ_X=(1/N) sum_j δ_{x_j} integrates every spherical polynomial p of degree at most t exactly as the uniform probability measure σ=ω/(4π): (1/N)sum_j p(x_j)=∫_{S^2}p(u)dσ(u). Degree-one and degree-two exactness imply (1/N)sum_j x_j=0 and (1/N)sum_j x_jx_j^T=(1/3)I_3, where I_3 is the 3-dimensional identity. The associated Minkowski polytope P_X has facet normals x_j and equal facet areas 4π/N, equivalently S_{P_X}=4πμ_X. Spherical Jackson approximation gives W_1(μ_X,σ)=O(t^{-1}), where W_1 is 1-Wasserstein distance, while the paper gives d_H(P_X,B)=O(t^{-1/2}) after Steiner normalization, with d_H the Hausdorff distance and B the unit ball. The transferable asset is exact low-order directional moments and deterministic quadrature accuracy for smooth angular functions.

## Key formulas

- $$\mu_X=\frac{1}{N}\sum_{j=1}^{N}\delta_{x_j},\qquad \sigma=\frac{\omega}{4\pi},\qquad \frac{1}{N}\sum_{j=1}^{N}p(x_j)=\int_{\mathbb S^2}p(u)\,d\sigma(u)\quad\text{for every spherical polynomial }p\text{ of degree }\le t.$$
- $$\frac{1}{N}\sum_{j=1}^{N}x_j=0,\qquad \frac{1}{N}\sum_{j=1}^{N}x_jx_j^{\mathsf T}=\frac{1}{3}I_3.$$
- $$W_1(\mu_X,\sigma)=O(t^{-1}),\qquad d_H(P_X,B)=O(t^{-1/2})\quad\text{after Steiner normalization}.$$
- $$y=\frac{1}{N}\sum_{j=1}^{N}\phi\!\left(q^{\mathsf T}x_j,\,k_j,\,v_j\right),\qquad q=A h\in\mathbb R^3.$$

## Implementation notes

Integrate this at the directional-feature or attention-head aggregation point. Generate or import a spherical t-design X={x_j in R^3} with N directions, normalize every x_j to unit length, and freeze these directions during training. For each token embedding h, use a learned projection q=A h in R^3. For direction j, compute the angular score z_j=q^T x_j, form a value v_j=V_j h or a shared equivariant branch v_j=V(h,x_j), and aggregate either by deterministic quadrature y=(1/N)sum_j phi(z_j,v_j) or by attention y=sum_j softmax_j(z)v_j. Pseudocode: input h; q=A(h); for j=1,...,N compute z_j=dot(q,x_j) and v_j=V(h,x_j); compute a_j=softmax(z)_j if using normalized attention; output y=sum_j a_j v_j, or output y=(1/N)sum_j phi(z_j,v_j) for the strict quadrature variant. If directions are made learnable, add L_iso=||N^{-1}sum_j x_jx_j^T-I_3/3||_F^2 and L_mean=||N^{-1}sum_j x_j||_2^2. The paper supplies the exact moment identities and the expected O(t^{-1}) error for smooth directional observables. Estimate actual approximation error by evaluating random spherical polynomials of degrees 1 through t and comparing design averages with dense sphere quadrature. First experiment: compare random directions, learned directions, and a fixed t=4 design in a small ViT on CIFAR-10 and on a synthetic 3D rotation-classification task, keeping FLOPs and parameter counts fixed. Success is lower seed-to-seed variance, better rotated-input accuracy, and equal accuracy with fewer directions or heads.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.