Intrinsic Active-Subspace Adapter

Integrate this at a normalized embedding or latent-state tensor, not at raw Euclidean weights. First choose a unit center c, for example the normalized running mean of a minibatch of embeddings x_i; restrict the covariance batch to points with geodesic distance arccos(x_i^T c) below a radius such as 0.5 radians. Compute the network scalar of interest f(x_i), usually the per-example loss or a selected logit, and obtain its gradient g_i with respect to x_i. Project each gradient onto the sphere tangent space with g_i <- g_i - (g_i^T x_i)x_i. Transport each projected gradient to T_c S^{d-1} with the displayed P_{x_i->c} formula, set w_i=1/N or use normalized local kernel weights w_i proportional to exp(-theta_i^2/(2*sigma^2)), and accumulate C. Every K steps, compute the top r eigenvectors U_r by power iteration or randomized SVD; detach U_r for the next K steps. For parameter-efficient fine-tuning, replace a full adapter delta(x) with U_r A phi(x), where A maps the r active coordinates to the adapter output, or feed z=U_r^T log_c(x) into a small ridge head h. Estimate gradients and the covariance empirically; the geometry formulas and eigen-decomposition are exact, while c, r, sigma, and the update period are hyperparameters. Start with a frozen 2-layer MLP on MNIST or CIFAR-10 using normalized penultimate embeddings, compare a full linear adapter against r=8 and r=16 intrinsic adapters, and track validation accuracy, trainable parameters, wall-clock time, and covariance spectral gap. Success is comparable accuracy with substantially fewer adapter parameters, or faster loss descent at equal trainable FLOPs; failure is a small spectral gap and no compression benefit.