# Successive Orthogonal Innovation Blocks

- ID: 2095
- Canonical URL: https://synthcore.org/idea/2095/successive-orthogonal-innovation-blocks
- API JSON: https://synthcore.org/api/idea/2095.json
- API Markdown: https://synthcore.org/api/idea/2095.md
- Verification status: unverified
- Source: [arXiv:2608.10757](https://arxiv.org/abs/2608.10757)
- Category: architecture
- Solves: memory, scalability, stability
- ML areas: mlp, transformer, pruning, fine-tuning
- Math tags: linear-algebra, geometry, optimization, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Add a neural feature, adapter, or expert block only through the component of its outputs that is orthogonal to the span of all previously installed blocks. Quotient coefficient directions that produce nearly identical outputs with an SVD or pseudoinverse, so the new block contributes intrinsic representational dimensions instead of duplicating old features. The expected benefit is a smaller effective architecture and better-conditioned block expansion.

## Mathematical statement

The paper defines coefficient spaces and assembled generators by $E^{(m)}=\bigoplus_{\ell=1}^{m}E_{\ell}$, $S^{(m)}c=\sum_{\ell=1}^{m}S_{\ell}c_{\ell}$, and $V_m=S^{(m)}(E^{(m)})$. Here $E_{\ell}$ is the parameter space of block $\ell$, $S_{\ell}$ maps its coefficients to functions or outputs, and $V_m$ is the nested output space. The paper explicitly notes that different coefficient vectors can represent the same element of $V_m$ and that the new contribution must be measured relative to $V_{m-1}$. For a finite minibatch, represent the old output space by a matrix $Y_{<m}\in\mathbb{R}^{n\times d}$ and candidate block outputs by $Y_m\in\mathbb{R}^{n\times d_m}$, where rows correspond to examples or tokens. Let $Q$ have orthonormal columns spanning the numerically nonzero left singular vectors of $Y_{<m}$; the successive orthogonal innovation is $R_m=(I-QQ^{\top})Y_m$. The block Gram matrix $G=\begin{bmatrix}G_{11}&G_{12}\\G_{21}&G_{22}\end{bmatrix}$ with $G_{11}=Y_{<m}^{\top}Y_{<m}$, $G_{12}=Y_{<m}^{\top}Y_m$, and $G_{22}=Y_m^{\top}Y_m$ has the Schur-complement innovation matrix $G_{m\mid <m}=G_{22}-G_{21}G_{11}^{\dagger}G_{12}=R_m^{\top}R_m$. Its rank or thresholded effective rank estimates the intrinsic new dimension. The pseudoinverse $G_{11}^{\dagger}$ implements the quotient over redundant old coefficients.

## Key formulas

- $$E^{(m)}=\bigoplus_{\ell=1}^{m}E_{\ell},\qquad S^{(m)}c=\sum_{\ell=1}^{m}S_{\ell}c_{\ell},\qquad V_m=S^{(m)}(E^{(m)})$$
- $$G=\begin{bmatrix}G_{11}&G_{12}\\G_{21}&G_{22}\end{bmatrix},\qquad G_{m\mid <m}=G_{22}-G_{21}G_{11}^{\dagger}G_{12}$$
- $$R_m=(I-QQ^{\top})Y_m,\qquad G_{m\mid <m}=R_m^{\top}R_m$$
- $$r_{\mathrm{eff}}(R_m;\tau)=\#\{i:\sigma_i(R_m)^2\geq\tau\,\sigma_1(R_m)^2\}$$

## Implementation notes

(1) Integration point: use this for a residual MLP block, LoRA adapter, or MoE expert expansion. For a transformer MVP, take one MLP layer and let each block produce token outputs $u_\ell=\mathrm{MLP}_\ell(x)\in\mathbb{R}^{n\times d}$ before the residual addition. Maintain a low-rank basis $Q$ for the concatenated outputs of all accepted earlier blocks. Do not modify attention logits or the main residual stream initially; replace a candidate block output $u_m$ by its innovation $r_m=u_m-Q(Q^\top u_m)$.

(2) Pseudocode: collect a calibration minibatch $X$; compute $Y_{<m}=\mathrm{concat}(u_1,\ldots,u_{m-1})$ and $Y_m=u_m$; run randomized or thin QR/SVD on $Y_{<m}$; retain singular vectors with $\sigma_i^2\geq\epsilon\sigma_1^2$ to form $Q$; compute $R_m=(I-QQ^\top)Y_m$; compute its singular values; accept block $m$ only if $r_{\mathrm{eff}}(R_m;\tau)\geq k_{\min}$; during training use $u_m^{\mathrm{innov}}=u_m-Q\operatorname{stopgrad}(Q^\top u_m)$ and add $\alpha_m u_m^{\mathrm{innov}}$ to the layer output. Periodically refresh $Q$ using an exponential moving average of covariance or a new calibration batch. If the block is rejected, prune it or reduce its hidden width.

(3) Computed from the mathematics: the orthogonal projection, pseudoinverse/QR quotient, Schur-complement Gram matrix, singular-value rank, and innovation energy $\|R_m\|_F^2$. Estimated empirically: the calibration-batch output span, the threshold $\tau$, and whether validation loss improves; use $\epsilon=10^{-5}$ for numerical rank and sweep $\tau\in\{10^{-2},10^{-3},10^{-4}\}$. To avoid a full $n\times n$ projector, store $Q\in\mathbb{R}^{n\times r}$ and compute $u-Q(Q^\top u)$ in two matrix multiplications.

(4) First cheap experiment: train a 6-layer small Transformer on WikiText-2 or a synthetic next-token task with four expandable MLP blocks per layer. Compare ordinary parallel blocks, width-matched blocks with orthogonal initialization, and successive innovation blocks. Keep parameter count and FLOPs fixed where possible, and separately test adaptive pruning of blocks whose innovation rank is below threshold. Measure validation perplexity at equal training FLOPs, gradient-condition proxies, cosine similarity between block outputs, retained parameter count, and activation-memory cost. Success means lower block-output cosine redundancy, no increase in training instability, and either at least 10% fewer retained MLP parameters at matched perplexity or faster perplexity reduction at equal FLOPs.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.