Identity-Paired Progressive Depth

Integrate this at the depth-growth mechanism of a residual MLP, encoder-only Transformer, or sequence model. Prefer additive-coupling blocks because their inverse is explicit and does not require a matrix inverse. Split each hidden vector into two equal channel groups $(x_a,x_b)$ and implement one coupling block $C_{f,g}$ using small MLPs or depthwise linear layers for $f$ and $g$. During progressive training, first train a shallow model. To expand it, copy an existing coupling block with parameters $\theta_0$, append $C_{\theta_0}$, and immediately append its inverse $C^{-1}_{\theta_0}$; the forward pass is unchanged up to floating-point error. At the next optimizer step, replace the copied parameters by two independent parameter sets $\theta_+$ and $\theta_-$, while retaining the inverse-form computation for the second block only at initialization. In a simpler residual variant, use an invertible linear or orthogonal block $B_W(x)=Wx$ and append $W^{-1}W$, estimating $W^{-1}$ exactly from a stored orthogonal parameterization. For a Transformer, apply the pair around an FFN or around one attention-mixing layer; test a low-cost version in which only the first block contains attention and later paired blocks contain channelwise FFNs. After every expansion, run a short fixed budget of minibatches, evaluate validation loss, and accept the expansion only if $\mathcal L_{t+1}\leq\mathcal L_t+\epsilon$; otherwise remove the pair or reduce its learning rate. What is exact from the mathematics is the identity cancellation and reversible inverse. What must be estimated empirically is whether the extra untied parameters improve optimization after cancellation. The first experiment should use a 6-layer versus progressively grown 12-layer Transformer on WikiText-2 or CIFAR-10, with standard same-width training as baseline. Measure loss immediately before and after insertion, loss after 1,000 updates, gradient norm spikes, wall-clock cost, and final validation accuracy. Success means no insertion loss spike, lower loss at equal optimizer steps, and at least comparable final accuracy despite starting from the shallow model.