Identity-Paired Progressive Depth Training: When Trainability Persists Beyond Expressibility

What the math gives to ML

The paper's transferable contribution is an identity-preserving method for increasing model depth without causing an optimization shock. Appending a forward/inverse pair keeps the represented function unchanged at insertion time, while independently trainable parameters create a larger optimization space after training resumes. The reachable-set saturation result also suggests separating expressive expansion from parameter overcapacity: introduce expensive global mixing only once, then add cheap local transformations when additional depth is useful. A practical classical analogue is progressive insertion of reversible or analytically invertible residual blocks, combined with an acceptance rule that rejects expansions that immediately worsen validation loss.