# Identity-Paired Progressive Depth

- ID: 1029
- Canonical URL: https://synthcore.org/idea/1029/identity-paired-progressive-depth
- API JSON: https://synthcore.org/api/idea/1029.json
- API Markdown: https://synthcore.org/api/idea/1029.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.16800](https://arxiv.org/abs/2607.16800)
- Category: architecture
- Solves: stability, speedup, scalability
- ML areas: mlp, transformer, training-dynamics, initialization
- Math tags: linear-algebra, algebra, dynamical-systems, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Grow a neural network by appending a trainable block together with an analytically initialized inverse block, so the newly added depth is exactly the identity at insertion time. After insertion, untie and optimize the two blocks independently; this preserves the current function while providing additional trainable degrees of freedom. For architectures with one expensive mixing operation followed by cheap channelwise blocks, the same construction can increase depth without repeatedly paying for global mixing.

## Mathematical statement

The paper uses the mixed-product identity for tensor-product operators, $\left(\bigotimes_{q=1}^{n}\mathbf{A}_{q}\right)\left(\bigotimes_{q=1}^{n}\mathbf{B}_{q}\right)=\bigotimes_{q=1}^{n}(\mathbf{A}_{q}\mathbf{B}_{q})$, where $\mathbf A_q$ and $\mathbf B_q$ are square operators on qubit $q$. Its identity-paired construction relies on the cancellation law $U^{-1}U=I$: a forward rotation/entangler block followed by its reverse has no functional effect at initialization. We transfer this to an invertible neural block $B_\theta$ with an exactly computable inverse $B_\theta^{-1}$, and insert $B_{\theta_0}^{-1}\circ B_{\theta_0}$, where $\theta_0$ is the copied parameter tensor and $I$ is the identity map. The two copies are then untied, so the inserted pair remains identity only at insertion time, not as a permanent architectural constraint. For a classical residual implementation, exact inversion is guaranteed by additive coupling: for $x=(x_a,x_b)$, $C_{f,g}(x_a,x_b)=(x_a+f(x_b),x_b+g(x_a+f(x_b)))$ and $C_{f,g}^{-1}(y_a,y_b)=(y_a-f(y_b-g(y_a)),y_b-g(y_a))$. The continuation acceptance rule is $\mathcal L_{t+1}\leq\mathcal L_t$, where $\mathcal L_t$ is the monitored training or validation loss after expanding to depth $t$.

## Key formulas

- $$\left(\bigotimes_{q=1}^{n}\mathbf{A}_{q}\right)\left(\bigotimes_{q=1}^{n}\mathbf{B}_{q}\right)=\bigotimes_{q=1}^{n}(\mathbf{A}_{q}\mathbf{B}_{q})$$
- $$U^{-1}U=I,\qquad F_{\mathrm{new}}(x)=B_{\theta_0}^{-1}(B_{\theta_0}(F_{\mathrm{old}}(x)))=F_{\mathrm{old}}(x)$$
- $$C_{f,g}(x_a,x_b)=\left(x_a+f(x_b),\;x_b+g\bigl(x_a+f(x_b)\bigr)\right)$$
$$C_{f,g}^{-1}(y_a,y_b)=\left(y_a-f\bigl(y_b-g(y_a)\bigr),\;y_b-g(y_a)\right)$$
- $$\mathcal L_{t+1}\leq\mathcal L_t\quad\text{accept the depth expansion; otherwise restore depth }t$$

## Implementation notes

Integrate this at the depth-growth mechanism of a residual MLP, encoder-only Transformer, or sequence model. Prefer additive-coupling blocks because their inverse is explicit and does not require a matrix inverse. Split each hidden vector into two equal channel groups $(x_a,x_b)$ and implement one coupling block $C_{f,g}$ using small MLPs or depthwise linear layers for $f$ and $g$. During progressive training, first train a shallow model. To expand it, copy an existing coupling block with parameters $\theta_0$, append $C_{\theta_0}$, and immediately append its inverse $C^{-1}_{\theta_0}$; the forward pass is unchanged up to floating-point error. At the next optimizer step, replace the copied parameters by two independent parameter sets $\theta_+$ and $\theta_-$, while retaining the inverse-form computation for the second block only at initialization. In a simpler residual variant, use an invertible linear or orthogonal block $B_W(x)=Wx$ and append $W^{-1}W$, estimating $W^{-1}$ exactly from a stored orthogonal parameterization. For a Transformer, apply the pair around an FFN or around one attention-mixing layer; test a low-cost version in which only the first block contains attention and later paired blocks contain channelwise FFNs. After every expansion, run a short fixed budget of minibatches, evaluate validation loss, and accept the expansion only if $\mathcal L_{t+1}\leq\mathcal L_t+\epsilon$; otherwise remove the pair or reduce its learning rate. What is exact from the mathematics is the identity cancellation and reversible inverse. What must be estimated empirically is whether the extra untied parameters improve optimization after cancellation. The first experiment should use a 6-layer versus progressively grown 12-layer Transformer on WikiText-2 or CIFAR-10, with standard same-width training as baseline. Measure loss immediately before and after insertion, loss after 1,000 updates, gradient norm spikes, wall-clock cost, and final validation accuracy. Success means no insertion loss spike, lower loss at equal optimizer steps, and at least comparable final accuracy despite starting from the shallow model.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built an additive-coupling identity-paired depth-growth module with an explicit inverse, plus reproducible math sweeps and a mini regression experiment. The mechanism manifested: inverse composition remained numerically identity, cancellation error increased with coupling scale, and untied output perturbation scaled linearly with step size. Insertion caused only a 7.0e-8 relative loss change, but the paired model did not beat continued shallow training after 300 updates in this tiny task.

### Mechanism check

- Verdict: Built an additive-coupling identity-paired depth-growth module with an explicit inverse, plus reproducible math sweeps and a mini regression experiment. The mechanism manifested: inverse composition remained numerically identity, cancellation error increased with coupling scale, and untied output perturbation scaled linearly with step size. Insertion caused only a 7.0e-8 relative loss change, but the paired model did not beat continued shallow training after 300 updates in this tiny task.
- Confidence: 8/10
- Limitations: The experiment used a tiny synthetic regression problem rather than CIFAR-10, WikiText-2, or a Transformer. It tested one seed, one width, and one expansion, with no large-scale FLOP-normalized comparison or validation-based rollback sweep.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.5933
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_identity_paired.py](https://synthcore.org/code/421/bench_identity_paired.py)
- [bench_report.json](https://synthcore.org/code/421/bench_report.json)
- [identity_paired_experiment.py](https://synthcore.org/code/421/identity_paired_experiment.py)
- [report.md](https://synthcore.org/code/421/report.md)
- [report_bench_2026-08-31T114732.md](https://synthcore.org/code/421/report_bench_2026-08-31T114732.md)
- [results.json](https://synthcore.org/code/421/results.json)
- [Download all files as ZIP](https://synthcore.org/download/421)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.