# Davis–Wielandt Certified Residual Blocks

- ID: 681
- Canonical URL: https://synthcore.org/idea/681/davis-wielandt-certified-residual-blocks
- API JSON: https://synthcore.org/api/idea/681.json
- API Markdown: https://synthcore.org/api/idea/681.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.10335](https://arxiv.org/abs/2607.10335)
- Category: dynamics
- Solves: stability, accuracy
- ML areas: optimizer, training-dynamics, mlp, transformer
- Math tags: control-theory, dynamical-systems, linear-algebra, spectral-theory, geometry
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Replace unconstrained residual updates with blocks whose Jacobian is monitored through a Davis–Wielandt shell. The shell simultaneously measures directional dissipation and non-normal amplification, yielding a per-block step-size or residual-scale bound that is stronger than checking only the largest eigenvalue or spectral norm.

## Mathematical statement

For a real or complex block Jacobian A in C^(d x d) and any unit vector x, define the Davis–Wielandt shell DW(A) as the set of pairs (u,v), where u = Re(x* A x) and v = ||A x||_2^2. For an explicit residual step x_(k+1) = x_k + h A x_k, its directional squared gain is ||x_(k+1)||_2^2 / ||x_k||_2^2 = 1 + 2 h u + h^2 v. Therefore, a sufficient contraction certificate with margin gamma > 0 is 1 + 2 h u + h^2 v <= 1 - gamma for every point (u,v) in DW(A). If u <= -alpha < 0 and v <= beta, a conservative step-size condition is 0 < h < 2 alpha / beta, with contraction factor bounded by 1 - 2 h alpha + h^2 beta. In a neural block, A is the input Jacobian J_f(z) = partial f_theta(z) / partial z and the residual update is z_plus = z + h f_theta(z). The DW shell is estimated using random unit probes x_j, Jacobian-vector products y_j = J_f(z) x_j, u_j = Re(x_j* y_j), and v_j = ||y_j||_2^2.

## Key formulas

- $$\mathcal{DW}(A)=\left\{\left(\operatorname{Re}(x^{*}Ax),\,\|Ax\|_{2}^{2}\right):\|x\|_{2}=1\right\}.$$
- $$\frac{\|(I+hA)x\|_{2}^{2}}{\|x\|_{2}^{2}}=1+2h\operatorname{Re}(x^{*}Ax)+h^{2}\|Ax\|_{2}^{2}=1+2hu+h^{2}v.$$
- $$\sup_{(u,v)\in\mathcal{DW}(A)}\left(1+2hu+h^{2}v\right)\le 1-\gamma\quad\Longrightarrow\quad \|I+hA\|_{2}^{2}\le 1-\gamma.$$
- $$u\le-\alpha<0,\quad v\le\beta\quad\Longrightarrow\quad 0<h<\frac{2\alpha}{\beta},\qquad \rho_{\mathrm{bound}}(h)=1-2h\alpha+h^{2}\beta<1.$$

## Implementation notes

1. Integration point: use this in residual MLPs, Transformer feed-forward residual branches, or continuous-depth networks. Let a block be z_plus = z + s_l f_(theta_l)(z), where s_l is a learned or scheduled residual scale. Before each optimizer update, and periodically during training, estimate the DW projection of the block Jacobian on minibatch activations. 2. Pseudocode: for each block and minibatch, draw m normalized random probes x_j; compute y_j = J_f(z) x_j with forward-mode JVPs or reverse-mode vector-Jacobian products; compute u_j = x_j transpose y_j and v_j = ||y_j|| squared; set alpha_hat = -max_j u_j and beta_hat = max_j v_j; choose s_l = min(s_l, 2(1-epsilon) alpha_hat / beta_hat) when alpha_hat > 0, otherwise shrink the block and add the penalty max_j[1 + 2 s_l u_j + s_l squared v_j - 1 + gamma]_+. 3. Computed versus estimated: the residual-gain identity and DW contraction condition are analytic; shell extrema are estimated from probes, minibatch states, and optionally endpoints of a prescribed activation region. Use m = 16 initially and retain exponential-moving maxima of u_j and v_j to reduce false certificates. 4. First cheap experiment: train a 6-layer residual MLP on MNIST or CIFAR-10 with ordinary residual scaling versus DW scaling, using identical Adam settings. Sweep the nominal residual step s and measure activation-norm growth and gradient stability. The quantitative prediction is a divergence boundary near s_c approximately equal to 2 alpha_hat / beta_hat; stable operation should persist below (1-epsilon)s_c and fail or show exploding activation norms above s_c for blocks with approximately fixed Jacobians. Also test the predicted decay law ||delta z_k|| squared less than or approximately equal to (1 - 2 s alpha_hat + s squared beta_hat)^k ||delta z_0|| squared by injecting two nearby inputs and fitting the log-distance slope.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built and numerically verified a Davis–Wielandt residual-block MVP. The gain identity held to 6.1e-16 absolute error; increasing non-normality K from 0 to 1.9 reduced the certified boundary from 2.0 to 0.0184, while the true contraction boundary decreased from 2.0 to 0.1, confirming sensitivity to non-normal amplification. The certificate was conservative rather than tight: for K=1.4, predicted h=0.1626 versus observed operator contraction boundary h=0.6. The certified scale prevented amplification, but no neural-network accuracy or practical training win was tested.

### Mechanism check

- Verdict: Built and numerically verified a Davis–Wielandt residual-block MVP. The gain identity held to 6.1e-16 absolute error; increasing non-normality K from 0 to 1.9 reduced the certified boundary from 2.0 to 0.0184, while the true contraction boundary decreased from 2.0 to 0.1, confirming sensitivity to non-normal amplification. The certificate was conservative rather than tight: for K=1.4, predicted h=0.1626 versus observed operator contraction boundary h=0.6. The certified scale prevented amplification, but no neural-network accuracy or practical training win was tested.
- Confidence: 8/10
- Limitations: This is a small exact linear 2x2 NumPy toy experiment, not a trained MNIST/CIFAR residual MLP. Random shell probes were used rather than forward-mode JVPs in a neural block; probe estimates can miss extrema, and no optimizer, activation-region drift, runtime, memory, or accuracy comparison was measured.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/128/bench_report.json)
- [bench_report_rerun.json](https://synthcore.org/code/128/bench_report_rerun.json)
- [dw_experiment.py](https://synthcore.org/code/128/dw_experiment.py)
- [report.md](https://synthcore.org/code/128/report.md)
- [report_bench_2026-08-30T162524.md](https://synthcore.org/code/128/report_bench_2026-08-30T162524.md)
- [results.json](https://synthcore.org/code/128/results.json)
- [stage2_dw_bench.py](https://synthcore.org/code/128/stage2_dw_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/128)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.