# NTK-guided multilevel residual continuation

- ID: 310
- Canonical URL: https://synthcore.org/idea/310/ntk-guided-multilevel-residual-continuation
- API JSON: https://synthcore.org/api/idea/310.json
- API Markdown: https://synthcore.org/api/idea/310.md
- Verification status: unverified
- Source: [arXiv:2607.00560](https://arxiv.org/abs/2607.00560)
- Category: training
- Solves: speedup, stability, sample-efficiency
- ML areas: optimizer, training-dynamics, loss
- Math tags: spectral-theory, functional-analysis, numerical-analysis, optimization
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Train an operator-residual network on a ladder of increasingly accurate discretizations instead of optimizing the finest residual from the beginning. At each level, reuse the previous parameters and introduce finer collocation or quadrature points only after the current residual has reached a plateau, allowing the network's spectral bias to act as a coarse-to-fine smoother.

## Mathematical statement

The paper starts from the second-kind boundary integral equation \(\mathcal A\rho=g\), with \(\mathcal A=\tfrac12 I+\mathcal K\) and \((\mathcal K\rho)(x)=\int_\Gamma k(x,y)\rho(y)\,dS(y)\), where \(\rho\) is the unknown boundary density, \(g\) is prescribed data, \(\Gamma\) is the boundary, and \(k\) is the integral kernel. For discretization level \(\ell\), let \(A_\ell\) and \(g_\ell\) be the quadrature or collocation approximation and define \(r_\ell(\theta)=A_\ell\rho_\theta-g_\ell\), where \(\rho_\theta\) is the neural representation and \(\theta\) its parameters. With Jacobian \(J_\ell=\partial r_\ell/\partial\theta\) and residual weighting matrix \(W_\ell\succeq0\), gradient descent on \(L_\ell=\tfrac12r_\ell^TW_\ell r_\ell\) gives \(\theta_{t+1}=\theta_t-\eta J_\ell^TW_\ell r_\ell\). Under local linearization, the residual evolves as \(r_{t+1}\approx(I-\eta K_\ell W_\ell)r_t\), where \(K_\ell=J_\ell J_\ell^T\) is the empirical neural tangent kernel. Thus modes associated with small eigenvalues of \(K_\ell W_\ell\) contract slowly. The adaptation uses levels \(\ell=0,\ldots,L\), warm-starting \(\theta^{(\ell+1)}_0=\theta^{(\ell)}_{T_\ell}\).

## Key formulas

- $$\mathcal{A}\rho:=\tfrac{1}{2}\rho+\mathcal{K}\rho=g,\qquad (\mathcal{K}\rho)(x):=\int_{\Gamma}k(x,y)\rho(y)\,dS(y).$$
- $$r_\ell(\theta)=A_\ell\rho_\theta-g_\ell,\qquad L_\ell(\theta)=\tfrac12 r_\ell(\theta)^T W_\ell r_\ell(\theta).$$
- $$\theta_{t+1}=\theta_t-\eta J_\ell(\theta_t)^T W_\ell r_\ell(\theta_t),\qquad J_\ell=\frac{\partial r_\ell}{\partial\theta}.$$
- $$r_{t+1}\approx\left(I-\eta K_\ell W_\ell\right)r_t,\qquad K_\ell=J_\ell J_\ell^T.$$

## Implementation notes

(1) Exact integration point: replace the ordinary fixed-collocation training loop in a PINN, boundary-integral network, or neural operator residual trainer. Keep one network \(\rho_\theta\), but create datasets \(D_0,\ldots,D_L\) with increasing numbers of points and, where relevant, increasing quadrature accuracy. For a PDE, \(A_\ell\rho_\theta-g_\ell\) can be the PDE, boundary, or integral-equation residual evaluated on level-\(\ell\) points. Use quadrature weights in \(W_\ell\), normalized so that the loss estimates the same continuum norm at every level. (2) Pseudocode: initialize \(\theta\); for levels \(\ell=0,\ldots,L\), load \(D_\ell\), compute \(r=A_\ell\rho_\theta-g_\ell\), and take minibatch updates \(\theta\leftarrow\theta-\eta J^TW_\ell r\) for at most \(T_\ell\) steps. Every validation interval, evaluate the normalized residual on held-out points. Advance when the moving-average loss reduction over the last \(q\) steps is below a threshold, or when the residual reaches a prescribed level target. Warm-start each level with the final parameters from the previous level. (3) Computed directly from the mathematics: the discretized operator residual, weighted least-squares loss, Jacobian-gradient update, and warm-start continuation. Estimated empirically: the plateau criterion and, optionally, the NTK spectrum. Estimate the largest and smallest relevant eigenvalues of \(K_\ell W_\ell\) with 5--10 power or Lanczos iterations using Jacobian-vector and vector-Jacobian products; never form the dense kernel. (4) First cheap experiment: solve a 2-D Poisson problem with a SIREN or standard MLP PINN using 1k, 4k, 16k, and 64k collocation points. Compare the schedule against direct training on 64k points with the same optimizer, parameter count, and total gradient FLOPs. Measure wall-clock time to a fixed held-out PDE residual, final relative solution error, and the number of fine-level updates. Success means at least 1.5x lower time-to-tolerance or substantially fewer fine-level updates without worse final error. A useful diagnostic is that validation residual modes or spatial-frequency errors that stall at coarse resolution decrease after refinement.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.