# Residual-greedy latent basis expansion

- ID: 3067
- Canonical URL: https://synthcore.org/idea/3067/residual-greedy-latent-basis-expansion
- API JSON: https://synthcore.org/api/idea/3067.json
- API Markdown: https://synthcore.org/api/idea/3067.md
- Verification status: unverified
- Source: [arXiv:2609.02578](https://arxiv.org/abs/2609.02578)
- Category: training
- Solves: accuracy, sample-efficiency, scalability
- ML areas: world-model, curriculum, data-augmentation, fine-tuning
- Math tags: approximation-theory, optimization, linear-algebra, pde
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Construct a compact latent basis for a parameter-conditioned neural operator by repeatedly adding the parameter sample with the largest normalized prediction residual. This replaces uniform parameter coverage with adaptive basis growth and supplies a stopping rule for when the latent approximation is sufficiently accurate.

## Mathematical statement

The paper constructs V_N=span{f(.;mu_i)} from parameterized solution snapshots and selects new snapshots by a residual-based greedy rule. Let U_N=[u_1,...,u_N] be orthonormal basis vectors, c_N(mu) reduced coordinates, and y_N(mu)=U_N c_N(mu) the reduced prediction. Let F_theta(y;mu) be a model or governing-equation residual, b(mu) a scale vector, and epsilon>0 a numerical safeguard. Define rho_N(mu)=||F_theta(y_N(mu);mu)||/max(||b(mu)||,epsilon). Choose mu_{N+1}=argmax_{mu in P_train}rho_N(mu), append its snapshot s(mu_{N+1}), and orthogonalize it with u_{N+1}=(I-U_NU_N^T)s/||(I-U_NU_N^T)s||. Stop when the maximum residual is at most tau. The transferable property is adaptive approximation of a parameterized solution manifold using residual information rather than random sampling alone.

## Key formulas

- $$\mathbb{V}_{N}=\operatorname{span}\{f(\cdot\,;\bm{\mu}_{i})\}_{i=1}^{N},\qquad f_{\mathrm{rb}}(\cdot\,;\bm{\mu})=\sum_{i=1}^{N}c_i(\bm{\mu})u_i.$$
- $$\mu_{N+1}=\underset{\mu\in\mathcal{P}_{\mathrm{train}}}{\operatorname{arg\,max}}\;\rho_N(\mu),\qquad \rho_N(\mu)=\frac{\|F_\theta(y_N(\mu);\mu)\|}{\max(\|b(\mu)\|,\varepsilon)}.$$
- $$u_{N+1}=\frac{(I-U_NU_N^{\top})s(\mu_{N+1})}{\|(I-U_NU_N^{\top})s(\mu_{N+1})\|},\qquad \text{stop when }\max_{\mu\in\mathcal{P}_{\mathrm{train}}}\rho_N(\mu)\leq\tau.$$
- $$J(\bm{\mu})=\left\|T\left(\sum_{i=1}^{N}c_i(\bm{\mu})u_i\right)-T^{\mathrm{data}}\right\|_x^2.$$

## Implementation notes

(1) Integration point: use this procedure for adaptive data selection and latent-basis construction in a parametric neural operator or differentiable simulator. A snapshot s(mu) may be an encoder embedding, decoder output, hidden rollout state, or full predicted field compressed by randomized SVD. Define F_theta as one-step prediction error on a target batch; if a governing equation is available, concatenate its discretized residual. Do not use training loss alone, because it may miss poorly represented parameter regions.

(2) Pseudocode:
```text
P = candidate_parameter_samples
choose mu_1; S = [snapshot(mu_1)]; U = orthonormalize(S)
repeat:
  for mu in P:
    z = snapshot_or_encoder(mu)
    c = U.T @ z
    yN = U @ c
    rho[mu] = norm(residual(theta, yN, mu)) / max(norm(scale(mu)), eps)
  mu_new = argmax(rho)
  if rho[mu_new] <= tau: break
  s = snapshot(mu_new)
  s = s - U @ (U.T @ s)
  U = concatenate(U, s / (norm(s)+eps))
  optionally fine-tune theta on selected high-rho samples
```
(3) Orthonormalization, residual scoring, greedy argmax selection, and the threshold stopping rule come from the reduced-basis idea. The snapshot type, residual normalization, candidate pool, and threshold tau must be selected empirically. Compute scores in batches and validate them on a disjoint parameter set. (4) First cheap experiment: use a small Fourier neural operator or MLP neural operator on a 1D parametric advection-diffusion family with two to four parameters. Compare random, Latin-hypercube, and residual-greedy sampling using the same number of expensive training solves. Report test error versus snapshot count, worst-case error over parameters, and cost to reach a target error. Success is a 2x reduction in required snapshots or lower worst-case error at equal data-generation cost; for inverse estimation also compare observable misfit J(mu) and recovered-parameter error.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.