△ Mechanism confirmed, baseline not beaten 2026

Accumulator-Carrying Picard ResNet

Implementation & benchmark of arXiv:2609.03626 — Residual neural networks overcome the curse of dimensionality for semilinear heat equations

Usefulness7/10

Difficulty4/10

Novelty5/10

Architecture Scalability Stability Accuracy Mlp Training dynamics World model Pde Stochastic processes Approximation theory Dynamical systems

Source paper: Residual neural networks overcome the curse of dimensionality for semilinear heat equations arXiv:2609.03626 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Build a residual module whose state explicitly contains both a persistent context representation and an accumulator. Each residual branch computes one learned correction and adds it to the accumulator, instead of forcing every layer to represent the complete output from scratch. This provides a concrete solver-like architecture for high-dimensional regression and iterative latent prediction.

Formulas

$$(\tfrac{\partial}{\partial t}u_d)(t,x)=\tfrac{1}{2}\Delta_xu_d(t,x)+f\big(u_d(t,x)\big),\qquad u_d(0,x)=g_d(x).$$

$$\mathfrak{R}_{a}(\Theta_{2}\ast\Theta_{1})=\mathfrak{R}_{a}(\Theta_{2})\circ\mathfrak{R}_{a}(\Theta_{1}),\qquad \mathfrak{P}(\Theta_{2}\ast\Theta_{1})=\mathfrak{P}(\Theta_{1})+\mathfrak{P}(\Theta_{2}).$$

$$z_k=(h(x),s_k),\qquad z_{k+1}=z_k+B_k(z_k),\qquad s_{k+1}=s_k+\Delta_k(h(x),s_k).$$

$$\mathfrak{P}(\Psi_{d,\varepsilon})\le C_{\xi}d^{4+\xi}\varepsilon^{-(3+\xi)},\qquad \xi>0.$$

Mathematical statement

The paper studies the semilinear heat equation $(\partial_t u_d)(t,x)=\frac{1}{2}\Delta_xu_d(t,x)+f(u_d(t,x))$ with $u_d(0,x)=g_d(x)$, where $x\in\mathbb{R}^d$ and $f:\mathbb{R}\to\mathbb{R}$ is globally Lipschitz. Its constructive proof represents a deterministic multilevel Picard estimator by a ResNet: shortcut coordinates transmit the spatial variable and an accumulator, while residual branches add successive estimator summands. The composition result is $\mathfrak{R}_a(\Theta_2\ast\Theta_1)=\mathfrak{R}_a(\Theta_2)\circ\mathfrak{R}_a(\Theta_1)$ and $\mathfrak{P}(\Theta_2\ast\Theta_1)=\mathfrak{P}(\Theta_1)+\mathfrak{P}(\Theta_2)$, where $\mathfrak{R}_a$ is the realization of a ResNet with activation $a$, $\ast$ concatenates blocks, and $\mathfrak{P}$ counts parameters. The adapted state is $z_k=(h(x),s_k)$, with context $h(x)$ transmitted through the shortcut and accumulator update $s_{k+1}=s_k+\Delta_k(h(x),s_k)$. The paper reports the explicit ridge-sum scaling $\mathfrak{P}(\Psi_{d,\varepsilon})\le C_\xi d^{4+\xi}\varepsilon^{-(3+\xi)}$ for every $\xi>0$, under its network-approximability and globally Lipschitz-truncation assumptions; $d$ is input dimension, $\varepsilon$ is target $L^2$ error, and $C_\xi$ is independent of $d$ and $\varepsilon$.

Implementation notes

Integrate this at the architecture level as a replacement for repeated MLP blocks in a high-dimensional regression or iterative prediction model. Given $x\in\mathbb{R}^d$, compute a context embedding $h_0=E(x)\in\mathbb{R}^q$ and initialize an accumulator $s_0\in\mathbb{R}^r$ using a learned constant or a small head. Form $z_0=[h_0,s_0]$. For $k=0,\ldots,K-1$, apply a branch $B_k$ to the concatenated state, split its output into context and accumulator corrections, and update h = h + alpha[k] * dh and s = s + beta[k] * ds. To mimic the paper, set alpha[k]=0 initially or constrain it to be small, so the context is transmitted by a shortcut, and set beta[k]=1 so branches add corrections to the accumulator. Use tanh or clipped preactivations to make each branch bounded and approximately globally Lipschitz. Finish with an output head $H([h_K,s_K])$. Pseudocode is: h=E(x); s=s0; for k in range(K): dh,ds=B[k](concat(h,s)); h=h+alpha[k]*dh; s=s+beta[k]*ds; return H(concat(h,s)). The paper-derived parts are the persistent context, additive accumulator, and compositional parameter accounting; widths, number of branches, and gates are empirical. First test on synthetic targets from a $d=50,100,200$ semilinear heat-equation Monte Carlo estimator, using a same-parameter plain MLP and standard ResNet as baselines. Record test MSE, loss versus training FLOPs, gradient variance, and parameter count versus $d$. Success means lower MSE or more stable optimization at equal parameters, and polynomial rather than rapidly worsening performance as $d$ increases.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an accumulator-carrying ResNet with persistent context and additive accumulator updates, plus plain-MLP and standard-ResNet controls. The telescoping accumulator identity and block-composition check passed to numerical precision (8.9e-16 and 0 error). At matched parameters, the accumulator had lower test MSE than the standard ResNet at d=50,100,200 (0.0860 vs 0.1041; 0.1185 vs 0.1371; 0.1139 vs 0.1255), a promising small-scale signal, but it was not consistently better than the plain MLP.

Agent confidence: 8/10

Baseline: Standard ResNet: d=50/100/200 test MSE = 0.104082/0.137112/0.125480; parameters = 16310/17910/21110.
Idea: Accumulator ResNet: d=50/100/200 test MSE = 0.086046/0.118547/0.113869; parameters = 16310/17910/21110. Plain MLP control MSE = 0.095627/0.124152/0.101153.

Limitations: This is a small synthetic regression benchmark rather than a semilinear heat-equation Monte Carlo estimator. Only one seed and 450 optimization steps were tested; FLOPs, gradient variance, multiple seeds, and broader widths/depths were not measured. The plain MLP has a different parameter count, and the context gate was fixed at alpha=0, so the result does not validate the paper's approximation or dimensional-scaling theorem.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03626) Sep 4, 2026

Significantly WORSE than the tuned baseline (+131.3%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 10.5936
Idea mean: 24.5047
Effect (Δ): 13.9111 (+131.3%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±12.2%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in tabular Friedman#1 track was tested; vision, sequence, dynamics, and a PDE-specific custom track were not tested. The accumulator model has a higher parameter count than mlp_tiny despite matched training conditions, and the mechanism signature verifies additive behavior rather than approximation quality or the paper's dimensional-scaling theorem.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 10/10

Built and evaluated an accumulator-carrying MLP against the standard bench mlp_tiny on Friedman#1 regression, using the same training path, shared learning-rate sweep, and eight paired seeds. The mechanism signature was confirmed from trained-model behavior, but the accumulator had substantially higher test MSE (24.505 vs 10.594), with paired delta +13.911 and permutation p=0.0081; therefore it is a significant loss, not a win.

Baseline: Friedman#1 mlp_tiny: best lr=0.006, 18 epochs; full 8-seed test MSE mean=10.593586, std=1.186041; per-seed=[9.266414,10.942157,10.532804,9.811106,9.271466,10.483624,11.298464,13.142654]; parameter count=4929.
Baseline sweep: lr=0.0015 mean=16.413998; lr=0.003 mean=13.256238; lr=0.006 mean=10.138120 on sweep seeds 0-3.
Idea: AccumulatorMLP: best among shared lr sweep at lr=0.003, 18 epochs; full 8-seed test MSE mean=24.504708, std=1.473834; per-seed=[25.353020,24.542402,26.259078,22.042612,22.800880,25.556654,23.438637,26.044382]; parameter count=7076.
Idea sweep: lr=0.0015 mean=48.017033; lr=0.003 mean=24.504708; lr=0.006 mean=24.529536, all at 18 epochs.
Mechanism signature: Observed trained accumulator branch correction norms=[0.999964,0.999960] mean per branch, final accumulator norm=2.128705, prediction standard deviation=0.064016; confirmed=true for finite nonzero additive accumulated behavior.
Paired comparison: delta_mean=+13.911122 (idea-baseline, lower is better), idea_wins=0/8, permutation p=0.0081, verdict='idea worse (significant)'

Benchmark result