Moment-Tuple Propagation for Compressed Neural Inference
Implementation & benchmark of arXiv:2608.26912 — TOPIQ: Statistical Error Propagation for Quantity-of-Interest Prediction under Lossy Compression
Source paper: TOPIQ: Statistical Error Propagation for Quantity-of-Interest Prediction under Lossy Compression arXiv:2608.26912 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Represent each compressed input or activation as a tuple containing its clean mean, clean variance, compression bias, compression-error variance, and signal-error covariance. Propagate this tuple through affine layers exactly and through activations with a second-order Taylor correction, producing an approximate prediction mean and uncertainty from one compressed forward pass. This can replace repeated decompression or Monte Carlo perturbation when serving neural networks over lossy scientific data.
Formulas
Mathematical statement
The paper defines a shared five-tuple interface written as $T_x=(\mu,\sigma^2,b,v,c)$, where $\mu=\mathbb{E}[x]$ is the clean signal mean, $\sigma^2=\operatorname{Var}(x)$ is its data variance, $b=\mathbb{E}[e]$ is the mean compression error for $\hat{x}=x+e$, $v=\operatorname{Var}(e)$ is compression-error variance, and $c=\operatorname{Cov}(x,e)$ is signal-error coupling. For a constant shift and scalar multiplication, the paper gives $T_{x+a}=(\mu+a,\sigma^2,b,v,c)$ and $T_{kx}=(k\mu,k^2\sigma^2,kb,k^2v,kc)$. For a differentiable nonlinearity $f$, it uses the second-order expansion $f(\hat{x})\approx f(\mu)+f'(\mu)[(x-\mu)+e]+\tfrac12 f''(\mu)[(x-\mu)+e]^2$. Define $z=(x-\mu)+e$, $m_1=\mathbb{E}[z]=b$, and $m_2=\mathbb{E}[z^2]=\sigma^2+v+2c+b^2$. The adapted output mean is $\mu_y=f(\mu)+f'(\mu)b+\tfrac12 f''(\mu)m_2$. Under a Gaussian or second-moment closure, approximate $\operatorname{Var}(z)=s_z^2=\sigma^2+v+2c$ and use $\operatorname{Var}(f(\hat{x}))\approx [f'(\mu)+f''(\mu)b]^2s_z^2+\tfrac12[f''(\mu)]^2s_z^4$. This separates ordinary activation variability from compression-induced uncertainty and includes the curvature-induced bias correction.
Implementation notes
Integrate this at inference time into a small MLP operating on compressed scientific fields, starting with scalar or channelwise independent features. Each tensor element carries five buffers (mu, sig2, bias, errvar, cov). At input, obtain mu by decoding the compressed value or by using the compressor's reconstruction, and obtain bias, errvar, and cov from stored compression metadata or calibration samples; initialize sig2 from a training-set or spatial-window variance estimate. For an affine layer y=W x+d, propagate mu_y=W mu_x+d, sig2_y=W^2 sig2_x elementwise under independence, bias_y=W bias_x, errvar_y=W^2 errvar_x, and cov_y=W^2 cov_x for diagonal feature statistics. More accurately, retain a diagonal covariance estimate and use W diag(.) W^T for the variance buffers. For an activation, compute f(mu), fprime, and fsecond elementwise; set m2=sig2+errvar+2*cov+bias^2, then update mu=f(mu)+fprime*bias+0.5*fsecond*m2 and var=(fprime+fsecond*bias)^2*(sig2+errvar+2*cov)+0.5*fsecond^2*(sig2+errvar+2*cov)^2. Propagate bias and error variance as the difference between this tuple output and a clean-reference tuple, or initially use the same second-order decomposition with sig2 treated as intrinsic variability and errvar+2*cov as compression uncertainty. Estimate metadata statistics empirically on a held-out calibration set; the Taylor equations are fixed by the paper. First test a one-hidden-layer MLP on a compressed climate or fluid-field dataset, comparing ordinary decompression, 32-sample noise Monte Carlo, and tuple propagation at equal latency. Measure RMSE of predicted output means, coverage of nominal 90% intervals, and latency and memory. Success is at least 90% interval coverage with under 5% mean error while requiring one forward pass instead of 32 and storing less than 1% of raw-field metadata.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built experiment.py with five-tuple affine propagation, second-order tanh activation propagation, numerical moment checks, and a seeded MLP compression experiment. Affine identities were exact and the Taylor formulas were accurate in a small-variance local regime, but they failed badly for the broader test distribution. Tuple inference was about 4.6x faster than 32-sample Monte Carlo, yet nominal-90% interval coverage was only 75.7% versus 89.1% for Monte Carlo, so the promised uncertainty quality was not observed.
- Agent confidence
- 9/10
- Baseline
- RMSE to clean model output 0.10151; 0.000329 ms/sample
- Idea
- RMSE to clean output 0.10131; 75.7% nominal-90% coverage; 0.002252 ms/sample; 0.01814 RMSE versus MC mean
Limitations: Only a synthetic two-feature, one-hidden-layer MLP and uniform scalar quantization were tested. The implementation uses diagonal or independent feature statistics, does not calibrate compression metadata from real scientific fields, does not test CUDA, and does not measure metadata memory against the claimed 1% target.
How to run: python3 experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- experiment.py 5.6 KB View Raw
- report.md 1.3 KB View