Adaptive CUR Neural Layer
Implementation & benchmark of arXiv:2607.06021 — A Sub-linear Low-Rank Solver for Poisson's Equation using Machine Learning Frameworks for GPU Acceleration
Source paper: A Sub-linear Low-Rank Solver for Poisson's Equation using Machine Learning Frameworks for GPU Acceleration arXiv:2607.06021 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a dense weight matrix by a cross approximation built from selected rows and columns rather than by a conventional truncated SVD. Periodically refresh the selected indices using residual leverage scores, warm-starting from the previous factorization so that the compressed layer can track weight changes during fine-tuning.
Formulas
Mathematical statement
The paper's cross approximation represents a matrix G in R^{m x n} using column indices J and row indices I as G approximately G(:,J) M G(I,:), where J contains r_1 selected columns, I contains r_2 selected rows, and M = G(I,J)^+ is the Moore-Penrose pseudoinverse of the intersection matrix. For a neural weight matrix W in R^{d_in x d_out}, define A = W(:,J), B = W(I,:), and M = W(I,J)^+; then W_hat = A M B. A row-vector input x is evaluated as ((x A) M) B, avoiding the dense d_in by d_out multiplication. To select informative indices, let Q_c and Q_r be orthonormal bases for column and row residual sketches. The statistical leverage scores are ell_j^col = ||Q_c[j,:]||_2^2 and ell_i^row = ||Q_r[i,:]||_2^2. The largest scores are selected, and M is recomputed with a truncated SVD pseudoinverse. The transferable asset is reconstruction from a small intersection and matrix slices, with adaptive index selection guided by subspace importance.
Implementation notes
Integrate this at the weight level of a linear or MLP projection, starting with Transformer feed-forward matrices or attention output projections. Let W have shape [d_in, d_out]. Store A = W[:, J], M, and B = W[I, :], and replace nn.Linear(x, W) by ((x @ A) @ M) @ B. Initialize from a pretrained dense layer by choosing ranks r1 and r2, extracting the initial rows and columns, and computing C = W[I, J]. Compute M with an SVD pseudoinverse: if C = U diag(s) V^T, retain singular values satisfying s_k > tau*s_max and set M = V diag(1/s_k) U^T. During training, refresh the index sets every T optimizer steps. Form a residual sketch R = W - A M B, use random probes P to compute S_c = R P and S_r = P^T R, orthonormalize the sketches by QR to obtain Q_c and Q_r, then calculate leverage scores ell_j^col = ||Q_c[j,:]||_2^2 and ell_i^row = ||Q_r[i,:]||_2^2. Select the highest-scoring indices while retaining a fraction of the old indices as a warm start, re-extract A and B, and recompute M. Freeze indices between refreshes to avoid discontinuous updates. Add an activation reconstruction term L = L_task + lambda ||XW - XAMB||_F^2 for representative activations X during a short distillation phase. The cross reconstruction and pseudoinverse are mathematical components; refresh period T, probe dimension, ranks, threshold tau, and retention fraction are empirical hyperparameters. First test on a pretrained CIFAR-10 MLP and a small WikiText-2 Transformer, comparing dense, fixed-random CUR, truncated SVD, and adaptive leverage-CUR. Measure parameter memory, actual latency, FLOPs, validation accuracy or perplexity, residual norm, and intersection condition number. A successful result is at least 2x lower weight memory with less than 1% accuracy degradation and lower latency at equal batch size.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built an adaptive CUR layer with truncated-SVD pseudoinverse, residual random-sketch leverage refresh, warm-start index retention, and factorized forward evaluation. The core identity was verified to numerical precision (maximum batch-output gap 2.3e-14), and a rank-8 matrix was recovered at about 1e-15 relative error with 2.74x parameter-count compression. However, after an out-of-index-set weight change, adaptive refresh had mean relative error 4.89 versus 0.864 for stale CUR and 1.52 for random CUR across 20 seeds, beating stale only once; the claimed adaptive tracking effect was not observed.
- Agent confidence
- 9/10
- Baseline
- Dense reconstruction error 0; stale fixed CUR post-change mean relative error 0.864; random CUR post-change mean relative error 1.521; dense parameters 5120.
- Idea
- Adaptive residual-leverage CUR post-change mean relative error 4.890, median 2.388, beating stale in 1/20 seeds; compressed factor count 1872 (2.74x fewer parameters); formula evaluation max absolute gap 2.33e-14.
Limitations: Only a small synthetic matrix experiment was tested; no CIFAR MLP, Transformer, optimizer fine-tuning, activation distillation, refresh-period training study, GPU benchmark, wall-clock latency, FLOP measurement, or condition-number mitigation was evaluated. The observed instability likely requires better sampling, rank/retention scheduling, or conditioning safeguards before neural-layer testing.
How to run: python3 adaptive_cur.py && /home/maxwelhelp/main/bin/python3 sweep.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.