△ Mechanism confirmed, baseline not beaten 2026

Task-Tangent Capture Pruning

Implementation & benchmark of arXiv:2608.27748 — The role of parameter Jacobians in the stability of network outputs

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Memory Stability Accuracy Pruning Training dynamics Fine tuning Functional analysis Linear algebra Dynamical systems

Source paper: The role of parameter Jacobians in the stability of network outputs arXiv:2608.27748 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Prune parameter directions according to how much task-relevant Jacobian energy they carry, rather than by weight magnitude or individual gradient magnitude. Keep a mask whose discarded tangent component is at most an empirical fraction epsilon of the full tangent vector for calibration task directions, thereby preserving the local output dynamics seen by the task.

Formulas

$$\|(I-P)T^{*}u\|_{H}\leq\epsilon\|T^{*}u\|_{H}$$

$$\widehat r(P)^2=\frac{\sum_{i=1}^{m}\|(I-P)T^{*}u_i\|_2^2}{\sum_{i=1}^{m}\|T^{*}u_i\|_2^2}\leq\epsilon^2$$

$$G=TT^{*},\qquad T_P=TP,\qquad G_P=T_PT_P^{*}=TPT^{*}$$

$$\dot z(t)=-Gz(t)\quad\longrightarrow\quad\dot z_P(t)=-G_Pz_P(t)$$

Mathematical statement

The paper's relative task-capture condition is $\|(I-P)T^{*}u\|_{H}\leq\epsilon\|T^{*}u\|_{H}$. Here $H$ is the parameter Hilbert space with its Euclidean inner product, $T:H\to Y$ is the network parameter Jacobian mapping parameter perturbations to output perturbations, $T^{*}:Y\to H$ is its adjoint, $u\in Y$ is a task-space output direction, and $P:H\to H$ is an orthogonal pruning projection that retains the unpruned parameter coordinates. The left side is the norm of the parameter update direction discarded by pruning, while the right side is the total parameter tangent direction induced by $u$. The condition is imposed for task-relevant $u$, such as residuals, label directions, or a low-dimensional basis of the cyclic task space generated by the output-space operator $G=TT^{*}$ and task subspace $M$. For a finite calibration set with output-direction matrix $U=[u_1,\ldots,u_m]$, use the empirical squared ratio $\widehat r(P)^2=\frac{\sum_i\|(I-P)T^{*}u_i\|_2^2}{\sum_i\|T^{*}u_i\|_2^2}\leq\epsilon^2$. The ML adaptation uses $T_P=TP$ and compares the full local kernel $G=TT^{*}$ with the pruned kernel $G_P=T_PT_P^{*}=TPT^{*}$; low tangent loss is expected to preserve short-time linearized training dynamics.

Implementation notes

Integrate this into structured or unstructured weight pruning after a short calibration phase, using the parameter Jacobian of network outputs on a small representative dataset. (1) Exact integration point: choose a candidate binary parameter mask $P$ for one layer or the whole model; take the Jacobian from output logits or regression outputs with respect to candidate weights, and make pruning decisions before the main fine-tuning run. Do not form the full Jacobian: use reverse-mode vector-Jacobian products to compute $g_i=T^*u_i$ for each calibration direction. (2) Pseudocode: collect calibration examples $(x_i,y_i)$; compute outputs $f_\theta(x_i)$; set $u_i=f_\theta(x_i)-y_i$, or use several random output-space directions for classification; compute $g_i=T^*u_i$; initialize a mask with magnitude pruning or all weights retained; for each candidate block or weight set $S$, score removal by $q(S)=\sum_i\|g_i[S]\|_2^2/\sum_i\|g_i\|_2^2$; greedily remove the lowest-score candidates while $q(S)\leq\epsilon^2$, setting $P_{jj}=0$ for removed coordinates and one otherwise; then fine-tune the masked network. (3) The mathematical quantity is the discarded-to-total tangent norm ratio; task directions, mask granularity, and greedy approximation are engineering choices. Estimate the ratio on held-out calibration batches and optionally refresh it after each fine-tuning epoch. For a dynamics diagnostic, estimate kernel-vector products and monitor $\|(G-G_P)v\|/\|Gv\|$ for random output vectors $v$. (4) First experiment: use a ResNet-20 on CIFAR-10 or a 6-layer Transformer on a small language-modeling corpus, comparing magnitude pruning, gradient pruning, and tangent-capture pruning at 50% and 80% sparsity. Test $\epsilon\in\{0.05,0.1,0.2\}$. Measure immediate validation loss increase, fine-tuning loss descent at equal optimizer steps, final accuracy or perplexity, and wall-clock and memory reduction with a genuinely sparse kernel. Success means a smaller immediate loss shock and faster recovery than magnitude pruning at equal sparsity, while the held-out empirical ratio stays below the selected epsilon.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible Task-Tangent Capture Pruning MVP with epsilon-controlled tangent-energy masks and a small masked-regression comparison against magnitude pruning. The mechanism manifested: discarded tangent ratios stayed below every epsilon target, the squared-energy identity matched to 1.1e-16, and tangent ranking discarded less task energy at all tested sparsities. In the toy regression, tangent pruning achieved lower final loss after 30 steps at both 50% and 80% pruning, although its immediate loss shock was larger in this single run.

Agent confidence: 8/10

Baseline: Magnitude-pruning task-energy ratios: 0.4866, 0.7363, 0.8798 at 25%, 50%, and 80% sparsity; final MSE 0.4182 and 0.4624 at 50% and 80%.
Idea: Tangent-pruning task-energy ratios: 0.2303, 0.4128, 0.6882 at 25%, 50%, and 80% sparsity; final MSE 0.3942 and 0.4111 at 50% and 80%. Epsilon sweep ratios were 0.04953, 0.09938, 0.19810, 0.29999, and 0.49954 for targets 0.05, 0.10, 0.20, 0.30, and 0.50.

Limitations: This was a toy random-operator verification plus one tiny tanh regression and one seed. It did not test ResNet/CIFAR, gradient pruning, held-out calibration generalization, mask refresh, genuinely sparse kernels, wall-clock speed, or memory reduction.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.27748) Sep 2, 2026

No significant difference vs. the tuned baseline (+0.7%, p=0.4521, wins 1 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.2611
Idea mean: 7.3093
Effect (Δ): 0.0483 (+0.7%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.4521 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±2.0%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the registered tabular track was tested. Vision, sequence, and dynamics tracks were not tested. The experiment used 50% coordinate sparsity, 96 calibration examples, and a small MLP; genuinely sparse-kernel speed, memory reduction, mask refresh, and additional sparsity levels were not measured.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 9/10

Task-Tangent Capture Pruning was implemented and evaluated on the structurally matched registered tabular Friedman regression track with the shared mlp_tiny architecture. The mechanism prediction held quantitatively: the observed mean discarded tangent ratio was 0.00271 and the maximum was 0.01561. The idea nevertheless had slightly higher test MSE than tuned magnitude pruning, with delta_mean=+0.04828 and p=0.4521, so it is not a benchmark win.

Baseline: Magnitude pruning, 50% sparsity, best lr=0.01: mean test MSE 7.261052 over 8 seeds.
Idea: Task-tangent pruning, 50% sparsity, best lr=0.01: mean test MSE 7.309336 over 8 seeds; delta_mean=+0.048284 and permutation p=0.4521.

Benchmark result