# Sensitivity-Conditioned Neural ODE Pruning

- ID: 2218
- Canonical URL: https://synthcore.org/idea/2218/sensitivity-conditioned-neural-ode-pruning
- API JSON: https://synthcore.org/api/idea/2218.json
- API Markdown: https://synthcore.org/api/idea/2218.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.13044](https://arxiv.org/abs/2608.13044)
- Category: architecture
- Solves: stability, generalization, speedup
- ML areas: ssm, rnn, pruning, inference-speedup, regularization
- Math tags: dynamical-systems, statistics, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 8/10

## Idea description

Use trajectory sensitivities to remove neural units or parameter groups whose effects are redundant over the available data support. A parameter group is pruned when its Fisher contribution is small or its sensitivity is nearly collinear with other groups, producing a compact neural ODE without relying only on parameter magnitude.

## Mathematical statement

Partition the parameter vector into groups $\theta=(\theta_1,\ldots,\theta_G)$, where a group may be one hidden unit, one neuron's incoming weights, or one mechanistic coefficient. Let $J_g$ be the stacked observed-output sensitivity for group $g$, with $J_g$ formed from $H_kS(t_k)[:,g]$ over all observation times. The group information contribution is $I_g=J_g^TWJ_g$, where $W$ is the block-diagonal inverse observation-noise covariance. A group is redundant when its sensitivity is small, $\operatorname{tr}(I_g)$ is low, or its sensitivity lies in the span of other groups, measured by $r_g=\|(I-P_{-g})W^{1/2}J_g\|_F^2$, where $P_{-g}$ projects onto the weighted sensitivity span of all remaining groups. Iterative pruning keeps groups with sufficient incremental sensitivity rank and retrains after each pruning round.

## Key formulas

- $$J_g=\begin{bmatrix}H_1S(t_1)[:,g]\\ \vdots\\ H_KS(t_K)[:,g]\end{bmatrix},\qquad I_g=J_g^TWJ_g.$$
- $$r_g=\left\|(I-P_{-g})W^{1/2}J_g\right\|_F^2,\qquad P_{-g}=A_{-g}(A_{-g}^TA_{-g}+\epsilon I)^{-1}A_{-g}^T,$$
- $$\theta_g\ \text{is pruned if}\quad \operatorname{tr}(I_g)<\delta_I\ \text{or}\ r_g<\delta_R.$$

## Implementation notes

1. Exact integration point: apply this after ordinary neural ODE training, before final retraining, to hidden units or parameter blocks in the neural component of the vector field. Do not prune using raw parameter magnitude; collect each group's sensitivity over the actual training trajectories and observed time points. 2. Pseudocode: train a candidate NODE; solve the augmented state-sensitivity system or obtain $S_k$ with automatic differentiation; construct weighted sensitivity columns $A_g=W^{1/2}J_g$; compute each group's $\operatorname{tr}(I_g)$ and residual $r_g$ against the span of all other groups; remove the lowest-scoring group if either score is below threshold; retrain the reduced model; repeat until validation error increases or every remaining group exceeds the threshold. Use QR factorization or randomized low-rank linear algebra rather than explicitly forming a dense Fisher matrix for large networks. 3. Taken from the mechanism: sensitivity-based information and redundancy tests. Estimated empirically: pruning thresholds $\delta_I,\delta_R$, observation-noise weighting $W$, and the stopping criterion. 4. First cheap experiment: use a width-64 neural ODE to model noisy Lotka-Volterra or damped-oscillator trajectories, then compare magnitude pruning, random pruning, and sensitivity-conditioned pruning at equal parameter budgets of 50%, 25%, and 10% of the original size. Measure one-step error, long-horizon trajectory error, solver function evaluations, parameter count, and the Fisher condition number. The prediction is that sensitivity pruning removes parameters with low incremental Fisher rank while preserving validation error, whereas magnitude pruning sometimes removes dynamically important but small-weight parameters. A successful run should show approximately unchanged short-horizon error, lower long-horizon error than the unpruned overparameterized model, and solver cost decreasing with retained units; after retraining, the smallest nonzero Fisher eigenvalue should not decrease by more than 20%.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a compact neural-ODE unit-pruning experiment with weighted Fisher traces and leave-one-group sensitivity residuals, plus reproducible numerical verification. The mechanism checks passed: observed exponents were 2.000 for both information and residual amplitude scaling, -2.000 for noise scaling, and the collinearity residual matched C·sin(angle)^2 with ratio CV 1.0e-15. At 8/16 retained units, sensitivity pruning reached MSE 1.27e-6 and terminal error 0.00221 versus magnitude pruning MSE 2.31e-6 and terminal error 0.00297, but random pruning was better at MSE 3.94e-7 and terminal error 0.00131; therefore the mathematical mechanism manifested, but a practical accuracy win over random pruning was not demonstrated.

### Mechanism check

- Verdict: Built a compact neural-ODE unit-pruning experiment with weighted Fisher traces and leave-one-group sensitivity residuals, plus reproducible numerical verification. The mechanism checks passed: observed exponents were 2.000 for both information and residual amplitude scaling, -2.000 for noise scaling, and the collinearity residual matched C·sin(angle)^2 with ratio CV 1.0e-15. At 8/16 retained units, sensitivity pruning reached MSE 1.27e-6 and terminal error 0.00221 versus magnitude pruning MSE 2.31e-6 and terminal error 0.00297, but random pruning was better at MSE 3.94e-7 and terminal error 0.00131; therefore the mathematical mechanism manifested, but a practical accuracy win over random pruning was not demonstrated.
- Confidence: 8/10
- Limitations: Only a scalar damped nonlinear ODE and one 50% retention budget were tested; the requested 25% and 10% budgets, Lotka-Volterra trajectories, validation stopping, solver function-evaluation speed, Fisher condition numbers, and multiple random seeds were not evaluated. The sensitivity estimate uses finite differences through a fixed-step RK4 integrator rather than an augmented continuous adjoint sensitivity solve.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/881/bench_report.json)
- [bench_sensitivity_pruning.py](https://synthcore.org/code/881/bench_sensitivity_pruning.py)
- [report.md](https://synthcore.org/code/881/report.md)
- [report_bench_2026-09-01T135357.md](https://synthcore.org/code/881/report_bench_2026-09-01T135357.md)
- [results.json](https://synthcore.org/code/881/results.json)
- [sensitivity_pruning_experiment.py](https://synthcore.org/code/881/sensitivity_pruning_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/881)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.