# Derivative-Jet MLP Compression

- ID: 40
- Canonical URL: https://synthcore.org/idea/40/derivative-jet-mlp-compression
- API JSON: https://synthcore.org/api/idea/40.json
- API Markdown: https://synthcore.org/api/idea/40.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.21752](https://arxiv.org/abs/2608.21752)
- Category: memory
- Solves: memory, scalability, accuracy
- ML areas: mlp, distillation, memory, inference-speedup
- Math tags: approximation-theory, functional-analysis, differential-geometry, optimization
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Compress a trained wide analytic-activation MLP by fitting a narrow same-depth student to the teacher's function values and input derivatives, rather than matching only outputs on a calibration dataset. Choose the student width from the input dimension and target error, with a target scaling m = O((log(1/epsilon))^d_in), and use sequential layer fitting plus channel reweighting to limit error accumulation through depth.

## Mathematical statement

The paper's central width-independent compressibility claim is that, for a fixed deep MLP with analytic activation, a narrow same-depth network can uniformly approximate the teacher on a bounded domain with width m = O((log(1/epsilon))^d_in). Here epsilon > 0 is the allowed uniform approximation error and d_in is the effective input dimension, not the teacher hidden width. The construction matches a finite derivative jet J_K f(x_0) = {partial^alpha f(x_0) : |alpha| <= K} at anchor points x_0, where alpha = (alpha_1,...,alpha_d_in) is a multi-index, |alpha| = sum_i alpha_i, and partial^alpha = partial_(x_1)^alpha_1 ... partial_(x_d_in)^alpha_d_in. Analyticity makes the Taylor remainder decrease rapidly with K, so K is proportional to log(1/epsilon). The number of multivariate derivatives through order K is binomial(K + d_in, d_in), which is O(K^d_in), giving the width dependence. The practical adaptation minimizes value and derivative mismatch, while channel rescaling preserves layer output scale.

## Key formulas

- $$m(\varepsilon,d_{\mathrm{in}})=O\!\left((\log(1/\varepsilon))^{d_{\mathrm{in}}}\right)$$
- $$J_K f(x_0)=\left\{\partial^\alpha f(x_0):\alpha\in\mathbb{N}_0^{d_{\mathrm{in}}},\ |\alpha|\le K\right\},\qquad \#J_K=\binom{K+d_{\mathrm{in}}}{d_{\mathrm{in}}}$$
- $$f(x)=\sum_{|\alpha|\le K}\frac{\partial^\alpha f(x_0)}{\alpha!}(x-x_0)^\alpha+R_{K+1}(x),\qquad (x-x_0)^\alpha=\prod_{i=1}^{d_{\mathrm{in}}}(x_i-x_{0,i})^{\alpha_i}$$
- $$\mathcal{L}_{\mathrm{jet}}=\sum_{a=1}^{A}\sum_{|\alpha|\le K}w_\alpha\left\|\partial^\alpha h_{\mathrm{student}}(x_a)-\partial^\alpha h_{\mathrm{teacher}}(x_a)\right\|_2^2+\lambda\sum_{a=1}^{A}\left\|h_{\mathrm{student}}(x_a)-h_{\mathrm{teacher}}(x_a)\right\|_2^2$$

## Implementation notes

Integrate this as an offline compressor for a trained wide MLP, not as a modification to the ordinary training forward pass. Let the teacher be h_0(x) = x and h_l(x) = phi(W_l h_(l-1)(x) + b_l), for layers l = 1,...,L, with analytic activation phi. Construct a student with the same depth L and target hidden width m. Set d_in from the actual input representation, choose derivative order K, and initialize m = c times binomial(K + d_in, d_in), testing several constants c. Generate A anchor points in the bounded deployment domain using a low-discrepancy sequence, a grid for low-dimensional inputs, or a calibration set. Use forward-mode automatic differentiation to compute teacher layer values and derivatives partial^alpha h_l at every anchor; begin with K = 1 because higher-order jets become expensive. Initialize the student by random projection, neuron clustering, or ordinary weight matching. Fit layers sequentially: freeze earlier student layers, compute student jets, and minimize the displayed L_jet with Adam. Use w_alpha = rho^|alpha| divided by alpha factorial so high-order derivatives do not dominate. After each layer, normalize each student channel by its activation RMS s_j, divide its incoming parameters by s_j + delta, and multiply the matching outgoing weight column by s_j + delta. This preserves the layer's scale while improving conditioning. The mathematics supplies the jet objective and width scaling; anchor density, constants, derivative order, and reweighting strength are empirical. First test a width-1024, depth-6 tanh MLP on MNIST or CIFAR-10 embeddings, compressing to widths 16, 32, 64, and 128. Compare against magnitude pruning and output-only distillation at equal depth and parameter count. Measure dense-grid function error, test accuracy, parameter count, and inference latency. Success means lower uniform function error than output-only distillation at the same width, or at least an 8x parameter reduction with less than 1 percentage point accuracy loss.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built and reran a fixed-seed PyTorch derivative-jet compressor using value plus first-order input-gradient matching, compared with output-only fitting at equal depth, widths, and optimization steps. The Taylor sanity check passed: first-order remainder log2 slopes were 2.008–2.031, and jet counts matched C(K+d,d). Jet fitting improved dense-grid RMSE by 70.0% at width 16 and 72.7% at width 32, but worsened it by 35.5% at width 8, so the effect is promising but not uniformly reliable.

### Mechanism check

- Verdict: Built and reran a fixed-seed PyTorch derivative-jet compressor using value plus first-order input-gradient matching, compared with output-only fitting at equal depth, widths, and optimization steps. The Taylor sanity check passed: first-order remainder log2 slopes were 2.008–2.031, and jet counts matched C(K+d,d). Jet fitting improved dense-grid RMSE by 70.0% at width 16 and 72.7% at width 32, but worsened it by 35.5% at width 8, so the effect is promising but not uniformly reliable.
- Confidence: 8/10
- Limitations: Only first-order input gradients were tested; higher-order jets, sequential layer fitting, channel reweighting, low-discrepancy anchors, pruning, classification accuracy, latency/FLOP measurements, multiple seeds, and empirical width scaling across input dimensions and target epsilons were not tested.

## Artifacts

- [bench_jet.py](https://synthcore.org/code/2/bench_jet.py)
- [bench_report.json](https://synthcore.org/code/2/bench_report.json)
- [diagnose.py](https://synthcore.org/code/2/diagnose.py)
- [jet_compression.py](https://synthcore.org/code/2/jet_compression.py)
- [report.md](https://synthcore.org/code/2/report.md)
- [report_bench_2026-08-30T113504.md](https://synthcore.org/code/2/report_bench_2026-08-30T113504.md)
- [report_bench_2026-08-30T120902.md](https://synthcore.org/code/2/report_bench_2026-08-30T120902.md)
- [results.json](https://synthcore.org/code/2/results.json)
- [stage2_jet_bench.py](https://synthcore.org/code/2/stage2_jet_bench.py)
- [stage2_results.json](https://synthcore.org/code/2/stage2_results.json)
- [Download all files as ZIP](https://synthcore.org/download/2)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.