Derivative-Jet MLP Compression

Integrate this as an offline compressor for a trained wide MLP, not as a modification to the ordinary training forward pass. Let the teacher be h_0(x) = x and h_l(x) = phi(W_l h_(l-1)(x) + b_l), for layers l = 1,...,L, with analytic activation phi. Construct a student with the same depth L and target hidden width m. Set d_in from the actual input representation, choose derivative order K, and initialize m = c times binomial(K + d_in, d_in), testing several constants c. Generate A anchor points in the bounded deployment domain using a low-discrepancy sequence, a grid for low-dimensional inputs, or a calibration set. Use forward-mode automatic differentiation to compute teacher layer values and derivatives partial^alpha h_l at every anchor; begin with K = 1 because higher-order jets become expensive. Initialize the student by random projection, neuron clustering, or ordinary weight matching. Fit layers sequentially: freeze earlier student layers, compute student jets, and minimize the displayed L_jet with Adam. Use w_alpha = rho^|alpha| divided by alpha factorial so high-order derivatives do not dominate. After each layer, normalize each student channel by its activation RMS s_j, divide its incoming parameters by s_j + delta, and multiply the matching outgoing weight column by s_j + delta. This preserves the layer's scale while improving conditioning. The mathematics supplies the jet objective and width scaling; anchor density, constants, derivative order, and reweighting strength are empirical. First test a width-1024, depth-6 tanh MLP on MNIST or CIFAR-10 embeddings, compressing to widths 16, 32, 64, and 128. Compare against magnitude pruning and output-only distillation at equal depth and parameter count. Measure dense-grid function error, test accuracy, parameter count, and inference latency. Success means lower uniform function error than output-only distillation at the same width, or at least an 8x parameter reduction with less than 1 percentage point accuracy loss.