{
 "artifacts": null,
 "category": "regularization",
 "description": "Train a wide tanh network with a regularizer that constrains input Jacobians and sampled square-free mixed derivatives according to the theorem's depth scaling. First-order derivatives receive a depth-independent target, while an order-k mixed derivative is allowed to scale like k! L^(k-1), avoiding the exponentially conservative penalties implied by multiplying layer operator norms.",
 "formulas_latex": [
  "$\\operatorname{Lip}_{2}(f;D):=\\sup_{\\substack{x,y\\in D\\\\x\\neq y}}\\frac{|f(x)-f(y)|}{\\|x-y\\|_{2}},\\qquad \\operatorname{Lip}_{2}(f;D)\\leq\\sup_{x\\in D}\\|\\nabla f(x)\\|_{2}$",
  "$$n\\geq C\\left(L^{3}n_{0}^{2}(1+\\log n_{0})+L^{2}\\left(1+\\log(L/\\eta)\\right)\\right)$$",
  "$$\\left|D^{u}\\mathcal{R}_{\\Phi^{(L)}}(x)\\right|\\leq C_{0}|u|!(C_{1}L)^{|u|-1}\\prod_{j\\in u}\\beta_{j}(\\eta,n_{0})$$",
  "$$\\mathcal{L}_{\\mathrm{deriv}}=\\frac{1}{B}\\sum_{b=1}^{B}\\left[\\frac{\\|\\nabla_x f_\\theta(x_b)\\|_2^2}{\\tau_1^2}+\\lambda\\sum_{k\\in\\mathcal{K}}\\frac{1}{m_k}\\sum_{u\\in\\mathcal{U}_k}\\frac{|D^{u}f_\\theta(x_b)|^2}{\\left(k!(L+\\varepsilon)^{k-1}\\tau_k\\right)^2}\\right]$$"
 ],
 "id": 203,
 "implementation": "(1) Integration point: add the displayed derivative loss to the training objective of a wide scalar-output tanh MLP, evaluated with respect to the input tensor x. Use Xavier Gaussian initialization for every affine layer. For vector outputs, either sum over output coordinates or apply the penalty to randomly selected coordinates. Keep the task loss unchanged and use the derivative penalty only during training.\n\n(2) Pseudocode:\n```\nfor x, y in minibatch:\n    pred = f_theta(x)\n    loss = task_loss(pred, y)\n    g = grad_x(f_theta(x), x)                 # first order\n    reg = mean(norm(g, dim=-1)**2 / tau1**2)\n    for k in K:                               # e.g. K={2}\n        for u in random_coordinate_subsets(k):\n            d_u = mixed_derivative(f_theta(x), x, u)\n            target = tau_k * factorial(k) * (L + eps)**(k-1)\n            reg += mean(d_u**2 / target**2) / number_of_subsets\n    (loss + lambda * reg).backward()\n    optimizer.step()\n```\nUse forward-mode JVPs for sampled mixed derivatives when input dimension is moderate; use nested reverse-mode autodiff otherwise. The formula's factor k!(L+eps)^(k-1) is the adapted version of the theorem's C_0|u|!(C_1L)^(|u|-1) bound.\n\n(3) Computed from the mathematics: the width scaling L^3 n_0^2 and the factorial/polynomial depth dependence. Estimated empirically: tau_1 and tau_k, because the extracted theorem does not provide numerical C_0, C_1, or beta_j. Initialize tau values from the 90th percentile of derivative magnitudes on a few untrained minibatches, then optionally update them with an exponential moving average. Estimate the Lipschitz proxy by the maximum observed ||grad_x f|| on validation samples.\n\n(4) First experiment: use a depth-8, width-512 tanh MLP on a 20-dimensional smooth synthetic regression problem with a known target. Compare Xavier initialization with no penalty against the same network with k=1 and k=2 penalties. Repeat for depths 4, 8, and 16. Measure validation MSE, maximum sampled Jacobian norm, sampled mixed-derivative magnitudes, robustness to projected-gradient input perturbations, and wall-clock training cost. Success is derivative growth that remains approximately depth-independent for k=1 and polynomial rather than exponential for k=2, with improved perturbation robustness and less than 20% training overhead.",
 "math_summary": "The paper defines the Euclidean Lipschitz constant as Lip_2(f;D)=sup_{x != y}|f(x)-f(y)|/||x-y||_2 and uses Lip_2(f;D)\u003c=sup_{x in D}||grad f(x)||_2 for convex D and continuously differentiable f. For a scalar-output tanh network R_{Phi^(L)} with input dimension n_0, depth L, Xavier Gaussian weights, and hidden width n satisfying n \u003e= C[L^3 n_0^2(1+log n_0)+L^2(1+log(L/eta))], the paper gives, with probability at least 1-eta, |D^u R_{Phi^(L)}(x)| \u003c= C_0 |u|! (C_1 L)^(|u|-1) product_{j in u} beta_j(eta,n_0), simultaneously for x in [0,1]^(n_0) and every nonempty coordinate subset u of [n_0]. Here D^u is the mixed derivative over distinct coordinates in u, |u| is its order, eta is the failure probability, beta_j is the coordinate-dependent confidence factor, and C,C_0,C_1 are constants. Since the supplied extraction does not specify beta_j or numerical values for C_0,C_1, the implementation treats their product as tunable scales tau_k and preserves the theorem's factorial and polynomial-in-L dependence.",
 "math_tags": [
  "probability",
  "approximation-theory",
  "dynamical-systems"
 ],
 "ml_areas": [
  "mlp",
  "regularization",
  "initialization",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.26526",
  "arxiv_url": "https://arxiv.org/abs/2608.26526",
  "summary_what_math_gives_to_ml": "The paper replaces worst-case layer-norm multiplication with high-probability derivative control for wide random tanh networks. Under Xavier Gaussian initialization and an explicit width condition, first-order input sensitivity is independent of depth, while square-free mixed derivatives grow polynomially in depth rather than exponentially. The most direct neural-network transfer is a derivative regularizer whose order-dependent targets follow this scaling, combined with width-aware initialization and monitoring of Jacobian growth. This is particularly suitable for smooth surrogate models, robustness-sensitive regressors, and neural operators where input regularity matters.",
  "title": "High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercube",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "generalization",
  "accuracy"
 ],
 "title": "Polynomial-depth derivative regularization",
 "url": "https://synthcore.org/idea/203/polynomial-depth-derivative-regularization",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}