# Order-Sensitivity Margin Regularizer

- ID: 2412
- Canonical URL: https://synthcore.org/idea/2412/order-sensitivity-margin-regularizer
- API JSON: https://synthcore.org/api/idea/2412.json
- API Markdown: https://synthcore.org/api/idea/2412.md
- Verification status: unverified
- Source: [arXiv:2608.16701](https://arxiv.org/abs/2608.16701)
- Category: regularization
- Solves: stability, generalization
- ML areas: rnn, regularization, training-dynamics
- Math tags: dynamical-systems, optimization, computation
- Ratings: usefulness 6/10; difficulty 4/10; novelty 8/10

## Idea description

Train a threshold-reset recurrent network to suppress dependence on unresolved excitatory/inhibitory arrival order. Penalize states that fall in the paper's order-sensitive firing interval, or augment training with excitatory-first and inhibitory-first counterfactuals and enforce consistent outputs.

## Mathematical statement

For pre-threshold state x<theta, excitatory mass a>=0, and inhibitory magnitude b>=0, the paper proves that the two processing orders disagree precisely when x+a-b<theta<=x+a. Define d_net=theta-(x+a-b) and d_E=x+a-theta. Their minimum m=min{d_net,d_E} is positive exactly in the order-sensitive region and measures the distance to the relevant threshold boundaries. Use R_ord=softplus(kappa*m) indicator(m>0), with kappa>0 controlling sharpness, to penalize ambiguous states. Alternatively define y_E and y_I as network outputs under E-first and I-first processing and add a consistency penalty R_inv=||y_E-y_I||_2^2. The first regularizer pushes the operating state away from order-sensitive thresholds; the second trains a coarse simulator whose output is invariant to unresolved within-bin order.

## Key formulas

- $$x+a-b<\theta\le x+a$$
- $$d_{\mathrm{net}}=\theta-(x+a-b),\qquad d_{\mathrm{E}}=x+a-\theta$$
- $$m=\min\{d_{\mathrm{net}},d_{\mathrm{E}}\},\qquad R_{\mathrm{ord}}=\operatorname{softplus}(\kappa m)\,\mathbf{1}_{\{m>0\}}$$
- $$\mathcal{L}=\mathcal{L}_{\mathrm{task}}+\lambda R_{\mathrm{ord}}+\mu\left\|y_{\mathrm{E}}-y_{\mathrm{I}}\right\|_2^2$$

## Implementation notes

Add the regularizer at each recurrent layer and timestep immediately before threshold/reset. During the forward pass, separate incoming pulses into positive and negative masses over the current coarse timestep: a=sum_j max(Delta_j,0), b=sum_j max(-Delta_j,0), and let x be the membrane potential after leak but before these pulses. Compute d_net, d_E, and m using the displayed equations. For the robust-simulation variant, construct two counterfactual states from the same events: E-first applies all positive pulses, thresholds and resets, then applies negative pulses; I-first applies all negative pulses, then positive pulses. Run the remaining recurrent computation for both branches to obtain y_E and y_I, then add mu times their squared difference to the task loss. For the margin variant, use a straight-through or detached indicator for m>0 and clip m to a fixed range. Start with lambda and mu between 0.001 and 0.1 times the task loss, then sweep them. First test on a synthetic temporal classification task whose labels are invariant to within-bin event order, followed by SHD or N-MNIST with deliberately coarse bins. Compare baseline BPTT, margin-only training, and counterfactual consistency training at equal compute; compute counterfactuals on a random 25 percent of timesteps if needed. Success is lower E-first/I-first output disagreement and equal or better test accuracy under random delay perturbations, plus fewer threshold oscillations and lower seed variance. For order-sensitive tasks, omit the consistency term and test margin-only training.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.