✗ Failed on benchmark 2026

Permutation-Sensitivity Regularization

Implementation & benchmark of arXiv:2608.08377 — Price Information Is Not Enough: Ordering and Decision Rules in Storage Bidding

Usefulness7/10

Difficulty6/10

Novelty7/10

Regularization Stability Generalization Accuracy Regularization Optimizer Loss Training dynamics Optimization Statistics Control theory Linear algebra

Source paper: Price Information Is Not Enough: Ordering and Decision Rules in Storage Bidding arXiv:2608.08377 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Regularize a neural decision policy against economically harmful changes in its action when the predicted price ordering is perturbed. Targeted swaps of extrema and threshold-adjacent entries directly test the paper’s mechanism that a small number of ordering mistakes can cause a disproportionate revenue loss.

Formulas

$$(PσΠ̂)ₜ=π̂σ(t), x=g(Π̂,C), xσ=g(PσΠ̂,C)$$

$$Rperm(θ)=Eσ∼Q[qσ ||g(Pσfθ(C,S),C)−g(fθ(C,S),C)||₁]$$

$$Rrev(θ)=Eσ∼Q[max(0,⟨g(fθ,C),Π⟩−⟨g(Pσfθ,C),Π⟩)]$$

$$Ltotal=Lbase+βRperm or Ltotal=Lbase+βRrev$$

Mathematical statement

Let fθ(C,S) = Π̂ be a predicted price vector and g be a feasible scheduling rule. For a permutation σ of time indices, define (PσΠ̂)ₜ = π̂σ(t), x = g(Π̂,C), and xσ = g(PσΠ̂,C). Let Q be a distribution over targeted permutations, including minimum-maximum exchanges and swaps near the action threshold. A schedule-stability penalty is Rperm = Eσ∼Q[qσ ||xσ − x||₁], where qσ is the predicted price-gap magnitude affected by σ. If true prices are available during training, a direct counterfactual revenue penalty is Rrev = Eσ∼Q[max(0, ⟨x,Π⟩ − ⟨xσ,Π⟩)]. Add either penalty to the base objective. The scheduler must preserve physical feasibility, so the regularizer tests policy sensitivity rather than allowing invalid actions.

Implementation notes

Insert the regularizer after the network outputs its price vector and before the storage schedule is executed. Use a differentiable scheduler when possible; for an initial implementation, use a linear-program scheduler and either differentiate through its solution or apply the penalty to continuous charge/discharge scores before projection. At each minibatch, create three perturbation types: exchange the predicted minimum and maximum, swap adjacent ranks around the predicted charge/discharge cutoff, and add small noise followed by a sorting-based permutation. Compute the original schedule, each perturbed schedule, and the L1 action difference or true-revenue difference. Pseudocode: pred = net(context); x = scheduler(pred); perms = targeted_permutations(pred); penalty = 0; for perm in perms: xp = scheduler(pred[perm]); gap = abs(pred[perm[0]]-pred[perm[1]]); penalty += gap * abs(xp-x).sum(); loss = base_loss + beta * penalty; backprop(loss). The paper provides the key calibration signature: exchanging the cheapest and dearest entries of an otherwise exact vector reduced total revenue by 53 percent. First run the same swap test on the French dataset with an MLP forecaster and LP scheduler, comparing no regularization, random permutation augmentation, and targeted regularization. Plot revenue loss against Kendall rank distance and forecast noise. The prediction is that the regularized model has a flatter local revenue-sensitivity curve and a smaller worst-case drop, while its nominal perfect-forecast revenue decreases only slightly.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a differentiable budget-feasible softmax scheduler with extrema and cutoff-adjacent permutation penalties, plus baseline, random-permutation, and targeted-regularization MLP variants. The mechanism manifested: zero affected gap gave zero penalty, penalty increased monotonically with gap, q-scaling was exactly linear, and extrema swaps were consistently more damaging than adjacent swaps. Targeted training reduced extrema revenue drop from 4.716 to 2.710 versus baseline, but nominal revenue and forecast quality declined substantially, so robustness improved but there is not yet a clear practical win.

Agent confidence: 8/10

Baseline: Forecast MSE 0.0869; nominal revenue 2.5622; extrema-perturbed revenue -2.1538; extrema drop 4.7160; adjacent drop 0.0088.
Idea: Forecast MSE 0.7039; nominal revenue 1.9128; extrema-perturbed revenue -0.7969; extrema drop 2.7097; adjacent drop 0.1110. Math checks: zero-gap penalty 0.0000; gap penalties for [0.1,0.2,0.4,0.8,1.6] were [0.0199,0.0790,0.3040,1.0625,2.9493]; q scaling was exact for q=[0.25,0.5,1,2,4] with raw action L1=1.4434; extrema/adjacent penalty ratios were [26.0,29.2,44.1,153.6,3137.1]. Random augmentation had MSE 0.0891, nominal revenue 2.5605, and extrema drop 4.7038.

Limitations: This is a synthetic toy dataset, not the French dataset or a true LP/storage simulator. The scheduler is softmax allocation rather than a physically detailed differentiable LP, only extrema perturbations were used during targeted training, and results come from one seed without confidence intervals or hyperparameter tuning. The q-linearity check directly evaluates the stated q weighting rather than estimating q from a learned permutation distribution.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.08377) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Price_ordering
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 0.0996
Idea mean: 0.0996
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

The custom track is synthetic rather than the French dataset or a physical LP scheduler. Only extrema swaps were used during training; adjacent-threshold and noisy sorting perturbations were not included. The run used 400 training examples, 200 test examples, 20 epochs, and a softmax budget readout instead of a full constrained storage simulator.

How to run: python3 run_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Built a structurally matched custom multi-period price-vector regression track and trained identical mlp_tiny systems end-to-end, differing only by the targeted extrema permutation penalty. Across 8 paired seeds, the best idea setting was beta=0 and exactly matched the baseline; beta=0.01 and beta=0.05 worsened test MSE. The trained-model mechanism signature did not confirm reduced revenue sensitivity, so there is no real transfer win.

Baseline: Test MSE mean 0.0996346 across 8 seeds; best lr=0.01 and weight_decay=0.0001.
Idea: Best MSE mean 0.0996346 at beta=0.0; beta=0.01 mean 0.1005472; beta=0.05 mean 0.1286849. Paired delta=0.0 and permutation p=1.0.
Mechanism: Mean nominal revenue=4.18094, extrema-perturbed revenue=0.28705, revenue drop=3.89389; reduced sensitivity was not demonstrated. Math checks: zero-gap penalty=0 and q-linearity error=0.

Benchmark result