✗ Failed on benchmark 2026

Calorimetric Training Transition Detector

Implementation & benchmark of arXiv:2608.17903 — Heat capacity as a marker for shape and jamming transitions in active systems

Usefulness7/10

Difficulty5/10

Novelty7/10

Dynamics Stability Accuracy Generalization Optimizer Scheduler Training dynamics Regularization Statistical mechanics Stochastic processes Dynamical systems Control theory

Source paper: Heat capacity as a marker for shape and jamming transitions in active systems arXiv:2608.17903 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat optimizer stochasticity as an effective temperature and periodically apply a small temperature pulse, such as a temporary change in minibatch size, learning rate, dropout, or Langevin-noise amplitude. Measure the transient excess optimization dissipation and use its integrated response as a heat-capacity-like signal; sharp peaks provide a principled trigger for learning-rate changes, regularization changes, or phase-transition logging.

Formulas

$$\frac{k(x\to x')}{k(x'\to x)}=\exp\left[\beta\left(W_{x\to x'}-\Delta U\right)\right],\qquad \Delta U=U(x')-U(x),\quad \beta=1/T.$$

$$k(x\to x')=\nu\exp\left[\frac{\beta}{2}\left(W_{x\to x'}-\Delta U\right)\right].$$

$$Q_{\mathrm{ex}}(T,\delta T)=\int_{0}^{\tau}\left[q_{T+\delta T}(t)-q_{T,\mathrm{ss}}\right]dt,\qquad C_{\mathrm{train}}(T)=\lim_{\delta T\to0}\frac{Q_{\mathrm{ex}}(T,\delta T)}{\delta T}.$$

$$T_{\mathrm{eff}}\propto \eta\,\frac{1-b/B}{b},\qquad C_{\mathrm{train}}(T_{\mathrm{eff}})=\frac{\partial}{\partial T_{\mathrm{eff}}}\int_0^{\tau}\left[\langle q(t)\rangle_{T_{\mathrm{eff}}+\delta T}-\langle q\rangle_{\mathrm{ss},T_{\mathrm{eff}}}\right]dt,$$

Mathematical statement

The paper's microscopic dynamics obey local detailed balance, with forward/backward hopping-rate ratio $k(x\to x')/k(x'\to x)=\exp[\beta(W_{x\to x'}-\Delta U)]$, where $\beta=1/T$, $W$ is active work, and $\Delta U$ is the potential-energy change. Symmetric rates are $k(x\to x')=\nu\exp[\beta(W_{x\to x'}-\Delta U)/2]$. The transferable construction is an excess-heat response after a small temperature perturbation: if the stationary system at temperature $T$ is perturbed to $T+\delta T$, record transient heat release $q(t)$, subtract its stationary baseline $q_{\mathrm{ss}}$, and define $C_{\mathrm{neq}}(T)\approx Q_{\mathrm{ex}}/\delta T$, with $Q_{\mathrm{ex}}=\int_0^{\tau}[q(t)-q_{\mathrm{ss}}]dt$. For a neural optimizer, $T$ is an effective noise temperature, $q(t)$ is a dissipation proxy such as squared update magnitude or update work, and $C_{\mathrm{train}}$ is the response integral. The predicted signature is a local maximum of $C_{\mathrm{train}}(T)$ near a training-dynamics transition, such as the boundary between stable learning and exploding updates or the onset of gradient jamming.

Implementation notes

1. Integration point: wrap a minibatch optimizer, initially SGD on a two-layer MLP or small ResNet. Use $T_{\mathrm{eff}}=\eta(1-b/B)/b$ as a controllable stochasticity proxy, where $\eta$ is learning rate, $b$ is minibatch size, and $B$ is dataset size. Every $K$ steps, save the current weights, run a baseline window of $L$ steps at temperature $T$, then apply a pulse to $T+\delta T$ for $P$ steps and return to $T$. 2. Pseudocode: estimate $q_t=\|\Delta\theta_t\|^2/\eta$ or $q_t=-\langle g_t,\Delta\theta_t\rangle$, compute $q_{\mathrm{ss}}$ from the baseline window, calculate $Q_{\mathrm{ex}}=\sum_{t=1}^{P}(q_t-q_{\mathrm{ss}})\Delta t$, and set $C_{\mathrm{train}}=Q_{\mathrm{ex}}/\delta T$. Smooth this estimate over several probes. If it reaches a statistically significant local maximum, reduce learning rate, increase batch size, or switch regularization. 3. Taken from the paper: the excess-response construction and interpretation of a response peak as a nonequilibrium transition marker. Estimated empirically: the effective-temperature calibration, baseline dissipation, pulse duration $P$, relaxation time, and confidence intervals. 4. First cheap experiment: train an MLP on MNIST and CIFAR-10 with SGD while sweeping minibatch size and learning rate, comparing ordinary gradient monitoring with calorimetric monitoring. Plot $C_{\mathrm{train}}(T)$ and independently identify the largest stable learning rate and onset of gradient-norm explosion. The quantitative prediction is that the response peak or sharp shoulder lies within 20% of the empirical stability-transition temperature and is reproducible for pulse amplitudes $0.02T\leq\delta T\leq0.1T$. Also fit post-pulse relaxation to $q(t)-q_{\mathrm{ss}}\sim A\exp(-t/\tau_{\mathrm{rel}})$; near the response peak, the relaxation time should increase, indicating critical slowing down.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a quadratic noisy-SGD calorimetric detector with detailed-balance verification, excess-response calculation, pulse-size and stability sweeps, relaxation-time measurement, Monte Carlo validation, and an adaptive toy optimizer. The stability boundary appeared at eta·lambda≈2.0, relaxation time increased from 1.09 to 99.5 steps near the boundary, and C_train was linear for pulse amplitudes 0.01T–0.1T with relative error below 4e-12. The response was monotonic and divergent rather than a finite local peak, so the transition signal is supported in this toy system but the full peak claim is not.

Agent confidence: 7/10

Baseline: Noisy quadratic SGD: mean final 30-step loss 1.9724; 0/20 divergent runs.
Idea: Calorimetric-triggered SGD: mean final 30-step loss 0.6365; 0/20 divergent runs; mean 9 triggers/run. Last stable eta·lambda=1.99 and first unstable=2.01. Relaxation time increased from 1.09 to 99.5 near the boundary. Pulse linearity error was 3.7e-12. Monte Carlo C=163.32 versus exact C=158.00, a 3.37% relative error.

Limitations: Only a one-dimensional quadratic noisy-SGD toy was tested. No MLP, MNIST, CIFAR-10, minibatch-size implementation, GPU experiment, confidence intervals across independent probes, or genuine finite calorimetric peak was evaluated. The adaptive threshold is a toy fixed threshold based on the analytic response, not a statistically calibrated online detector.

How to run: python3 calorimetric_detector.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.17903) Sep 2, 2026

Ahead of the baseline (−1.7%) but not statistically significant (p=0.1473, wins 7 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.4311
Idea mean: 7.302
Effect (Δ): -0.1292 (−1.7%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.1473 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±2.5%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched registered tabular optimizer track was tested. The run used 400 training and 200 test samples, 18 epochs, batch size 64, and a three-value shared learning-rate grid; vision, sequence, dynamics, longer training, alternative pulse mechanisms, finite-size scaling, and calibrated online trigger thresholds were not tested.

How to run: python3 bench_calorimetric.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

On the registered built-in tabular track with mlp_tiny and 8 paired seeds, the calorimetric minibatch pulse reduced mean test MSE from 7.43115 to 7.30198, but the paired permutation test gave p=0.1473, so there is no significant win. The trained-model mechanism signature measured mean pulse-to-baseline dissipation ratio 0.80083 rather than the predicted value above 1, so the proposed response did not transfer quantitatively.

Baseline: Adam, tuned lr=0.006; sweep means: lr=0.001 -> 14.98159, lr=0.003 -> 9.22344, lr=0.006 -> 7.29187; full 8-seed mean MSE 7.4311487, std 0.590073
Comparison: delta_mean=-0.1291721, 7/8 paired wins, permutation p_value=0.1473, verdict=no significant win
Idea: Calorimetric minibatch-noise pulse, best lr=0.006; sweep means: lr=0.001 -> 14.74152, lr=0.003 -> 8.37822, lr=0.006 -> 7.30198; full 8-seed mean MSE 7.3019766, std 0.773452

Benchmark result