# Cycle-aware softmax temperature control

- ID: 1101
- Canonical URL: https://synthcore.org/idea/1101/cycle-aware-softmax-temperature-control
- API JSON: https://synthcore.org/api/idea/1101.json
- API Markdown: https://synthcore.org/api/idea/1101.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.18128](https://arxiv.org/abs/2607.18128)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: attention, moe-routing, transformer, training-dynamics
- Math tags: dynamical-systems, spectral-theory, linear-algebra
- Ratings: usefulness 8/10; difficulty 6/10; novelty 6/10

## Idea description

Use an online estimate of the positive feedback gain among logits, routing probabilities, and representations to adjust the softmax temperature. Increase temperature when the estimated cyclic gain approaches the instability regime, preventing exponential amplification and router collapse without globally weakening all layers.

## Mathematical statement

The paper gives the tied-policy response $$\frac{v}{\tau}\exp\left\{\left(-\nu I+\frac{v}{\tau}K\right)(T-t)\right\},$$ where $v>0$ is a response scale, $    au>0$ is temperature, $
u>0$ is damping, $I$ is the identity matrix, $K\geq0$ is a nonnegative feedback matrix, and $T-t$ is the remaining horizon. For a positive cycle, the dominant Perron eigenvalue $\rho(K)$ contributes exponential amplification through the matrix exponential; stability is expected when the dominant gain $v\rho(K)/    au-
u$ is nonpositive. In contrast, a directed acyclic graph with longest path $L$ has susceptibility of order $\Theta(    au^{-(L+1)})$, because powers of a nilpotent $K$ terminate. For neural logits $z$, probabilities are $p_i=\exp(z_i/    au)/\sum_j\exp(z_j/    au)$. Define $K$ as a nonnegative estimate of the absolute cross-channel Jacobian from current probabilities through the network and back to logits. The transferable control variable is $g=\rho(K)/    au$.

## Key formulas

- $$\frac{v}{\tau}\exp\left\{\left(-\nu I+\frac{v}{\tau}K\right)(T-t)\right\}$$
- $$\text{cycle gain rate}\;\approx\;\frac{v\rho(K)}{\tau}-\nu$$
- $$p_i=\frac{\exp(z_i/\tau)}{\sum_j\exp(z_j/\tau)},\qquad \tau_{\mathrm{target}}=\max\left(\tau_{\min},\frac{v\rho(K)}{\nu+g_{\mathrm{margin}}}\right)$$
- $$\rho(K)\approx\frac{\|Kq\|_1}{\|q\|_1},\qquad q\leftarrow\frac{Kq}{\|Kq\|_1},\quad q\geq0$$

## Implementation notes

Integrate this into the logits-to-probabilities operation of a recurrent attention block, iterative token mixer, or MoE router. For each monitored layer, define logits $z\in\mathbb{R}^m$, probabilities $p=\operatorname{softmax}(z/\tau)$, and a small nonnegative feedback proxy $K\in\mathbb{R}_+^{m\times m}$. A practical construction is to retain the previous-step activation $h_{t-1}$, compute the absolute Jacobian-vector effect from probability perturbations to the next logits, and approximate it with $K_{ij}=|\partial z_{t+1,i}/\partial p_{t,j}|$; estimate this using one-sided finite differences on a detached minibatch or one reverse-mode Jacobian-vector product per column block. Every 20-100 optimizer steps, run 3-5 power iterations: initialize positive $q$, repeatedly set $q\leftarrow Kq/(\|Kq\|_1+\epsilon)$, and estimate $\rho(K)=\|Kq\|_1/\|q\|_1$. Set $\tau$ with an EMA controller using $\tau_{target}=\max(\tau_{min},v\rho(K)/(\nu+g_{margin}))$, then update slowly via $\tau\leftarrow(1-\alpha)\tau+\alpha\tau_{target}$, stopping gradients through the controller. Here $v$ is a calibrated response scale, $\nu$ is an explicit damping proxy estimated from residual decay, and $g_{margin}>0$ prevents operation at the boundary. The matrix-exponential stability principle and spectral estimate come from the paper; $K$, $v$, and $\nu$ are empirical engineering proxies. First test on a 6-layer recurrent attention toy transformer and a Switch-style MoE on WikiText-2 or C4, comparing fixed temperatures and standard annealing at equal FLOPs. Track training loss variance, maximum router probability, expert-load entropy, estimated $\rho(K)/\tau$, gradient norm spikes, and validation perplexity. Success means fewer collapse events and lower loss at equal final temperature, with the estimated gain remaining below threshold.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a reproducible cycle-aware temperature-control MVP with power-iteration spectral estimation, stability sweeps, cycle-versus-DAG response tests, and a tiny router comparison. The mechanism predictions manifested: the neutral-temperature boundary was localized to the predicted interval, spectral radius scaled linearly with cycle gain, and cycle response log-growth matched the predicted rate exactly. The adaptive controller reduced routing concentration substantially, but the crude proxy drove temperature to 63.6 and worsened task loss, so this is not yet an end-to-end performance win.

### Mechanism check

- Verdict: Built a reproducible cycle-aware temperature-control MVP with power-iteration spectral estimation, stability sweeps, cycle-versus-DAG response tests, and a tiny router comparison. The mechanism predictions manifested: the neutral-temperature boundary was localized to the predicted interval, spectral radius scaled linearly with cycle gain, and cycle response log-growth matched the predicted rate exactly. The adaptive controller reduced routing concentration substantially, but the crude proxy drove temperature to 63.6 and worsened task loss, so this is not yet an end-to-end performance win.
- Confidence: 9/10
- Limitations: Only a tiny synthetic router was tested, not WikiText-2/C4, a transformer, or a production MoE. The feedback matrix is a crude detached W_abs W_abs^T proxy rather than a measured probability-to-next-logit Jacobian; no temperature cap, calibration, multiple seeds, validation perplexity, or equal-FLOP large-scale comparison was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.98385
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/213/bench_report.json)
- [cycle_temperature_bench.py](https://synthcore.org/code/213/cycle_temperature_bench.py)
- [cycle_temperature_experiment.py](https://synthcore.org/code/213/cycle_temperature_experiment.py)
- [report.md](https://synthcore.org/code/213/report.md)
- [report_bench_2026-08-30T205257.md](https://synthcore.org/code/213/report_bench_2026-08-30T205257.md)
- [results.json](https://synthcore.org/code/213/results.json)
- [Download all files as ZIP](https://synthcore.org/download/213)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.