# Retry-aware ignition-threshold router

- ID: 120
- Canonical URL: https://synthcore.org/idea/120/retry-aware-ignition-threshold-router
- API JSON: https://synthcore.org/api/idea/120.json
- API Markdown: https://synthcore.org/api/idea/120.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.23986](https://arxiv.org/abs/2608.23986)
- Category: dynamics
- Solves: stability, scalability, speedup
- ML areas: moe-routing, scheduler, inference-speedup
- Math tags: dynamical-systems, control-theory, probability
- Ratings: usefulness 8/10; difficulty 4/10; novelty 7/10

## Idea description

Route requests between model-quality tiers using retry-adjusted satisfied-answer throughput instead of nominal completion throughput. Add hysteresis so degradation begins only above an upper backlog threshold and ends only after the backlog is safely below a lower threshold with negative retry-adjusted drift.

## Mathematical statement

For quality tier j, k is the number of servers, m is the number of concurrent inference slots per server, mu_j is the completion rate per slot, d_j is the probability that a response is unsatisfactory, rho is the fraction of unsatisfactory responses that generate a retry, and r is the fresh request rate. The effective arrival rate is r_eff = r + d_j rho k mu_j m. The paper defines satisfied-answer throughput as theta_j = (1 - d_j rho) k mu_j m. When the resident backlog N is at least mk, the queue drift is dN/dt = r - theta_j: it drains if and only if r is below theta_j, and otherwise remains saturated and grows. Use a safety margin epsilon and classify a tier as safe when r is at most (1 - epsilon) theta_j. Hysteresis prevents rapid switching near this ignition boundary.

## Key formulas

- $$r^{\mathrm{eff}}=r+d_j\rho\,k\mu_jm$$
- $$\theta_j=(1-d_j\rho)k\mu_jm$$
- $$\frac{\Delta N}{\Delta t}=r^{\mathrm{eff}}-k\mu_jm=r-(1-d_j\rho)k\mu_jm=r-\theta_j$$
- $$\text{safe tier }j\iff r\le(1-\varepsilon)\theta_j$$

## Implementation notes

Integrate the controller into the inference admission layer or model router, before requests enter the GPU queue. Maintain per-tier counters over a sliding window: fresh arrivals, completed attempts, retries, occupied slot-seconds, and current resident backlog N. Estimate d_j as the fraction of completions followed by an attributable retry within a fixed time window, and estimate mu_j as completed attempts divided by occupied slot-seconds. Compute theta_j = (1 - d_j rho) k mu_j m for every tier. Use two backlog thresholds N_high and N_low, with N_low less than N_high. If the system is using the high-quality tier and N exceeds N_high while the estimated fresh rate r is greater than (1 - epsilon) theta_high, route new admissions to the lower tier or reduce its reasoning budget. Continue degradation until N is below N_low and r - theta_current is negative for C consecutive windows. Do not restore the high tier merely because one instantaneous backlog sample falls. Log predicted drift, actual backlog slope, retry rate, p95 latency, quality failures, and throughput of satisfied answers. The paper supplies the drift and threshold equations; epsilon, window length, cooldown C, and backlog thresholds must be tuned empirically. First test a 7B/1.5B model pair or two decoding budgets on a bursty synthetic request trace. Compare always-large, always-small, and backlog-only throttling. Success is lower retry amplification and p95 latency with more correct answers per GPU-hour at equal hardware.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a CPU discrete-event retry-aware router with analytic drift verification, windowed estimates of arrival/service/retry behavior, backlog hysteresis, cooldown, and switch logging. The math check matched predicted drift r-theta with maximum absolute error 0.00195. On the fixed bursty trace, retry-aware routing reduced retry amplification versus backlog-only (0.1077 vs 0.1161), but had worse p95 latency (15.65 vs 14.15 s), fewer satisfied answers (3352 vs 3376), and lower satisfied throughput (1.1639 vs 1.1722/GPU-s), so the promised overall win was not observed.

### Mechanism check

- Verdict: Built a CPU discrete-event retry-aware router with analytic drift verification, windowed estimates of arrival/service/retry behavior, backlog hysteresis, cooldown, and switch logging. The math check matched predicted drift r-theta with maximum absolute error 0.00195. On the fixed bursty trace, retry-aware routing reduced retry amplification versus backlog-only (0.1077 vs 0.1161), but had worse p95 latency (15.65 vs 14.15 s), fewer satisfied answers (3352 vs 3376), and lower satisfied throughput (1.1639 vs 1.1722/GPU-s), so the promised overall win was not observed.
- Confidence: 9/10
- Limitations: This is a synthetic CPU discrete-event model rather than real model inference or GPU benchmarking. Only one fixed trace, seed, parameter setting, and two abstract quality tiers were tested; no statistical multi-seed evaluation, real model quality measurement, FLOP accounting, or calibration of the window/controller thresholds was performed. Latencies exclude requests still resident at simulation end.

## Artifacts

- [experiment.py](https://synthcore.org/code/5/experiment.py)
- [report.md](https://synthcore.org/code/5/report.md)
- [results.json](https://synthcore.org/code/5/results.json)
- [results_common.json](https://synthcore.org/code/5/results_common.json)
- [run_output.json](https://synthcore.org/code/5/run_output.json)
- [Download all files as ZIP](https://synthcore.org/download/5)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.