# Effective-Exploration Bias Correction

- ID: 1653
- Canonical URL: https://synthcore.org/idea/1653/effective-exploration-bias-correction
- API JSON: https://synthcore.org/api/idea/1653.json
- API Markdown: https://synthcore.org/api/idea/1653.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.01069](https://arxiv.org/abs/2608.01069)
- Category: training
- Solves: accuracy, sample-efficiency, generalization
- ML areas: rl, loss, training
- Math tags: statistics, probability, dynamical-systems
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Correct arm-conditioned targets in a neural contextual-bandit model using the exploration coefficient of the data-collection index. For a generalized UCB policy with index I_t(x,n)=x+f_t/sqrt(n), add approximately sigma_hat_a/f_T to the observed mean for arms that are plausibly non-unique-optimal, counteracting the negative post-bandit bias.

## Mathematical statement

The paper's leading-order result is Gamma_{a,T} E[Z_{a,T}] -> -(1 - (1/|O|) 1{a in O}) sigma_a, where Z_{a,T} is the arm-a standardized post-bandit statistic, O is the set of optimal arms, sigma_a is the reward standard deviation, and Gamma_{a,T} is the effective exploration rate determined by the index algorithm. Thus, for a non-optimal or non-uniquely-optimal arm, E[Z_{a,T}] is approximately -sigma_a/Gamma_{a,T}. In the generalized UCB family I_t(x,n)=x+f_t/sqrt(n), the extracted result states E[Z_{a,T}] is asymptotically proportional to -1/f_T. Define N_a(T) as the number of pulls of arm a, bar_r_a=(1/N_a(T)) sum_{i:A_i=a} r_i, sigma_hat_a as its empirical reward standard deviation, and Z_a=sqrt(N_a(T))(bar_r_a-mu_a)/sigma_hat_a. The adapted debiased target is mu_tilde_a=bar_r_a+sigma_hat_a/(sqrt(N_a(T)) f_T) for arms classified as non-unique-optimal; equivalently, its standardized correction is Z_tilde_a=Z_a+1/f_T.

## Key formulas

- $$\Gamma_{a,T}\mathbb{E}[Z_{a,T}]\longrightarrow-\left(1-\frac{1}{|\mathcal{O}|}\mathbbm{1}\{a\in\mathcal{O}\}\right)\sigma_{a}$$
- $$I_t(x,n)=x+\frac{f_t}{\sqrt n},\qquad \mathbb{E}[Z_{a,T}]\asymp-\frac{1}{f_T}$$
- $$\bar r_a=\frac{1}{N_a(T)}\sum_{i:A_i=a}r_i,\qquad Z_a=\frac{\sqrt{N_a(T)}(\bar r_a-\mu_a)}{\hat\sigma_a}$$
- $$\tilde\mu_a=\bar r_a+g_a\frac{\hat\sigma_a}{\sqrt{N_a(T)}f_T},\qquad \tilde Z_a=Z_a+\frac{g_a}{f_T}$$

## Implementation notes

Integrate this at the target-construction stage of a contextual-bandit reward model, not inside the neural encoder. Let a network q_theta(x,a) predict reward for context x and arm a, while the online policy selects arms with I_t=q_theta(x_t,a)+f_t/sqrt(N_a). Maintain per-arm or per-cluster count N_a, mean bar_r_a, and variance sigma_hat_a squared. At the end of each training window, compute c_a=sigma_hat_a/(sqrt(N_a) f_T). Estimate whether arm a is uniquely optimal by comparing its upper confidence estimate with the best competitor: u_a=bar_r_a+sigma_hat_a/sqrt(N_a), and u_minus_a=max_{b != a}(bar_r_b+sigma_hat_b/sqrt(N_b)). Set a soft gate g_a=sigmoid((u_minus_a-u_a)/tau), where tau is a temperature; g_a near one means likely non-unique-optimal and g_a near zero means likely uniquely optimal. Train with corrected regression targets y_i_corr=r_i+g_{A_i} c_{A_i}, or add this correction only to the arm-specific residual head while leaving the shared context representation unchanged. Pseudocode: collect (x_t,a_t,r_t); update N_a, bar_r_a, and sigma_hat_a; compute f_T, g_a, and c_a; replay minibatches with y_i_corr; periodically refresh statistics. The theorem supplies the sign and 1/f_T scaling, while sigma_hat_a, the optimality gates, and the effective f_T for nonstationary schedules are estimated empirically. First test on a synthetic contextual bandit with 10 arms, Gaussian rewards, a two-layer MLP, and UCB1 with f_t=sqrt(2 log t). Compare ordinary MSE with the correction using held-out uniformly randomized arm data. Success means lower held-out reward MSE and reduced signed calibration error for rarely selected arms at equal interaction budget, without degrading predictions for the uniquely best arm.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a reproducible UCB Gaussian-bandit simulation and a small contextual neural reward-model comparison in experiment.py. The suboptimal-arm bias was negative and adding 1/f_T moved the standardized statistic from -0.416 to -0.044, supporting the correction sign and local effect. However, f_T times the measured bias ranged from -0.38 to -2.50 in the exploration sweep, and the neural correction worsened held-out MSE (0.527 to 0.581) and rare-arm signed bias (0.365 to 0.406); therefore the idea did not produce a validated overall win.

### Mechanism check

- Verdict: Built a reproducible UCB Gaussian-bandit simulation and a small contextual neural reward-model comparison in experiment.py. The suboptimal-arm bias was negative and adding 1/f_T moved the standardized statistic from -0.416 to -0.044, supporting the correction sign and local effect. However, f_T times the measured bias ranged from -0.38 to -2.50 in the exploration sweep, and the neural correction worsened held-out MSE (0.527 to 0.581) and rare-arm signed bias (0.365 to 0.406); therefore the idea did not produce a validated overall win.
- Confidence: 8/10
- Limitations: Reduced Monte Carlo repetitions were used after the initial run exceeded the runtime budget. Only stationary synthetic Gaussian rewards were tested; the contextual policy used an oracle nonlinear mean for arm selection rather than a fully learned online neural UCB policy. The soft optimality gate was not separately calibrated, and no repeated neural seeds or FLOP-normalized study was performed.

## Artifacts

- [REPORT.json](https://synthcore.org/code/614/REPORT.json)
- [experiment.py](https://synthcore.org/code/614/experiment.py)
- [report.md](https://synthcore.org/code/614/report.md)
- [results.json](https://synthcore.org/code/614/results.json)
- [Download all files as ZIP](https://synthcore.org/download/614)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.