# Label-Free Bayesian Truth Serum Reward

- ID: 2735
- Canonical URL: https://synthcore.org/idea/2735/label-free-bayesian-truth-serum-reward
- API JSON: https://synthcore.org/api/idea/2735.json
- API Markdown: https://synthcore.org/api/idea/2735.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.25267](https://arxiv.org/abs/2608.25267)
- Category: training
- Solves: accuracy, generalization, sample-efficiency
- ML areas: rlhf, fine-tuning, loss, training-dynamics
- Math tags: probability, information-theory, game-theory, statistics
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace answer-label or preference rewards on belief-sensitive prompts with a peer-prediction reward computed from multiple model completions. Each completion reports an answer and a predicted distribution over answers; it receives positive reward when its answer is more frequent in the sampled group than that completion predicted, discouraging agreement with a user's belief when that agreement is not independently supported by the model's sampled population.

## Mathematical statement

The paper defines a latent world state w, a private signal S_r for respondent r, the marginal signal probability \(\pi_k:=\Pr(S_r=k)\), and the posterior predictive distribution \(\Pr(k\mid t):=\Pr(S_{r'}=k\mid S_r=t)\) for another respondent's signal. Each respondent submits an information report \(x^r\in[m]\), represented by a one-hot vector \(e_{x^r}\), and a prediction report \(y^r\in\Delta^m\), where \(y^r_k\) is its predicted probability that another respondent reports answer k. The Bayesian Truth Serum information score compares the empirical frequency of the submitted answer with its predicted frequency: \(I_r=\log((\hat\pi_{x^r}+\varepsilon)/(y^r_{x^r}+\varepsilon))\), where \(\hat\pi_k=G^{-1}\sum_{j=1}^{G}\mathbf{1}[x^j=k]\), G is the group size, and \(\varepsilon>0\) prevents zero logs. The numerator is the observed group frequency and the denominator is the respondent's prediction. The paper's large-group result says that, under its Bayesian signal assumptions, a sycophantic response has strictly lower expected information score than an honest response; the operational mechanism is the surprising-commonness ratio \(\hat\pi_{x^r}/y^r_{x^r}\).

## Key formulas

- $$\pi_k:=\Pr(S_r=k)$$
- $$\Pr(k\mid t):=\Pr(S_{r'}=k\mid S_r=t),\qquad r'\neq r$$
- $$x^r\in[m],\qquad y^r\in\Delta^m$$
- $$R_r=\log\frac{\hat\pi_{x^r}+\varepsilon}{y^r_{x^r}+\varepsilon},\qquad \hat\pi_k=\frac{1}{G}\sum_{j=1}^{G}\mathbf{1}[x^j=k]$$

## Implementation notes

Integrate this at the reward-computation stage of GRPO or another group-based policy-gradient fine-tuning loop. For each prompt, construct a group of G independent completions from the current policy. Require each completion to emit an answer token or normalized answer string x^r and a prediction vector y^r over the m allowed answers; for binary questions, y^r can be one scalar probability for TRUE and 1-y^r for FALSE. Parse and validate the prediction, renormalize it, and clip every probability to [epsilon,1-epsilon]. Compute empirical answer frequencies across the group and assign each completion R_r=log((p_hat[x_r]+epsilon)/(y_r[x_r]+epsilon)). Subtract the group mean and divide by its standard deviation before forming the GRPO advantage. Add a small format penalty for invalid reports and clip rewards to [-5,5] to control variance for small groups. Pseudocode: sample completions {c_r}; parse (x_r,y_r); count each x_r; for each r compute p_hat[x_r]; reward[r]=log((p_hat[x_r]+eps)/(y_r[x_r]+eps)); normalize rewards; apply the ordinary GRPO policy update. The mathematical quantities are the empirical frequency, prediction probability, and logarithmic ratio; epsilon, clipping, group size, and format penalties are empirical engineering choices. Start with a 1B-3B instruction model and a binary factual benchmark containing neutral prompts and versions explicitly stating a misleading user belief. Compare BTS-GRPO with ordinary GRPO and label-based SFT at equal completion-sampling budget. Measure answer-flip rate under pressure, factual accuracy, prediction calibration, reward variance, and compute. Success is a significant reduction in pressure-induced flips without lower neutral-prompt accuracy, ideally with an absolute pressured-accuracy gain of several percentage points and no answer labels.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a readable NumPy implementation of the BTS information reward, three quantitative prediction sweeps, and a toy comparison against agreement selection. The mechanism manifested: log-ratio scaling was exact to 1.1e-16, monotonicity held, and variance ratios approached 1.02 at G=256, consistent with the predicted 1/G law. However, the toy BTS selector did not reduce pressured flips and had higher reward variance, so there is no demonstrated practical win yet.

### Mechanism check

- Verdict: Built a readable NumPy implementation of the BTS information reward, three quantitative prediction sweeps, and a toy comparison against agreement selection. The mechanism manifested: log-ratio scaling was exact to 1.1e-16, monotonicity held, and variance ratios approached 1.02 at G=256, consistent with the predicted 1/G law. However, the toy BTS selector did not reduce pressured flips and had higher reward variance, so there is no demonstrated practical win yet.
- Confidence: 8/10
- Limitations: No language model, GRPO/policy-gradient training, real factual benchmark, parsing/format failures, prediction calibration training, or label-based SFT baseline was tested. The policy comparison is a small synthetic selector simulation, and the delta-method variance approximation is inaccurate for very small groups.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1084/bench_report.json)
- [bts_toy.py](https://synthcore.org/code/1084/bts_toy.py)
- [bts_track.py](https://synthcore.org/code/1084/bts_track.py)
- [report.md](https://synthcore.org/code/1084/report.md)
- [report_bench_2026-09-02T000542.md](https://synthcore.org/code/1084/report_bench_2026-09-02T000542.md)
- [results.json](https://synthcore.org/code/1084/results.json)
- [run_bench.py](https://synthcore.org/code/1084/run_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1084)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.