△ Mechanism confirmed, baseline not beaten 2026

Label-Free Bayesian Truth Serum Reward

Implementation & benchmark of arXiv:2608.25267 — Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

Usefulness7/10

Difficulty5/10

Novelty6/10

Training Accuracy Generalization Sample efficiency Rlhf Fine tuning Loss Training dynamics Probability Information theory Game theory Statistics

Source paper: Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach arXiv:2608.25267 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace answer-label or preference rewards on belief-sensitive prompts with a peer-prediction reward computed from multiple model completions. Each completion reports an answer and a predicted distribution over answers; it receives positive reward when its answer is more frequent in the sampled group than that completion predicted, discouraging agreement with a user's belief when that agreement is not independently supported by the model's sampled population.

Formulas

$$\pi_k:=\Pr(S_r=k)$$

$$\Pr(k\mid t):=\Pr(S_{r'}=k\mid S_r=t),\qquad r'\neq r$$

$$x^r\in[m],\qquad y^r\in\Delta^m$$

$$R_r=\log\frac{\hat\pi_{x^r}+\varepsilon}{y^r_{x^r}+\varepsilon},\qquad \hat\pi_k=\frac{1}{G}\sum_{j=1}^{G}\mathbf{1}[x^j=k]$$

Mathematical statement

The paper defines a latent world state w, a private signal S_r for respondent r, the marginal signal probability $\pi_k:=\Pr(S_r=k)$, and the posterior predictive distribution $\Pr(k\mid t):=\Pr(S_{r'}=k\mid S_r=t)$ for another respondent's signal. Each respondent submits an information report $x^r\in[m]$, represented by a one-hot vector $e_{x^r}$, and a prediction report $y^r\in\Delta^m$, where $y^r_k$ is its predicted probability that another respondent reports answer k. The Bayesian Truth Serum information score compares the empirical frequency of the submitted answer with its predicted frequency: $I_r=\log((\hat\pi_{x^r}+\varepsilon)/(y^r_{x^r}+\varepsilon))$, where $\hat\pi_k=G^{-1}\sum_{j=1}^{G}\mathbf{1}[x^j=k]$, G is the group size, and $\varepsilon>0$ prevents zero logs. The numerator is the observed group frequency and the denominator is the respondent's prediction. The paper's large-group result says that, under its Bayesian signal assumptions, a sycophantic response has strictly lower expected information score than an honest response; the operational mechanism is the surprising-commonness ratio $\hat\pi_{x^r}/y^r_{x^r}$.

Implementation notes

Integrate this at the reward-computation stage of GRPO or another group-based policy-gradient fine-tuning loop. For each prompt, construct a group of G independent completions from the current policy. Require each completion to emit an answer token or normalized answer string x^r and a prediction vector y^r over the m allowed answers; for binary questions, y^r can be one scalar probability for TRUE and 1-y^r for FALSE. Parse and validate the prediction, renormalize it, and clip every probability to [epsilon,1-epsilon]. Compute empirical answer frequencies across the group and assign each completion R_r=log((p_hat[x_r]+epsilon)/(y_r[x_r]+epsilon)). Subtract the group mean and divide by its standard deviation before forming the GRPO advantage. Add a small format penalty for invalid reports and clip rewards to [-5,5] to control variance for small groups. Pseudocode: sample completions {c_r}; parse (x_r,y_r); count each x_r; for each r compute p_hat[x_r]; reward[r]=log((p_hat[x_r]+eps)/(y_r[x_r]+eps)); normalize rewards; apply the ordinary GRPO policy update. The mathematical quantities are the empirical frequency, prediction probability, and logarithmic ratio; epsilon, clipping, group size, and format penalties are empirical engineering choices. Start with a 1B-3B instruction model and a binary factual benchmark containing neutral prompts and versions explicitly stating a misleading user belief. Compare BTS-GRPO with ordinary GRPO and label-based SFT at equal completion-sampling budget. Measure answer-flip rate under pressure, factual accuracy, prediction calibration, reward variance, and compute. Success is a significant reduction in pressure-induced flips without lower neutral-prompt accuracy, ideally with an absolute pressured-accuracy gain of several percentage points and no answer labels.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a readable NumPy implementation of the BTS information reward, three quantitative prediction sweeps, and a toy comparison against agreement selection. The mechanism manifested: log-ratio scaling was exact to 1.1e-16, monotonicity held, and variance ratios approached 1.02 at G=256, consistent with the predicted 1/G law. However, the toy BTS selector did not reduce pressured flips and had higher reward variance, so there is no demonstrated practical win yet.

Agent confidence: 8/10

Baseline: Agreement: neutral accuracy 0.9975, pressured accuracy 0.5105, pressured flip rate 0.4895, reward variance 0.0469.
Idea: BTS: neutral accuracy 0.8623, pressured accuracy 0.5033, pressured flip rate 0.4968, reward variance 0.2088; log-scaling max error 1.1e-16; variance observed/predicted ratio 1.02 at G=256.

Limitations: No language model, GRPO/policy-gradient training, real factual benchmark, parsing/format failures, prediction calibration training, or label-based SFT baseline was tested. The policy comparison is a small synthetic selector simulation, and the delta-method variance approximation is inaccurate for very small groups.

How to run: python3 bts_toy.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.25267) Sep 2, 2026

Significantly WORSE than the tuned baseline (+1848.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Belief_sensitive_peer_prediction
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 0.0169
Idea mean: 0.3287
Effect (Δ): 0.3119 (+1848.1%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

This was a small synthetic custom binary classification track rather than language-model GRPO on natural-language factual prompts. It did not test parsing failures, larger completion groups, pretrained models, preference/SFT baselines, or multi-turn pressure robustness.

How to run: python3 run_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 10/10

Built a structurally matched custom peer-prediction classification track and trained identical MLP systems with supervised baseline versus label-free BTS policy-gradient rewards. The trained-model signature confirmed the observed/predicted frequency log-ratio calculation, but BTS had substantially worse test error: 0.328750 versus 0.016875, with paired delta +0.311875 and permutation p=0.0081. Therefore the mechanism was numerically present but did not transfer into a benchmark win.

Baseline: Best lr=0.003; 8-seed test error mean 0.016875, std 0.007474.
Idea: Best lr=0.003; 8-seed test error mean 0.328750, std 0.184877; paired delta +0.311875, p=0.0081, 0/8 wins.

Benchmark result