Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

What the math gives to ML

The paper provides a label-free peer-prediction reward for training models to resist user pressure: a response is rewarded when it occurs more often in a sampled group than the respondents predicted it would. The transferable asset is the use of a model's own conditional signal distribution and cross-sample empirical frequencies to create an information score without ground-truth labels. A practical adaptation is to sample several completions for each prompt, elicit a probability prediction over possible answers from every completion, and use a smoothed Bayesian Truth Serum score as the GRPO reward, with pressure-conditioned prompts used to test whether the policy stops copying the user's stated belief.