{
 "artifacts": [
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/1084/bench_report.json"
  },
  {
   "name": "bts_toy.py",
   "url": "https://synthcore.org/code/1084/bts_toy.py"
  },
  {
   "name": "bts_track.py",
   "url": "https://synthcore.org/code/1084/bts_track.py"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/1084/report.md"
  },
  {
   "name": "report_bench_2026-09-02T000542.md",
   "url": "https://synthcore.org/code/1084/report_bench_2026-09-02T000542.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/1084/results.json"
  },
  {
   "name": "run_bench.py",
   "url": "https://synthcore.org/code/1084/run_bench.py"
  }
 ],
 "category": "training",
 "description": "Replace answer-label or preference rewards on belief-sensitive prompts with a peer-prediction reward computed from multiple model completions. Each completion reports an answer and a predicted distribution over answers; it receives positive reward when its answer is more frequent in the sampled group than that completion predicted, discouraging agreement with a user's belief when that agreement is not independently supported by the model's sampled population.",
 "download_zip": "https://synthcore.org/download/1084",
 "formulas_latex": [
  "$$\\pi_k:=\\Pr(S_r=k)$$",
  "$$\\Pr(k\\mid t):=\\Pr(S_{r'}=k\\mid S_r=t),\\qquad r'\\neq r$$",
  "$$x^r\\in[m],\\qquad y^r\\in\\Delta^m$$",
  "$$R_r=\\log\\frac{\\hat\\pi_{x^r}+\\varepsilon}{y^r_{x^r}+\\varepsilon},\\qquad \\hat\\pi_k=\\frac{1}{G}\\sum_{j=1}^{G}\\mathbf{1}[x^j=k]$$"
 ],
 "id": 2735,
 "implementation": "Integrate this at the reward-computation stage of GRPO or another group-based policy-gradient fine-tuning loop. For each prompt, construct a group of G independent completions from the current policy. Require each completion to emit an answer token or normalized answer string x^r and a prediction vector y^r over the m allowed answers; for binary questions, y^r can be one scalar probability for TRUE and 1-y^r for FALSE. Parse and validate the prediction, renormalize it, and clip every probability to [epsilon,1-epsilon]. Compute empirical answer frequencies across the group and assign each completion R_r=log((p_hat[x_r]+epsilon)/(y_r[x_r]+epsilon)). Subtract the group mean and divide by its standard deviation before forming the GRPO advantage. Add a small format penalty for invalid reports and clip rewards to [-5,5] to control variance for small groups. Pseudocode: sample completions {c_r}; parse (x_r,y_r); count each x_r; for each r compute p_hat[x_r]; reward[r]=log((p_hat[x_r]+eps)/(y_r[x_r]+eps)); normalize rewards; apply the ordinary GRPO policy update. The mathematical quantities are the empirical frequency, prediction probability, and logarithmic ratio; epsilon, clipping, group size, and format penalties are empirical engineering choices. Start with a 1B-3B instruction model and a binary factual benchmark containing neutral prompts and versions explicitly stating a misleading user belief. Compare BTS-GRPO with ordinary GRPO and label-based SFT at equal completion-sampling budget. Measure answer-flip rate under pressure, factual accuracy, prediction calibration, reward variance, and compute. Success is a significant reduction in pressure-induced flips without lower neutral-prompt accuracy, ideally with an absolute pressured-accuracy gain of several percentage points and no answer labels.",
 "math_summary": "The paper defines a latent world state w, a private signal S_r for respondent r, the marginal signal probability \\(\\pi_k:=\\Pr(S_r=k)\\), and the posterior predictive distribution \\(\\Pr(k\\mid t):=\\Pr(S_{r'}=k\\mid S_r=t)\\) for another respondent's signal. Each respondent submits an information report \\(x^r\\in[m]\\), represented by a one-hot vector \\(e_{x^r}\\), and a prediction report \\(y^r\\in\\Delta^m\\), where \\(y^r_k\\) is its predicted probability that another respondent reports answer k. The Bayesian Truth Serum information score compares the empirical frequency of the submitted answer with its predicted frequency: \\(I_r=\\log((\\hat\\pi_{x^r}+\\varepsilon)/(y^r_{x^r}+\\varepsilon))\\), where \\(\\hat\\pi_k=G^{-1}\\sum_{j=1}^{G}\\mathbf{1}[x^j=k]\\), G is the group size, and \\(\\varepsilon\u003e0\\) prevents zero logs. The numerator is the observed group frequency and the denominator is the respondent's prediction. The paper's large-group result says that, under its Bayesian signal assumptions, a sycophantic response has strictly lower expected information score than an honest response; the operational mechanism is the surprising-commonness ratio \\(\\hat\\pi_{x^r}/y^r_{x^r}\\).",
 "math_tags": [
  "probability",
  "information-theory",
  "game-theory",
  "statistics"
 ],
 "ml_areas": [
  "rlhf",
  "fine-tuning",
  "loss",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.25267",
  "arxiv_url": "https://arxiv.org/abs/2608.25267",
  "summary_what_math_gives_to_ml": "The paper provides a label-free peer-prediction reward for training models to resist user pressure: a response is rewarded when it occurs more often in a sampled group than the respondents predicted it would. The transferable asset is the use of a model's own conditional signal distribution and cross-sample empirical frequencies to create an information score without ground-truth labels. A practical adaptation is to sample several completions for each prompt, elicit a probability prediction over possible answers from every completion, and use a smoothed Bayesian Truth Serum score as the GRPO reward, with pressure-conditioned prompts used to test whether the policy stops copying the user's stated belief.",
  "title": "Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 6,
  "usefulness": 7
 },
 "solves": [
  "accuracy",
  "generalization",
  "sample-efficiency"
 ],
 "title": "Label-Free Bayesian Truth Serum Reward",
 "url": "https://synthcore.org/idea/2735/label-free-bayesian-truth-serum-reward",
 "verification": {
  "benchmark": {
   "track": "belief_sensitive_peer_prediction",
   "model": "mlp_tiny",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 0.016875,
    "idea_mean": 0.32875,
    "delta_mean": 0.311875,
    "per_seed_diffs": [
     0.1,
     0.14500000000000002,
     0.655,
     0.255,
     0.15500000000000003,
     0.27999999999999997,
     0.51,
     0.39499999999999996
    ],
    "idea_wins": 0,
    "n_pairs": 8,
    "p_value": 0.0081,
    "mde": 0,
    "mde_rel_pct": null,
    "verdict": "idea worse (significant)",
    "system_worked": false
   },
   "baseline": {
    "best_cfg": {
     "lr": 0.003
    },
    "sweep": [
     {
      "cfg": {
       "lr": 0.001
      },
      "mean": 0.01625
     },
     {
      "cfg": {
       "lr": 0.003
      },
      "mean": 0.015
     },
     {
      "cfg": {
       "lr": 0.01
      },
      "mean": 0.02
     }
    ],
    "full": {
     "mean": 0.016875,
     "std": 0.007474,
     "per_seed": [
      0.01,
      0.02,
      0.025,
      0.005,
      0.015,
      0.025,
      0.01,
      0.025
     ],
     "n": 8
    }
   },
   "idea": {
    "mean": 0.32875,
    "std": 0.184877,
    "per_seed": [
     0.11,
     0.165,
     0.68,
     0.26,
     0.17,
     0.305,
     0.52,
     0.42
    ],
    "n": 8
   },
   "mechanism_signature": {
    "confirmed": true,
    "group_size": 8,
    "log_ratio_max_error": 0,
    "n_model_completions": 1024,
    "observed_mean_frequency": 0.577148,
    "predicted_mean_frequency": 0.513152,
    "predicted_observed_correlation": 0.400833
   },
   "custom_track": null
  },
  "mechanism_confirmed": true,
  "peer_reviewed": false,
  "practical_verdict": "harms",
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 8,
   "verdict": "Built a readable NumPy implementation of the BTS information reward, three quantitative prediction sweeps, and a toy comparison against agreement selection. The mechanism manifested: log-ratio scaling was exact to 1.1e-16, monotonicity held, and variance ratios approached 1.02 at G=256, consistent with the predicted 1/G law. However, the toy BTS selector did not reduce pressured flips and had higher reward variance, so there is no demonstrated practical win yet.",
   "metrics": {
    "baseline": "Agreement: neutral accuracy 0.9975, pressured accuracy 0.5105, pressured flip rate 0.4895, reward variance 0.0469.",
    "idea": "BTS: neutral accuracy 0.8623, pressured accuracy 0.5033, pressured flip rate 0.4968, reward variance 0.2088; log-scaling max error 1.1e-16; variance observed/predicted ratio 1.02 at G=256."
   },
   "how_to_run": "python3 bts_toy.py",
   "files": [
    "bts_toy.py",
    "results.json",
    "report.md"
   ],
   "limitations": "No language model, GRPO/policy-gradient training, real factual benchmark, parsing/format failures, prediction calibration training, or label-based SFT baseline was tested. The policy comparison is a small synthetic selector simulation, and the delta-method variance approximation is inaccurate for very small groups."
  },
  "status": "mech_ok_no_baseline",
  "status_label": "Mechanism confirmed, baseline not beaten",
  "updated_at": "2026-09-02T00:05:42",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}