ML: Rlhf

Machine-learning ideas tagged Rlhf in the ML taxonomy of the Math2NN corpus.

△ Mechanism confirmed, baseline not beaten 2026

Bellman Stopping Controller for Self-Refinement

Attach a value-based stopping controller to any verifier-guided refinement loop. After each generated answer and verifier evaluation, estimate the value of accepting the current output and the value of continuing for one or more additional refinements; stop when the expected gain from continuation is no larger than its compute cost. The controller learns a score-dependent stopping boundary instead of using a fixed iteration count.

Useful8/10
Difficulty5/10
Novelty5/10
Paper: Optimal Stopping of Self-Refining Foundation Models arXiv:2608.10729
△ Mechanism confirmed, baseline not beaten 2026

Tail-Aware Verifier Portfolio

Use the paper's tail comparison to decide when another call from the same verifier family is useless and when to switch to a different model, modality, or evidence source. The objective is to reduce the high-alpha survivor population—the incorrect examples that consistently fool one verifier—rather than maximizing average one-shot verifier accuracy.

Useful8/10
Difficulty5/10
Novelty7/10
Paper: Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings arXiv:2607.13918
△ Mechanism confirmed, baseline not beaten 2026

Tikhonov-Stabilized Stochastic Extragradient

Replace the raw stochastic saddle objective by a strongly convex-strongly concave, quadratically anchored objective before applying stochastic extragradient. For a generator-discriminator or policy-rewarder game, anchor the minimizing and maximizing parameter vectors to reference parameters with opposite signs, suppressing persistent stochastic rotations and improving the quality of the final iterate.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Last-Iterate Convergence of Single-Loop Stochastic Methods for Constrained Convex-Concave Minimax Problems arXiv:2607.11056
Unverified 2026

Slack-aware anytime stopping

Attach a nonnegative e-process to a held-out stream used to monitor adaptively chosen neural-network checkpoints. Instead of using only Ville's conservative threshold b = 1/α, estimate overshoot, drift loss, and surviving mass, then test whether a conservative version of the exact identity permits earlier detection at the same empirical type-I error.

Useful5/10
Difficulty6/10
Novelty7/10
Paper: The Exact Ville Identity: From the Absorbing Case to the General Law with an Application to E-Values arXiv:2607.04620