{
 "artifacts": null,
 "category": "training",
 "description": "Replace the actor's point-quantile objective with the average of several lower quantiles of the learned return distribution. The averaging suppresses discontinuous policy updates caused by order-statistic noise and makes the risk objective less sensitive to small errors in the transition model or critic.",
 "formulas_latex": [
  "$$G_{h,s}^{\\pi,P}\\coloneqq\\mathcal{L}\\!\\left(\\sum_{k=h}^{H-1}r_k(S_k,A_k)\\,\\middle|\\,S_h=s,\\pi,P\\right),\\qquad V_{q,h}^{\\pi,P}(s)\\coloneqq Q_q\\!\\left(G_{h,s}^{\\pi,P}\\right),\\qquad V_{q,h}^{\\pi,P,\\beta}(s)\\coloneqq Q_q^\\beta\\!\\left(G_{h,s}^{\\pi,P}\\right).$$",
  "$$Q_{\\tau}^{\\beta}(G)=\\frac{1}{\\beta}\\int_{\\tau-\\beta}^{\\tau}Q_u(G)\\,du,\\qquad 0\u003c\\beta\\leq\\tau,$$",
  "$$\\widehat Q_{\\tau}^{\\beta}(s,a)=\\frac{1}{|I_{\\tau,\\beta}|}\\sum_{i\\in I_{\\tau,\\beta}}z_i(s,a),\\qquad I_{\\tau,\\beta}=\\{i:\\tau-\\beta\\leq u_i\\leq\\tau\\},$$",
  "$$\\operatorname{Reg}_{\\tau}(T)\\leq\\sum_{t=0}^{T-1}\\left(2\\Delta_{\\beta_t}+V_{\\tau,0}^{\\pi^t,P^t,\\beta_t}(\\bar{s})-V_{\\tau,0}^{\\pi^t,P^\\star,\\beta_t}(\\bar{s})\\right).$$"
 ],
 "id": 51,
 "implementation": "(1) Integration point: use a distributional actor-critic or quantile DQN with a critic \\(Z_\\theta(s,a)\\) that outputs \\(M\\) quantile values \\(z_i(s,a)\\) at fixed levels \\(u_i=(i-0.5)/M\\). Replace the actor objective and action-selection score with the buffered score \\(J_\\beta(s,a)=|I|^{-1}\\sum_{i\\in I}z_i(s,a)\\), with \\(I=\\{i:\\tau-\\beta\\leq u_i\\leq\\tau\\}\\). For continuous actions, maximize this score with the actor; for discrete actions, choose \\(\\arg\\max_a J_\\beta(s,a)\\). (2) Pseudocode: initialize critic parameters \\(\\theta\\), actor parameters \\(\\phi\\), target critic \\(\\bar\\theta\\), target level \\(\\tau\\), and buffer \\(\\beta\\). For each transition \\((s,a,r,s')\\), choose \\(a'\\) using \\(J_\\beta(s',a')\\); form quantile targets \\(y_i=r+\\gamma z_i^{\\bar\\theta}(s',a')\\); minimize the quantile Huber loss between \\(z_i^\\theta(s,a)\\) and \\(y_i\\); update the actor using \\(\\nabla_\\phi J_\\beta(s,\\pi_\\phi(s))\\); and periodically update \\(\\bar\\theta\\). Anneal \\(\\beta\\) according to \\(\\beta_t=\\max(\\beta_{\\min},\\beta_0(1-t/T_{\\mathrm{anneal}}))\\). (3) Computed from the mathematics: the lower-tail interval, the averaged quantile functional, and the distinction between buffer bias and estimation error. Estimated empirically: quantile atoms, critic targets, and \\(\\Delta_\\beta\\), using held-out trajectories and bootstrap intervals; estimate \\(\\widehat\\Delta_\\beta=|\\widehat Q_\\tau-\\widehat Q_\\tau^\\beta|\\). (4) First experiment: test on CartPole and a stochastic CliffWalking or MiniGrid task with catastrophic lower-tail outcomes. Compare point-quantile training, fixed \\(\\beta\\in\\{0.05,0.1,0.2\\}\\), and annealed buffering with identical networks, replay data, and gradient FLOPs. Measure held-out exact \\(\\tau\\)-quantile return, buffered return, seed variance, critic divergence count, and area under the learning curve. The prediction is lower seed variance and faster improvement in held-out lower-tail return for moderate \\(\\beta\\), with a measurable exact-quantile bias when \\(\\beta\\) is too large.",
 "math_summary": "For a policy \\(\\pi\\), transition kernel \\(P\\), stage \\(h\\), and state \\(s\\), the paper defines the remaining-return law \\(G_{h,s}^{\\pi,P}=\\mathcal{L}(\\sum_{k=h}^{H-1}r_k(S_k,A_k)\\mid S_h=s,\\pi,P)\\), with exact quantile value \\(V_{q,h}^{\\pi,P}(s)=Q_q(G_{h,s}^{\\pi,P})\\) and lower-buffered value \\(V_{q,h}^{\\pi,P,\\beta}(s)=Q_q^\\beta(G_{h,s}^{\\pi,P})\\). Here \\(Q_q\\) is the generalized inverse CDF quantile at probability level \\(q\\in(0,1)\\), and \\(\\beta\\) is the lower-buffer width. Use the lower-buffer functional \\(Q_{\\tau}^{\\beta}(G)=\\frac{1}{\\beta}\\int_{\\tau-\\beta}^{\\tau}Q_u(G)\\,du\\), where \\(\\tau\\) is the target quantile and \\(0\u003c\\beta\\leq\\tau\\). With \\(M\\) learned quantile atoms \\(z_i\\) at levels \\(u_i\\), approximate it by \\(\\widehat Q_{\\tau}^{\\beta}=|I|^{-1}\\sum_{i\\in I}z_i\\), where \\(I=\\{i:\\tau-\\beta\\leq u_i\\leq\\tau\\}\\). The paper's regret decomposition is \\(\\operatorname{Reg}_{\\tau}(T)\\leq\\sum_{t=0}^{T-1}[2\\Delta_{\\beta_t}+(V_{\\tau,0}^{\\pi^t,P^t,\\beta_t}(\\bar s)-V_{\\tau,0}^{\\pi^t,P^\\star,\\beta_t}(\\bar s))]\\), on a confidence event. Here \\(\\Delta_{\\beta_t}\\) is the buffering approximation gap, \\(P^t\\) is the estimated model, \\(P^\\star\\) is the true model, and \\(\\beta_t\\) is the buffer used at episode \\(t\\). This decomposition motivates measuring buffering bias separately from estimation error.",
 "math_tags": [
  "probability",
  "statistics",
  "optimization"
 ],
 "ml_areas": [
  "rl",
  "loss",
  "world-model"
 ],
 "paper": {
  "arxiv_id": "2608.22227",
  "arxiv_url": "https://arxiv.org/abs/2608.22227",
  "summary_what_math_gives_to_ml": "The paper's transferable asset is a stability-oriented replacement for a point quantile: average nearby lower quantiles so that small transition or return-distribution errors do not cause abrupt objective changes. This suggests a risk-sensitive distributional critic whose policy objective is a buffered lower-tail functional rather than a single estimated quantile, especially useful when return samples are sparse or learned from an imperfect model. The paper's optimism decomposition also gives a useful diagnostic by separating performance loss caused by buffering from loss caused by model or critic estimation error. The most direct neural implementation is a multi-quantile actor-critic with a differentiable averaged lower-tail objective and an annealed buffer width.",
  "title": "Risk-Sensitive Reinforcement Learning with Smoothed Quantile Objectives",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 6,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "accuracy",
  "sample-efficiency"
 ],
 "title": "Buffered Lower-Tail Distributional Critic",
 "url": "https://synthcore.org/idea/51/buffered-lower-tail-distributional-critic",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}