# Buffered Lower-Tail Distributional Critic

- ID: 51
- Canonical URL: https://synthcore.org/idea/51/buffered-lower-tail-distributional-critic
- API JSON: https://synthcore.org/api/idea/51.json
- API Markdown: https://synthcore.org/api/idea/51.md
- Verification status: unverified
- Source: [arXiv:2608.22227](https://arxiv.org/abs/2608.22227)
- Category: training
- Solves: stability, accuracy, sample-efficiency
- ML areas: rl, loss, world-model
- Math tags: probability, statistics, optimization
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Replace the actor's point-quantile objective with the average of several lower quantiles of the learned return distribution. The averaging suppresses discontinuous policy updates caused by order-statistic noise and makes the risk objective less sensitive to small errors in the transition model or critic.

## Mathematical statement

For a policy \(\pi\), transition kernel \(P\), stage \(h\), and state \(s\), the paper defines the remaining-return law \(G_{h,s}^{\pi,P}=\mathcal{L}(\sum_{k=h}^{H-1}r_k(S_k,A_k)\mid S_h=s,\pi,P)\), with exact quantile value \(V_{q,h}^{\pi,P}(s)=Q_q(G_{h,s}^{\pi,P})\) and lower-buffered value \(V_{q,h}^{\pi,P,\beta}(s)=Q_q^\beta(G_{h,s}^{\pi,P})\). Here \(Q_q\) is the generalized inverse CDF quantile at probability level \(q\in(0,1)\), and \(\beta\) is the lower-buffer width. Use the lower-buffer functional \(Q_{\tau}^{\beta}(G)=\frac{1}{\beta}\int_{\tau-\beta}^{\tau}Q_u(G)\,du\), where \(\tau\) is the target quantile and \(0<\beta\leq\tau\). With \(M\) learned quantile atoms \(z_i\) at levels \(u_i\), approximate it by \(\widehat Q_{\tau}^{\beta}=|I|^{-1}\sum_{i\in I}z_i\), where \(I=\{i:\tau-\beta\leq u_i\leq\tau\}\). The paper's regret decomposition is \(\operatorname{Reg}_{\tau}(T)\leq\sum_{t=0}^{T-1}[2\Delta_{\beta_t}+(V_{\tau,0}^{\pi^t,P^t,\beta_t}(\bar s)-V_{\tau,0}^{\pi^t,P^\star,\beta_t}(\bar s))]\), on a confidence event. Here \(\Delta_{\beta_t}\) is the buffering approximation gap, \(P^t\) is the estimated model, \(P^\star\) is the true model, and \(\beta_t\) is the buffer used at episode \(t\). This decomposition motivates measuring buffering bias separately from estimation error.

## Key formulas

- $$G_{h,s}^{\pi,P}\coloneqq\mathcal{L}\!\left(\sum_{k=h}^{H-1}r_k(S_k,A_k)\,\middle|\,S_h=s,\pi,P\right),\qquad V_{q,h}^{\pi,P}(s)\coloneqq Q_q\!\left(G_{h,s}^{\pi,P}\right),\qquad V_{q,h}^{\pi,P,\beta}(s)\coloneqq Q_q^\beta\!\left(G_{h,s}^{\pi,P}\right).$$
- $$Q_{\tau}^{\beta}(G)=\frac{1}{\beta}\int_{\tau-\beta}^{\tau}Q_u(G)\,du,\qquad 0<\beta\leq\tau,$$
- $$\widehat Q_{\tau}^{\beta}(s,a)=\frac{1}{|I_{\tau,\beta}|}\sum_{i\in I_{\tau,\beta}}z_i(s,a),\qquad I_{\tau,\beta}=\{i:\tau-\beta\leq u_i\leq\tau\},$$
- $$\operatorname{Reg}_{\tau}(T)\leq\sum_{t=0}^{T-1}\left(2\Delta_{\beta_t}+V_{\tau,0}^{\pi^t,P^t,\beta_t}(\bar{s})-V_{\tau,0}^{\pi^t,P^\star,\beta_t}(\bar{s})\right).$$

## Implementation notes

(1) Integration point: use a distributional actor-critic or quantile DQN with a critic \(Z_\theta(s,a)\) that outputs \(M\) quantile values \(z_i(s,a)\) at fixed levels \(u_i=(i-0.5)/M\). Replace the actor objective and action-selection score with the buffered score \(J_\beta(s,a)=|I|^{-1}\sum_{i\in I}z_i(s,a)\), with \(I=\{i:\tau-\beta\leq u_i\leq\tau\}\). For continuous actions, maximize this score with the actor; for discrete actions, choose \(\arg\max_a J_\beta(s,a)\). (2) Pseudocode: initialize critic parameters \(\theta\), actor parameters \(\phi\), target critic \(\bar\theta\), target level \(\tau\), and buffer \(\beta\). For each transition \((s,a,r,s')\), choose \(a'\) using \(J_\beta(s',a')\); form quantile targets \(y_i=r+\gamma z_i^{\bar\theta}(s',a')\); minimize the quantile Huber loss between \(z_i^\theta(s,a)\) and \(y_i\); update the actor using \(\nabla_\phi J_\beta(s,\pi_\phi(s))\); and periodically update \(\bar\theta\). Anneal \(\beta\) according to \(\beta_t=\max(\beta_{\min},\beta_0(1-t/T_{\mathrm{anneal}}))\). (3) Computed from the mathematics: the lower-tail interval, the averaged quantile functional, and the distinction between buffer bias and estimation error. Estimated empirically: quantile atoms, critic targets, and \(\Delta_\beta\), using held-out trajectories and bootstrap intervals; estimate \(\widehat\Delta_\beta=|\widehat Q_\tau-\widehat Q_\tau^\beta|\). (4) First experiment: test on CartPole and a stochastic CliffWalking or MiniGrid task with catastrophic lower-tail outcomes. Compare point-quantile training, fixed \(\beta\in\{0.05,0.1,0.2\}\), and annealed buffering with identical networks, replay data, and gradient FLOPs. Measure held-out exact \(\tau\)-quantile return, buffered return, seed variance, critic divergence count, and area under the learning curve. The prediction is lower seed variance and faster improvement in held-out lower-tail return for moderate \(\beta\), with a measurable exact-quantile bias when \(\beta\) is too large.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.