arXiv:2608.22227 2026 Training 1 ideas extracted · analyzed Aug 29, 2026
The paper's transferable asset is a stability-oriented replacement for a point quantile: average nearby lower quantiles so that small transition or return-distribution errors do not cause abrupt objective changes. This suggests a risk-sensitive distributional critic whose policy objective is a buffered lower-tail functional rather than a single estimated quantile, especially useful when return samples are sparse or learned from an imperfect model. The paper's optimism decomposition also gives a useful diagnostic by separating performance loss caused by buffering from loss caused by model or critic estimation error. The most direct neural implementation is a multi-quantile actor-critic with a differentiable averaged lower-tail objective and an annealed buffer width.
Replace the actor's point-quantile objective with the average of several lower quantiles of the learned return distribution. The averaging suppresses discontinuous policy updates caused by order-statistic noise and makes the risk objective less sensitive to small errors in the transition model or critic.