Regime-Adaptive Robust Critic

Integrate this method into the critic target of an average-reward neural actor-critic, not into the actor network initially. Maintain a scalar bias critic hθ(s), an average-reward estimate ρ̂, and a reservoir R of several thousand recently observed states plus uniformly sampled replay states. Every K critic updates, compute Ĥ0=maxR hθ−minR hθ using a slowly updated nominal critic or an exponential moving average of the current critic before robustification. Set the regime ratio τ̂=σĤ0/ε. For each transition (s,a,r,s′), form the nominal residual δnom=r−ρ̂+hθ(s′)−hθ(s). Form the robust residual δrob=δnom−(σ/2)Ĥ, where Ĥ is computed from the current reservoir; stop gradients through Ĥ so the critic cannot reduce the penalty simply by shrinking its own span. Use δrob when τ̂≤c and δnom when τ̂>c, with c initially 1 and optional hysteresis thresholds 0.8 and 1.2 to prevent rapid switching. Update θ by Huber loss on the selected residual, update ρ̂ by ρ̂←ρ̂+βρ(r−ρ̂), and update the actor using an advantage estimated from the selected critic. The mathematical quantities supplied by the paper are the regime boundary and span-dependent penalty; the state reservoir and uncertainty estimate are empirical engineering approximations. If σ is unknown, estimate it from an ensemble of transition models using the average clipped total variation between ensemble predictive distributions. First test on a 20-state uncertain chain or MiniGrid with a 2-layer MLP, comparing nominal actor-critic, always-robust critic, and this adaptive switch with equal replay samples. Evaluate nominal return, worst-case return under sampled TV perturbations, critic span, switching frequency, and steps to reach a fixed robust-return threshold. The method succeeds if it learns faster than always-robust training when ε>σH₀ and achieves higher worst-case return than nominal training when ε<σH₀, without oscillation near the boundary.