Environment Parameter Gradient Theorem for Policy-Environment Co-Design in Reinforcement Learning

What the math gives to ML

The paper provides a constructive gradient theorem for jointly optimizing a policy and parameters that alter the MDP transition kernel and instantaneous cost. Its key mechanism is to decouple the environment parameter used for the current transition from the parameter used in all future rollouts, yielding a local derivative involving the continuation value rather than an invalid derivative through the entire trajectory. This can transfer to neural networks by treating a differentiable data generator, simulator, augmentation policy, or world-model parameterization as a learnable environment jointly optimized with the network policy. The strongest test is whether the decoupled estimator remains unbiased and produces the predicted improvement over a policy-only baseline, especially when environment parameters affect both transition probabilities and costs.