Counterfactual-tracking policy ensemble

(1) Exact integration point: place the method between a neural policy ensemble and the environment action interface in an online RL or model-based control loop. Each member may be an MLP, GRU, transformer policy, or saved checkpoint with its own hidden state. Start with a known linear simulator \(x_{t+1}=Ax_t+Bu_t+w_t\), then test a locally identified model. Choose \(K\) offline using discrete LQR or pole placement so that \(F=A-BK\) has spectral radius below one. The wrapper changes the action immediately before environment execution. (2) Pseudocode: initialize \(q_i=1/N\), counterfactual states \(\hat x_i=x_0\), and hidden states \(h_i\). At each time \(t\), compute \(\hat u_i=\pi_i(\hat x_i,h_i)\) for every policy; compute \(\bar x=\sum_i q_i\hat x_i\) and \(\bar u=\sum_i q_i\hat u_i\); execute \(u=\bar u-K(x_t-\bar x)\), with optional actuator clipping; observe \(c_t\), \(x_{t+1}\), and recover the disturbance \(w_t=x_{t+1}-Ax_t-Bu_t\). Evaluate each counterfactual loss \(\ell_i=c_t(\hat x_i,\hat u_i)\), update \(q_i\leftarrow q_i\exp(-\eta\ell_i)\), normalize the weights, and advance every simulator using \(\hat x_i\leftarrow A\hat x_i+B\hat u_i+w_t\). (3) Compute counterfactual trajectories and exponential weights exactly. Estimate the finite-horizon impulse gain from \(G_H=\sum_{k=0}^{H}\lVert F^k\rVert_2\), using matrix powers; estimate the local cost Lipschitz constant from gradients or finite differences. Normalize costs to \([0,L]\) and initialize \(\eta=\sqrt{2\log N/(T L^2)}\), with a small validation grid for robustness. The mathematical prediction is the \(\sqrt{T\log N}\) selection term; the neural adaptation must empirically test whether the plant model and bounded-reference assumptions hold. (4) First cheap experiment: use a 4- to 8-dimensional stable linear system with bounded adversarial disturbances, four heterogeneous controllers consisting of an MLP, GRU, transformer policy, and poor baseline, and convex quadratic costs. Compare direct Hedge action selection, the best fixed policy, and counterfactual tracking at equal environment interactions. Record cumulative cost, maximum state norm, posterior concentration, tracking error, and action magnitude. Success means substantially lower worst-case state norm than direct switching with similar cumulative cost, plus cumulative regret approaching the best policy with an empirical slope consistent with \(\sqrt{T\log N}\).