# Counterfactual-tracking policy ensemble

- ID: 839
- Canonical URL: https://synthcore.org/idea/839/counterfactual-tracking-policy-ensemble
- API JSON: https://synthcore.org/api/idea/839.json
- API Markdown: https://synthcore.org/api/idea/839.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.13029](https://arxiv.org/abs/2607.13029)
- Category: dynamics
- Solves: stability, sample-efficiency, generalization
- ML areas: rl, world-model, training-dynamics
- Math tags: dynamical-systems, control-theory, optimization, probability
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Maintain a posterior over heterogeneous neural policies, simulate each policy on the same revealed disturbance sequence, and track a posterior-weighted counterfactual reference instead of directly switching among deployed policies. A stabilizing feedback correction keeps the physical state close to the reference, while exponential-weights updates favor policies with low counterfactual cost.

## Mathematical statement

The plant is the linear dynamical system \(x_{t+1}=Ax_t+Bu_t+w_t\), where \(x_t\in\mathbb{R}^n\) is the physical state, \(u_t\in\mathbb{R}^m\) is the applied control, \(A,B\) are known matrices, and \(w_t\) is the realized disturbance satisfying \(\lVert w_t\rVert\le W\). For policy \(\pi_i\), its counterfactual trajectory is \(\hat u_{i,t}=\pi_i(\hat x_{i,0:t},c_{1:t-1})\) and \(\hat x_{i,t+1}=A\hat x_{i,t}+B\hat u_{i,t}+w_t\), where \(c_s\) is the revealed cost function at time \(s\). The exponential-weights posterior has PAC-Bayes dependence \(O(\sqrt{T\,\mathrm{KL}(q\|p)})\), and for a finite class of \(N\) policies the target regret is \(O(\sqrt{T\log N})\). Here \(p\) is the prior over policies, \(q\) is the learned posterior, and \(T\) is the horizon. The tracking action is \(u_t=\bar u_t-K(x_t-\bar x_t)\), where \(K\) is selected so that \(F=A-BK\) is stable, and \((\bar x_t,\bar u_t)\) is a posterior-weighted counterfactual reference. The tracking error \(e_t=x_t-\bar x_t\) follows \(e_{t+1}=Fe_t\) when the reference uses the same disturbance sequence. A bounded impulse-response gain \(G_H=\sum_{k=0}^{H}\lVert F^k\rVert_2\) limits accumulated error over a finite horizon. If each convex cost is locally \(L_c\)-Lipschitz, tracking error contributes at most approximately \(L_c\sum_t\lVert e_t\rVert\) to excess cost.

## Key formulas

- $$x_{t+1}=Ax_t+Bu_t+w_t,\qquad \lVert w_t\rVert\le W.$$
- $$\hat u_{i,t}=\pi_i(\hat x_{i,0:t},c_{1:t-1}),\qquad \hat x_{i,t+1}=A\hat x_{i,t}+B\hat u_{i,t}+w_t.$$
- $$q_{t+1}(i)=\frac{q_t(i)\exp\{-\eta c_t(\hat x_{i,t},\hat u_{i,t})\}}{\sum_{j=1}^{N}q_t(j)\exp\{-\eta c_t(\hat x_{j,t},\hat u_{j,t})\}},\qquad \mathrm{Regret}_T=O\!\left(\sqrt{T\log N}\right).$$
- $$u_t=\bar u_t-K(x_t-\bar x_t),\qquad e_{t+1}=(A-BK)e_t,\qquad e_t=x_t-\bar x_t.$$

## Implementation notes

(1) Exact integration point: place the method between a neural policy ensemble and the environment action interface in an online RL or model-based control loop. Each member may be an MLP, GRU, transformer policy, or saved checkpoint with its own hidden state. Start with a known linear simulator \(x_{t+1}=Ax_t+Bu_t+w_t\), then test a locally identified model. Choose \(K\) offline using discrete LQR or pole placement so that \(F=A-BK\) has spectral radius below one. The wrapper changes the action immediately before environment execution. (2) Pseudocode: initialize \(q_i=1/N\), counterfactual states \(\hat x_i=x_0\), and hidden states \(h_i\). At each time \(t\), compute \(\hat u_i=\pi_i(\hat x_i,h_i)\) for every policy; compute \(\bar x=\sum_i q_i\hat x_i\) and \(\bar u=\sum_i q_i\hat u_i\); execute \(u=\bar u-K(x_t-\bar x)\), with optional actuator clipping; observe \(c_t\), \(x_{t+1}\), and recover the disturbance \(w_t=x_{t+1}-Ax_t-Bu_t\). Evaluate each counterfactual loss \(\ell_i=c_t(\hat x_i,\hat u_i)\), update \(q_i\leftarrow q_i\exp(-\eta\ell_i)\), normalize the weights, and advance every simulator using \(\hat x_i\leftarrow A\hat x_i+B\hat u_i+w_t\). (3) Compute counterfactual trajectories and exponential weights exactly. Estimate the finite-horizon impulse gain from \(G_H=\sum_{k=0}^{H}\lVert F^k\rVert_2\), using matrix powers; estimate the local cost Lipschitz constant from gradients or finite differences. Normalize costs to \([0,L]\) and initialize \(\eta=\sqrt{2\log N/(T L^2)}\), with a small validation grid for robustness. The mathematical prediction is the \(\sqrt{T\log N}\) selection term; the neural adaptation must empirically test whether the plant model and bounded-reference assumptions hold. (4) First cheap experiment: use a 4- to 8-dimensional stable linear system with bounded adversarial disturbances, four heterogeneous controllers consisting of an MLP, GRU, transformer policy, and poor baseline, and convex quadratic costs. Compare direct Hedge action selection, the best fixed policy, and counterfactual tracking at equal environment interactions. Record cumulative cost, maximum state norm, posterior concentration, tracking error, and action magnitude. Success means substantially lower worst-case state norm than direct switching with similar cumulative cost, plus cumulative regret approaching the best policy with an empirical slope consistent with \(\sqrt{T\log N}\).

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a self-contained counterfactual-tracking ensemble simulator with exponential-weights updates, disturbance sharing, LQR feedback correction, and direct-switching comparison. The stability boundary, finite-horizon impulse-gain formula, and Hedge square-root regret scaling all matched numerical predictions. Tracking reduced cumulative cost by 29.6% versus direct switching, although maximum state magnitude was identical in this benign toy setup.

### Mechanism check

- Verdict: Built a self-contained counterfactual-tracking ensemble simulator with exponential-weights updates, disturbance sharing, LQR feedback correction, and direct-switching comparison. The stability boundary, finite-horizon impulse-gain formula, and Hedge square-root regret scaling all matched numerical predictions. Tracking reduced cumulative cost by 29.6% versus direct switching, although maximum state magnitude was identical in this benign toy setup.
- Confidence: 8/10
- Limitations: The ensemble uses fixed scalar linear-controller surrogates rather than MLP, GRU, or transformer policies. The plant is one-dimensional, known, and benign; model mismatch, actuator failure, adversarial disturbances, partial observability, neural adaptation, and larger multidimensional systems were not tested. The maximum-state metric did not distinguish tracking from switching in this run.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.5872
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/342/bench_report.json)
- [experiment.py](https://synthcore.org/code/342/experiment.py)
- [report.md](https://synthcore.org/code/342/report.md)
- [report_bench_2026-08-31T014423.md](https://synthcore.org/code/342/report_bench_2026-08-31T014423.md)
- [results.json](https://synthcore.org/code/342/results.json)
- [stage2_bench.py](https://synthcore.org/code/342/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/342)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.