# Automaton-Gated Objective Switching

- ID: 1821
- Canonical URL: https://synthcore.org/idea/1821/automaton-gated-objective-switching
- API JSON: https://synthcore.org/api/idea/1821.json
- API Markdown: https://synthcore.org/api/idea/1821.md
- Verification status: unverified
- Source: [arXiv:2608.05021](https://arxiv.org/abs/2608.05021)
- Category: architecture
- Solves: accuracy, stability, sample-efficiency
- ML areas: rl, world-model, training-dynamics
- Math tags: logic, probability, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Augment a neural policy with deterministic DFA states for the task objective and safety constraint, then select among objective-specific policy heads using those states. Before either target is reached, execute a mixed policy; after one target is reached, switch permanently to the policy specialized for the remaining target.

## Mathematical statement

The paper constructs a product MDP state s^times=<s,q_phi,q_psi>, where s is the environment state and q_phi,q_psi are states of deterministic finite automata for formulas phi and psi. The product transition preserves the environment transition T while deterministically updating both automata using the label L(s) and DFA transition functions delta_phi and delta_psi. A stationary policy on the product MDP is implemented in the original environment by reconstructing q_phi and q_psi from the history h_t=(s_0,a_0,...,s_t). The switching policy uses pi_mix when neither target set has been reached, pi_psi when phi has been reached but psi has not, and pi_phi when psi has been reached but phi has not. Z_phi and Z_psi are absorbing target-state sets, so the switch is irreversible. In the neural adaptation, pi_mix, pi_phi, and pi_psi are separate policy heads, while the automaton states provide exact finite memory and determine the active head.

## Key formulas

- $$T^{\times}(s^{\times},a,s^{\times\prime})=\begin{cases}T(s,a,s^{\prime})&\text{if }s^{\times}=\langle s,q_{\phi},q_{\psi}\rangle,\\&s^{\times\prime}=\langle s^{\prime},\delta_{\phi}(q_{\phi},L(s)),\delta_{\psi}(q_{\psi},L(s))\rangle,\\0&\text{otherwise.}\end{cases}$$
- $$\pi(\cdot\mid h_t):=\pi^{\times}(\cdot\mid\langle s_t,q_{\phi,t},q_{\psi,t}\rangle).$$
- $$\\pi_{s\\to s}(\\langle s,q_{\\phi},q_{\\psi}\\rangle)=\\begin{cases}\\pi_{\\mathrm{mix}}(\\langle s,q_{\\phi},q_{\\psi}\\rangle),&q_{\\phi}\\notin Z_{\\phi}\\ \text{and}\\ q_{\\psi}\\notin Z_{\\psi},\\\\\\pi_{\\psi}(\\langle s,q_{\\phi},q_{\\psi}\\rangle),&q_{\\phi}\\in Z_{\\phi}\\ \text{and}\\ q_{\\psi}\\notin Z_{\\psi},\\\\\\pi_{\\phi}(\\langle s,q_{\\phi},q_{\\psi}\\rangle),&q_{\\phi}\\notin Z_{\\phi}\\ \text{and}\\ q_{\\psi}\\in Z_{\\psi}.\\end{cases}$$
- $$\mathcal{L}_{\mathrm{switch}}=-\mathbb{E}_t\left[\log\pi_{g_t}(a_t\mid o_t,q_{\phi,t},q_{\psi,t})\right],\qquad g_t=\begin{cases}\mathrm{mix},&q_{\phi,t}\notin Z_{\phi},\ q_{\psi,t}\notin Z_{\psi},\\\psi,&q_{\phi,t}\in Z_{\phi},\ q_{\psi,t}\notin Z_{\psi},\\\phi,&q_{\phi,t}\notin Z_{\phi},\ q_{\psi,t}\in Z_{\psi}.\end{cases}$$

## Implementation notes

Integrate the mechanism at the policy-selection layer of PPO or an actor-critic agent. Maintain two integer DFA states q_phi and q_psi outside the neural network. At every environment step, compute proposition labels L(o_t) from the observation, update q_phi <- delta_phi(q_phi,L(o_t)) and q_psi <- delta_psi(q_psi,L(o_t)), and classify the mode g_t using the three cases in the switching equation. Use a shared encoder z_t=f_theta(o_t), concatenate z_t with one-hot or learned embeddings of q_phi and q_psi, and attach three action heads pi_mix, pi_phi, and pi_psi. Sample actions only from the selected head and store the selected head index with each trajectory transition. For PPO, compute the log probability and ratio using the same selected head that generated the action; optionally use three value heads because the return distribution can change after switching. Pseudocode is: initialize automata; for each step observe o, compute l=L(o), update both DFA states, choose g, sample a from pi_g, execute a, store (o,q_phi,q_psi,g,a,r), and perform the usual PPO update. The exact automaton transitions and gate are taken from the paper; network weights, value estimates, and advantages are learned empirically. First test in a 10x10 MiniGrid environment with a goal-A reachability objective, a hazard-avoidance constraint, and a goal-B objective. Compare against monolithic PPO and PPO receiving only raw goal flags. Record both specification success probabilities, hazard violations, episodes to reach a constrained-return threshold, and gradient variance before versus after the first goal. Success would be fewer post-goal violations and faster learning of the second objective at equal environment steps, with no loss in final feasible return.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.