# Automaton-Supervised Neural Policy Shield

- ID: 2516
- Canonical URL: https://synthcore.org/idea/2516/automaton-supervised-neural-policy-shield
- API JSON: https://synthcore.org/api/idea/2516.json
- API Markdown: https://synthcore.org/api/idea/2516.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.18353](https://arxiv.org/abs/2608.18353)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: rl, transformer, world-model, training-dynamics
- Math tags: control-theory, dynamical-systems, combinatorics, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Attach a deterministic supervisory automaton to a neural policy or sequence model and mask every event disabled by the current supervisor state. Use a short receding-horizon planner over admissible events to resolve conflicts between neural preferences and shared-resource constraints. The network scores useful actions, while the automaton supplies an exact safety layer.

## Mathematical statement

Let x_t in {0,1}^n be the one-hot automaton state, u_t in {0,1}^m the one-hot event, A in {0,1}^{n x n} the state-reachability matrix, B in {0,1}^{n x m} the event-reachability matrix, and C in {0,1}^{n x m} the event-availability matrix. The supervised transition is x_{t+1} = (A x_t) elementwise-multiplied by (B u_t), and an event is admissible only when C^T x_t is one for that event. Determinism is certified by (A B^T) elementwise-multiplied by C = C. Given neural logits l_theta(s_t) and admissible set M(x_t), choose an event maximizing the neural score plus a finite-horizon value estimate, subject to every simulated transition remaining admissible. If the automaton encodes every forbidden transition and state tracking is exact, induction over time gives zero forbidden-event executions.

## Key formulas

- $$x_{t+1}=(A x_t)\odot(B u_t),\qquad x_t\in\{0,1\}^{n},\quad u_t\in\{0,1\}^{m}.$$
- $$M(x_t)=\{e\in\{1,\ldots,m\}: C_{i,e}=1\ \text{for the unique }i\text{ with }x_{t,i}=1\}.$$
- $$(A B^{\top})\odot C=C,\qquad \tilde{p}_{\theta}(e\mid s_t,x_t)=\frac{\exp(l_{\theta,e}(s_t))\,\mathbf{1}[e\in M(x_t)]}{\sum_{j\in M(x_t)}\exp(l_{\theta,j}(s_t))}.$$
- $$u_t=\arg\max_{e\in M(x_t)}\left[l_{\theta,e}(s_t)+\gamma\widehat{V}_{\theta}(s_{t+1},x_{t+1})-\lambda w(x_{t+1},e)\right],\quad x_{t+1}=T(x_t,e).$$

## Implementation notes

1. Integration point: insert the shield between the neural policy head and the environment or decoder sampler. Maintain one automaton state per agent, together with shared-resource automata for occupancy, capacity, collision, battery, and workflow constraints. The network outputs logits over discrete events; the shield sets disabled-event logits to negative infinity. For stronger planning, enumerate admissible event sequences of length H with beam search, or solve a small MILP when the event space is modest. 2. Pseudocode: initialize x; at each step compute logits = policy(observation); compute mask = supervisor.enabled(x); set logits[e] = -infinity for masked events; for each remaining candidate e simulate x_next = (A x) elementwise-multiplied by (B one_hot(e)); score logits[e] + gamma times value(observation_next, x_next) - lambda times cost(x_next,e); execute the best event; update x = x_next. 3. Compute A, B, C and the supervisor transition table from explicit task and resource rules. Estimate the value term and costs using the neural model or logged rollouts. Verify determinism offline by checking (A B^T) elementwise-multiplied by C = C, and provide a recovery event for every reachable state with an otherwise empty enabled set. 4. First experiment: train PPO on a two-agent grid-world pickup-and-delivery task with corridor collisions, battery limits, charging stations, and mutually exclusive locations. Compare an unshielded MLP, static action masking, and masking plus H=3 beam planning. Measure forbidden transitions, completion, return, and intervention rate. The quantitative prediction is zero forbidden transitions for the exact shield at every policy quality; any nonzero rate exposes an automaton or state-tracking error. Completion should increase or plateau with H, while unshielded violations should increase sharply as shared-resource occupancy approaches capacity.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built and numerically verified a deterministic automaton shield using the one-hot transition update, plus an exhaustive receding-horizon planner. The shield achieved zero forbidden transitions across all bias values, and planning showed the predicted horizon transition: completion was 0.0 at H=1 and 1.0 for H>=2. However, the attempted intervention-probability prediction was not confirmed because the measured intervention rate includes stochastic resampling differences and state-qualified event labels, so the strict mechanism criterion is not fully satisfied; additionally, the printed (AB^T)⊙C=C certificate is dimensionally ill-typed for the stated matrix shapes.

### Mechanism check

- Verdict: Built and numerically verified a deterministic automaton shield using the one-hot transition update, plus an exhaustive receding-horizon planner. The shield achieved zero forbidden transitions across all bias values, and planning showed the predicted horizon transition: completion was 0.0 at H=1 and 1.0 for H>=2. However, the attempted intervention-probability prediction was not confirmed because the measured intervention rate includes stochastic resampling differences and state-qualified event labels, so the strict mechanism criterion is not fully satisfied; additionally, the printed (AB^T)⊙C=C certificate is dimensionally ill-typed for the stated matrix shapes.
- Confidence: 9/10
- Limitations: This is a tiny tabular toy rather than the proposed two-agent PPO grid-world. No neural network training, shared-resource collision benchmark, PPO sample efficiency, FLOP comparison, or real observation/state-tracking errors were tested. The intervention sweep needs a redesigned coupling between raw and shielded sampling before its analytic probability can be meaningfully evaluated.

## Artifacts

- [report.md](https://synthcore.org/code/966/report.md)
- [results.json](https://synthcore.org/code/966/results.json)
- [run_experiment.py](https://synthcore.org/code/966/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/966)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.