# Companion Observer Memory for Neural Policies

- ID: 2527
- Canonical URL: https://synthcore.org/idea/2527/companion-observer-memory-for-neural-policies
- API JSON: https://synthcore.org/api/idea/2527.json
- API Markdown: https://synthcore.org/api/idea/2527.md
- Verification status: unverified
- Source: [arXiv:2608.18452](https://arxiv.org/abs/2608.18452)
- Category: architecture
- Solves: memory, stability, sample-efficiency
- ML areas: rnn, ssm, rl, world-model
- Math tags: control-theory, dynamical-systems, linear-algebra, representation-theory
- Ratings: usefulness 5/10; difficulty 4/10; novelty 6/10

## Idea description

Replace an unrestricted GRU or attention-based history encoder with a fixed companion-form shift register driven by the current action and observation, followed by a learned nonlinear policy. The register stores a structured finite history, while a learned matrix or MLP readout maps that history to a control-relevant latent state. This should provide a cheaper and more interpretable memory mechanism for partially observed environments, especially when the relevant dynamics are approximately low-order linear around the operating regime.

## Mathematical statement

The paper constructs a lifted memory state \(\zeta(t)\in\mathbb{R}^{n(m+p)}\) by concatenating companion-form states \(\zeta^i(t)\) for all \(m\) input channels and \(p\) output channels. Each companion state evolves with \(\mathcal A\in\mathbb{R}^{n\times n}\), whose last row is \((-\alpha_0,-\alpha_1,\ldots,-\alpha_{n-1})\), and with input vector \(b=(0,\ldots,0,1)^T\). The lifted dynamics are \(\zeta(t+1)=(I_{m+p}\otimes\mathcal A)\zeta(t)+[I_m\otimes b;\mathbf{0}]u(t)+[\mathbf{0};I_p\otimes b]y(t)\), where \(u(t)\in\mathbb{R}^m\) is the action and \(y(t)\in\mathbb{R}^p\) is the observation. The paper defines a readout \(\hat x(t)=M\zeta(t)\), with \(M=[M_1,\ldots,M_{m+p}]\), and establishes \(M(I_{m+p}\otimes\mathcal A)=(A-LC)M\) and \(M[I_m\otimes b;\mathbf{0}]=B\), where \(A,B,C\) are unknown system matrices and \(L\) is an observer gain. For the neural adaptation, choose \(\mathcal A\) with spectral radius \(\rho<1\), learn \(M\), and use \(h_t=M\zeta_t\) as the policy input. The fixed stable spectrum supplies bounded memory dynamics, while the learned readout extracts task-relevant state information.

## Key formulas

- $$\zeta(t+1)=(I_{m+p}\otimes\mathcal{A})\zeta(t)+\begin{bmatrix}I_{m}\otimes b\\ {\bf 0}\end{bmatrix}u(t)+\begin{bmatrix}{\bf 0}\\ I_{p}\otimes b\end{bmatrix}y(t)$$
- $$M(I_{m+p}\otimes\mathcal{A})=(A-LC)M$$
- $$M\begin{bmatrix}I_{m}\otimes b\\ {\bf 0}\end{bmatrix}=B$$
- $$h_t=M\zeta_t,\qquad u_t=\pi_\theta(h_t),\qquad \zeta_{t+1}=F_\mathcal{A}\zeta_t+G_u u_t+G_y y_t$$

## Implementation notes

Integrate the module between the environment observation and the policy/value network. Let the observation dimension be p, action dimension be m, and choose memory order n, initially 4, 8, or 16. Build a block-diagonal fixed matrix F_A=I_{m+p}\otimes Acal, where Acal is the companion matrix with last row -alpha_0,...,-alpha_{n-1}. Select the coefficients so every eigenvalue has modulus rho<1; a simple choice is a stable polynomial corresponding to repeated or real poles near rho=0.8. Construct G_u=[I_m\otimes b;0] and G_y=[0;I_p\otimes b]. Maintain zeta separately for every parallel environment. At each step, receive y_t, compute h_t=M zeta_t, concatenate h_t with y_t if desired, and produce u_t=pi_theta(h_t,y_t). After executing u_t and receiving y_{t+1}, update zeta_{t+1}=F_A zeta_t+G_u u_t+G_y y_{t+1}. Learn M jointly with policy parameters theta using PPO, SAC, or imitation learning. If privileged simulator states x_t are available, add L_obs=(1/T)sum_t ||M zeta_t-x_t||_2^2; otherwise train only from task return or prediction loss. Apply gradient clipping and optionally penalize ||M||_F^2. The fixed matrices are computed exactly from the construction; M, theta, and any observation encoder are estimated empirically. First test on partially observed CartPole, DeepMind Control with hidden velocity, and a small visual POMDP. Compare against a same-size GRU and a transformer over the last n observations/actions. Record return, samples to target return, wall-clock training cost, hidden-state norm, and robustness to longer episodes. Success is comparable or better return with lower recurrent compute, faster early learning, and bounded hidden-state norms as rho remains below one.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.