# Inverted recurrent surprise layers

- ID: 1845
- Canonical URL: https://synthcore.org/idea/1845/inverted-recurrent-surprise-layers
- API JSON: https://synthcore.org/api/idea/1845.json
- API Markdown: https://synthcore.org/api/idea/1845.md
- Verification status: unverified
- Source: [arXiv:2608.05481](https://arxiv.org/abs/2608.05481)
- Category: training
- Solves: memory, sample-efficiency, stability
- ML areas: training, mlp, embedding, fine-tuning
- Math tags: dynamical-systems, optimization, representation-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace globally backpropagated hidden-layer losses with independent recurrent layers that receive bottom-up, top-down, and lateral inputs. Train each layer to assign low activity to correctly paired input-label examples and high activity to mismatched examples, then classify by selecting the label with the lowest accumulated surprise.

## Mathematical statement

The paper defines a local objective of the form $\mathcal{L}_{\textrm{layer}}=\sigma\left((-1)^{\eta}[\cdots]\right)$, where $\sigma(z)=1/(1+e^{-z})$ is the logistic function and $\eta$ indicates whether a sample is positive or negative. We make the omitted scalar explicit using the standard layer goodness $g_l=\|x_l\|_2^2/d_l$, where $x_l\in\mathbb{R}^{d_l}$ is the activation of layer $l$ and $d_l$ is its width, and a learnable threshold $\theta_l$. Let $\eta=0$ denote a correctly paired positive example and $\eta=1$ a mismatched negative example. Minimizing $\mathcal{L}_l=\sigma((-1)^{\eta}(g_l-\theta_l))$ pushes $g_l<\theta_l$ for positives and $g_l>\theta_l$ for negatives. The recurrent state is $x_l^t=\phi(W_l^{\downarrow}\operatorname{sg}(x_{l-1}^t)+W_l^{\uparrow}\operatorname{sg}(x_{l+1}^{t-1})+R_lx_l^{t-1}+b_l)$, where $t$ is the recurrent iteration, $\phi$ is a bounded activation such as GELU followed by normalization, $W_l^{\downarrow}$, $W_l^{\uparrow}$, and $R_l$ are bottom-up, top-down, and lateral weights, and $\operatorname{sg}$ stops gradients between layers. The total candidate-label surprise is $S(y|u)=\sum_l\sum_{t=1}^{T}g_l^t(u,y)$ for input $u$ and candidate label $y$; predict $\arg\min_y S(y|u)$.

## Key formulas

- $$\mathcal{L}_{\textrm{layer}}=\sigma\left((-1)^{\eta}\left[g_l-\theta_l\right]\right),\qquad \sigma(z)=\frac{1}{1+e^{-z}},\qquad g_l=\frac{1}{d_l}\lVert x_l\rVert_2^2.$$
- $$x_l^t=\phi\!\left(W_l^{\downarrow}\operatorname{sg}(x_{l-1}^t)+W_l^{\uparrow}\operatorname{sg}(x_{l+1}^{t-1})+R_lx_l^{t-1}+b_l\right).$$
- $$S(y\mid u)=\sum_{l=1}^{L}\sum_{t=1}^{T}g_l^t(u,y),\qquad \hat y=\arg\min_{y\in\mathcal{Y}}S(y\mid u).$$

## Implementation notes

Integrate this at the hidden blocks of a small MLP or compact vision transformer used as a classifier. For an input minibatch $u$ and its labels $y$, construct positive pairs $(u,y)$ and negative pairs $(u,\tilde y)$ by independently permuting labels within the minibatch; ensure the permutation has no fixed points. Encode the input and candidate label into the first layer, then run $T$ recurrent updates. At update $t$, compute each layer in order using its current bottom-up input, the previous-step top-down state, and its own previous state: `x[l] = phi(Wdown[l] @ stopgrad(x[l-1]) + Wup[l] @ stopgrad(x_prev[l+1]) + R[l] @ x_prev[l] + b[l])`. Compute `g[l] = mean(x[l]**2)` and accumulate `loss += sigmoid((2*eta-1) * (g[l]-theta[l]))`; here `eta=0` for positives and `eta=1` for negatives, so minimizing the loss lowers positive goodness and raises negative goodness. Update only the parameters of layer `l` from its local loss, with AdamW or SGD; detach all neighboring activations before computing that layer's gradient. Optionally learn thresholds using an exponential moving median of goodness, or initialize each `theta[l]` to the midpoint between positive and negative running goodness means. The paper contributes the polarity of the local objective and the three-way recurrent integration; recurrent weights, normalization, and the candidate-label encoder are engineering choices. At evaluation, run the recurrent dynamics separately for each candidate label and choose the label with minimum summed goodness. First test on MNIST or CIFAR-10 with a 3-layer width-256 MLP, $T=3$, and compare against the same architecture trained by backpropagation and ordinary Forward-Forward. Measure accuracy, peak activation memory, wall-clock time, and accuracy after freezing lower layers. Success means substantially lower activation memory and competitive accuracy, plus stable positive-versus-negative goodness separation throughout training.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.