# Explicit fold residual block

- ID: 287
- Canonical URL: https://synthcore.org/idea/287/explicit-fold-residual-block
- API JSON: https://synthcore.org/api/idea/287.json
- API Markdown: https://synthcore.org/api/idea/287.md
- Verification status: unverified
- Source: [arXiv:2606.31856](https://arxiv.org/abs/2606.31856)
- Category: architecture
- Solves: accuracy, scalability
- ML areas: mlp, architecture, transformer
- Math tags: topology, geometry, dynamical-systems
- Ratings: usefulness 7/10; difficulty 3/10; novelty 5/10

## Idea description

Replace part of a narrow ReLU MLP or residual branch with an explicit coordinatewise fold that computes absolute value in one residual block. The fold introduces controlled non-injectivity without increasing representation width, potentially allowing width-constrained networks to represent multimodal or topologically folded decision boundaries that monotone feedforward layers represent inefficiently.

## Mathematical statement

The constructive identity used by the paper is $|x|=x+2\operatorname{ReLU}(-x)$. For a vector $x\in\mathbb{R}^{d}$, with coordinatewise ReLU, a residual map $F(x)=x+G(x)$ and residual branch $G(x)=2\operatorname{ReLU}(-x)$ gives $F(x)=|x|$ coordinatewise. A translated fold around a learned or fixed center $c\in\mathbb{R}^{d}$ is $F_c(x)=c+|x-c|=x+2\operatorname{ReLU}(c-x)$, where $c-x$ and the ReLU are coordinatewise. Repeated folds can create non-injective maps and can change linking configurations; this is impossible for a continuous invertible flow, whose layer maps are diffeomorphisms and preserve topological invariants. The transferable property is therefore an inexpensive, explicitly parameterized violation of injectivity, not merely the use of ReLU.

## Key formulas

- $$|x|=x+2\operatorname{ReLU}(-x)$$
- $$F(x)=x+G(x),\qquad G(x)=2\operatorname{ReLU}(-x)\quad\Longrightarrow\quad F(x)=|x|$$
- $$F_c(x)=c+|x-c|=x+2\operatorname{ReLU}(c-x),\qquad x,c\in\mathbb{R}^{d}$$

## Implementation notes

Integrate this at the residual-block level of a width-constrained MLP, small CNN bottleneck, or tokenwise transformer feed-forward sublayer. The input tensor is $X\in\mathbb{R}^{B\times T\times d}$; apply the fold independently to the final representation dimension, optionally after LayerNorm. Use either a learned center $c\in\mathbb{R}^{d}$ shared across tokens or a center generated by a small linear projection from the block input. A minimal block is: (1) $U=X-c$; (2) $V=\operatorname{ReLU}(-U)$; (3) $Y=X+2V$, which is exactly $c+|X-c|$. For a more flexible residual block, use $Y=X+\gamma W_o\operatorname{ReLU}(W_i(X-c)+b_i)+b_o$, initialize $W_i=-I$, $W_o=2I$, and $\gamma=1$, then allow fine-tuning while retaining the identity skip connection. Pseudocode is `U = X - c; R = relu(-U); Y = X + 2*R`; followed by the usual output projection, normalization, and dropout. The mathematical component is the exact fold initialization and its non-injectivity; learned deviations are empirical. Compare against a same-width plain ReLU MLP, a standard pre-activation ResNet block, and an invertible coupling block. First run a cheap synthetic test in $d=3$: classify two interleaved or folded curve/surface classes after projecting inputs through a width-3 bottleneck, then test CIFAR-10 with width-8 and width-16 MLP bottlenecks. Track accuracy at equal parameters, layers needed to reach a target accuracy, Jacobian rank-collapse frequency, and optimization loss. Success means the explicit fold reaches the same accuracy with fewer layers or improves fixed-width accuracy without an unstable increase in gradient norms.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.