Unverified Re-invented 2026

Explicit fold residual block

Implementation & benchmark of arXiv:2606.31856 — Low-dimensional topology of deep neural networks

Usefulness7/10
Difficulty3/10
Novelty5/10

Source paper: Low-dimensional topology of deep neural networks arXiv:2606.31856 · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace part of a narrow ReLU MLP or residual branch with an explicit coordinatewise fold that computes absolute value in one residual block. The fold introduces controlled non-injectivity without increasing representation width, potentially allowing width-constrained networks to represent multimodal or topologically folded decision boundaries that monotone feedforward layers represent inefficiently.

Formulas

$$|x|=x+2\operatorname{ReLU}(-x)$$
$$F(x)=x+G(x),\qquad G(x)=2\operatorname{ReLU}(-x)\quad\Longrightarrow\quad F(x)=|x|$$
$$F_c(x)=c+|x-c|=x+2\operatorname{ReLU}(c-x),\qquad x,c\in\mathbb{R}^{d}$$

Mathematical statement

The constructive identity used by the paper is $|x|=x+2\operatorname{ReLU}(-x)$. For a vector $x\in\mathbb{R}^{d}$, with coordinatewise ReLU, a residual map $F(x)=x+G(x)$ and residual branch $G(x)=2\operatorname{ReLU}(-x)$ gives $F(x)=|x|$ coordinatewise. A translated fold around a learned or fixed center $c\in\mathbb{R}^{d}$ is $F_c(x)=c+|x-c|=x+2\operatorname{ReLU}(c-x)$, where $c-x$ and the ReLU are coordinatewise. Repeated folds can create non-injective maps and can change linking configurations; this is impossible for a continuous invertible flow, whose layer maps are diffeomorphisms and preserve topological invariants. The transferable property is therefore an inexpensive, explicitly parameterized violation of injectivity, not merely the use of ReLU.

Implementation notes

Integrate this at the residual-block level of a width-constrained MLP, small CNN bottleneck, or tokenwise transformer feed-forward sublayer. The input tensor is $X\in\mathbb{R}^{B\times T\times d}$; apply the fold independently to the final representation dimension, optionally after LayerNorm. Use either a learned center $c\in\mathbb{R}^{d}$ shared across tokens or a center generated by a small linear projection from the block input. A minimal block is: (1) $U=X-c$; (2) $V=\operatorname{ReLU}(-U)$; (3) $Y=X+2V$, which is exactly $c+|X-c|$. For a more flexible residual block, use $Y=X+\gamma W_o\operatorname{ReLU}(W_i(X-c)+b_i)+b_o$, initialize $W_i=-I$, $W_o=2I$, and $\gamma=1$, then allow fine-tuning while retaining the identity skip connection. Pseudocode is U = X - c; R = relu(-U); Y = X + 2*R; followed by the usual output projection, normalization, and dropout. The mathematical component is the exact fold initialization and its non-injectivity; learned deviations are empirical. Compare against a same-width plain ReLU MLP, a standard pre-activation ResNet block, and an invertible coupling block. First run a cheap synthetic test in $d=3$: classify two interleaved or folded curve/surface classes after projecting inputs through a width-3 bottleneck, then test CIFAR-10 with width-8 and width-16 MLP bottlenecks. Track accuracy at equal parameters, layers needed to reach a target accuracy, Jacobian rank-collapse frequency, and optimization loss. Success means the explicit fold reaches the same accuracy with fewer layers or improves fixed-width accuracy without an unstable increase in gradient norms.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.