# Cyclotomic p-Cap Layer

- ID: 431
- Canonical URL: https://synthcore.org/idea/431/cyclotomic-p-cap-layer
- API JSON: https://synthcore.org/api/idea/431.json
- API Markdown: https://synthcore.org/api/idea/431.md
- Verification status: unverified
- Source: [arXiv:2607.05360](https://arxiv.org/abs/2607.05360)
- Category: architecture
- Solves: accuracy, stability, sample-efficiency
- ML areas: attention, transformer, regularization
- Math tags: algebra, representation-theory, topology, linear-algebra
- Ratings: usefulness 5/10; difficulty 5/10; novelty 8/10

## Idea description

Replace an ordinary token aggregation step with a p-replica cyclic-equivariant block. Features are copied into p replicas, processed by shared operators, coupled through a cap-like bilinear interaction, and projected onto cyclic invariants. An auxiliary commutation loss enforces that applying the operator before or after the p-fold lift gives similar outputs.

## Mathematical statement

The paper defines the cap product as an action of a cohomology-like operator on a homology-like object, \(\cap: THC^{p}(A)\wedge THH_q(A)\to THH_{q-p}(A)\), where \(THC^p(A)\) is the degree-\(p\) topological Hochschild cohomology component and \(THH_q(A)\) is the degree-\(q\) homology component. Its cyclotomic construction uses the p-fold norm and the diagonal \(A^{\wedge p}\to\Phi^{C_p}N_e^{C_p}A^{\wedge p}\), where \(C_p\) is the cyclic group of order \(p\), \(N_e^{C_p}\) is the norm with cyclic permutation of factors, and \(\Phi^{C_p}\) denotes geometric fixed points. The central compatibility is represented by \(\Phi^{C_p}(\cap^p)\circ(\Delta\wedge\phi)\simeq\phi\circ\cap\): lifting both inputs and applying the p-fold cap operation agrees, up to the cyclotomic equivalence \(\phi\), with first applying the ordinary cap product and then lifting. For neural tensors, let \(x\in\mathbb{R}^{b\times n\times d}\) be token features, let \(\rho(r)\) cyclically shift replica indices by \(r\), let \(L_\theta\) be a shared feature operator, and let \(B_\psi\) be a learned bilinear-inspired cap map. Define \(P_{C_p}(z)=p^{-1}\sum_{r=0}^{p-1}\rho(r)z\). The finite-dimensional analogue is \(P_{C_p}B_\psi(L_\theta^{\otimes p}x)\approx B_\psi(L_\theta x)\), with the single-replica result broadcast across replicas before projection. The paper provides a naturality/commutation identity at the homotopy level, not a direct optimization or generalization bound, so the neural benefit is falsifiable rather than guaranteed.

## Key formulas

- $$\cap\colon THC^{p}(A)\wedge THH_{q}(A)\to THH_{q-p}(A).$$
- $$A^{\wedge p}\longrightarrow\Phi^{C_{p}}N_{e}^{C_{p}}A^{\wedge p}.$$
- $$\Phi^{C_{p}}(\cap^{p})\circ(\Delta\wedge\phi)\simeq\phi\circ\cap.$$
- $$P_{C_p}(z)=\frac{1}{p}\sum_{r=0}^{p-1}\rho(r)z,\qquad \mathcal{L}_p(x)=P_{C_p}\!\left(B_\psi\big(L_\theta x^{(0)},\ldots,L_\theta x^{(p-1)}\big)\right).$$

## Implementation notes

(1) Integration point: insert the block after a Transformer self-attention or MLP sublayer and before the residual addition. The input is token features \(x\in\mathbb{R}^{b\times n\times d}\). Choose a small prime \(p\in\{2,3,5\}\), create p replica features, and share all weights across replicas. Preserve sequence length and hidden width so the block can replace a lightweight feed-forward or aggregation sublayer. (2) Pseudocode: `xrep = stack([x for r in range(p)], dim=replica)`; compute `z[r] = L_theta(xrep[r])`; compute `u[r] = B_psi(z[r], xrep[(r+1) % p])`; compute `u_inv = mean(cyclic_shift(u, r) for r in range(p))`; return `y = x + gate * u_inv`. Here `L_theta` is a shared linear-GELU-linear map, `B_psi(a,b)=W_o GELU(W_a a + W_b b)` is a cheap cap-like interaction, and `cyclic_shift` implements the \(C_p\) action on replicas. Also compute `u_single = B_psi(L_theta(x), x)` and add `lambda_cap * mean_squared_error(mean(u), u_single)` to the task loss. (3) Computed from the paper's mathematics: p-fold replication, cyclic permutation, invariant projection, and the cap-before/after consistency relation. Estimated empirically: the best p, gate initialization, loss coefficient \(\lambda_{cap}\), and whether replica-specific dropout or token offsets help. The exact homotopy fixed-point functor is not implemented; finite-dimensional group averaging is its explicit surrogate. (4) First cheap experiment: train a 2-layer, 4-head Transformer on a small character-level Penn Treebank language-modeling task or CIFAR-10 image patches. Compare a baseline MLP sublayer, ordinary mean pooling, and this block at matched parameter count and approximately matched FLOPs. Start with \(p=3\), gate initialized to 0.1, and \(\lambda_{cap}\in\{0,10^{-3},10^{-2},10^{-1}\}\). Measure validation perplexity or accuracy, robustness under cyclic shifts of replica/token order, gradient-norm variance, and loss after equal training steps. A successful signal is improved cyclic-shift consistency together with lower validation loss at equal FLOPs, without gradient explosions or slower convergence.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.