# Signed-base fractal positional features

- ID: 1590
- Canonical URL: https://synthcore.org/idea/1590/signed-base-fractal-positional-features
- API JSON: https://synthcore.org/api/idea/1590.json
- API Markdown: https://synthcore.org/api/idea/1590.md
- Verification status: unverified
- Source: [arXiv:2607.29327](https://arxiv.org/abs/2607.29327)
- Category: architecture
- Solves: accuracy, sample-efficiency
- ML areas: transformer, attention, embedding
- Math tags: probability, dynamical-systems, measure-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Augment standard Transformer positional embeddings with coordinates generated by the paper's signed-base digit expansion. Previous binary digits determine the sign and scale of later contributions, while a two-state Markov chain controls correlations between digits. This supplies multiscale positional structure using a small number of transition and base parameters.

## Mathematical statement

The paper sets g_0 in [1/2,1) and g_1=g_0-1, and maps binary digits a_1,a_2,... to x=sum over k of a_k g_(1-a_k) times the product over i<k of g_(a_i). Here a_k is the kth binary digit, g_(1-a_k) is its additive coefficient, and the product is the signed contraction induced by previous digits. Theorem 1 states that every x in [0,g_0] has at least one such representation. At g_0=1/2, the expansion becomes x=a_1/2+sum over k>=2 of a_k(-1)^(sigma_k)/2^k, where sigma_k=sum over i<k of a_i; thus every one digit flips the sign of all later scales. A Markov digit source uses p_uv=P(a_(k+1)=v given a_k=u), for u,v in {0,1}. The transferable asset is an explicit bounded multiscale coordinate with controllable digit correlations.

## Key formulas

- $$x=\delta_{a_1}+\sum_{k=2}^{\infty}\left(\delta_{a_k}\prod_{j=1}^{k-1}g_{a_j}\right),\qquad \delta_{a_k}=a_k g_{1-a_k},\qquad g_1=g_0-1$$
- $$x=\frac{a_1}{2}+\sum_{k=2}^{\infty}\frac{a_k(-1)^{\sigma_k}}{2^k},\qquad \sigma_k=\sum_{i=1}^{k-1}a_i\qquad (g_0=1/2)$$
- $$P(a_{k+1}=v\mid a_k=u)=p_{uv},\qquad \sum_{v\in\{0,1\}}p_{uv}=1$$
- $$\phi_{m,r}(t)=\left[\sin(2\pi b_m x_{m,r}(t)),\cos(2\pi b_m x_{m,r}(t))\right]$$

## Implementation notes

Integrate this at the Transformer input, either by replacing learned absolute position embeddings or by adding a residual fractal positional branch. For each feature channel m, choose g0[m] in [0.5,0.95], depth K, initial digit probability pi[m], transition matrix P[m], and frequency b[m]. Generate one digit sequence for every position t and channel m, compute the truncated recurrence, then project sine/cosine features into d_model dimensions. Pseudocode: initialize a[1] from Bernoulli(pi); set x=a[1]*g_(1-a[1]) and prod=g_(a[1]); for k=2,...,K sample a[k] from softmax(P[a[k-1],:]), set x += a[k]*g_(1-a[k])*prod, and set prod *= g_(a[k]); concatenate sin(2*pi*b*x) and cos(2*pi*b*x) across m and apply a learned linear projection. The recurrence, bounded coordinate, and Markov parameterization come from the paper; P, pi, g0, frequencies, and projection weights are tuned or learned. Start with fixed deterministic digits for reproducibility, then test resampling as augmentation. Train a six-layer Transformer on WikiText-2 at sequence lengths 128 and 512, comparing learned and sinusoidal positions at equal FLOPs. Test iid digits against p00=p11=0.9. Success means lower validation perplexity, faster loss descent, or better length extrapolation without added attention cost.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.