# Hard-Constrained Bernstein Memory Head

- ID: 728
- Canonical URL: https://synthcore.org/idea/728/hard-constrained-bernstein-memory-head
- API JSON: https://synthcore.org/api/idea/728.json
- API Markdown: https://synthcore.org/api/idea/728.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.11110](https://arxiv.org/abs/2607.11110)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: transformer, ssm, attention, memory
- Math tags: approximation-theory, convex-analysis, numerical-analysis
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

Add a causal memory branch whose lag-response function is represented by a Bernstein polynomial with coefficients constrained to produce a nonnegative, decreasing, convex kernel. The branch aggregates past hidden states using this kernel, giving the model a learnable long-memory profile while preventing oscillatory, negative, or increasing historical influence.

## Mathematical statement

The paper formulates an integro-differential operator containing a causal memory kernel, $K(t-s,\mathbf{x}-\mathbf{y})$, through $\int_0^t\int_{\Omega}K(t-s,\mathbf{x}-\mathbf{y})\mathcal{F}[u](s,\mathbf{y})\,d\mathbf{y}\,ds$. Here $t$ is the current time, $s$ is a past time, $\mathbf{x},\mathbf{y}\in\Omega$ are spatial locations, $u$ is the observed field, $\mathcal{F}[u]$ is the quantity filtered by the kernel, and $K$ is the unknown causal kernel. The MC-KAN construction described in the paper uses a Bernstein polynomial $k(z)=\sum_{j=0}^{m}c_j B_{j,m}(z)$ on $z\in[0,1]$, where $B_{j,m}(z)=\binom{m}{j}z^j(1-z)^{m-j}$, $m$ is the polynomial degree, and $c_j$ are trainable coefficients. Because Bernstein basis functions are nonnegative and sum to one, $c_j\geq 0$ implies $k(z)\geq0$; $c_j\geq c_{j+1}$ implies monotone decrease; and $c_j-2c_{j+1}+c_{j+2}\geq0$ implies convexity. These conditions follow from the Bernstein derivative identities $k'(z)=m\sum_{j=0}^{m-1}(c_{j+1}-c_j)B_{j,m-1}(z)$ and $k''(z)=m(m-1)\sum_{j=0}^{m-2}(c_j-2c_{j+1}+c_{j+2})B_{j,m-2}(z)$. For a neural sequence model, $z_r=\min(r\Delta/\tau,1)$ is the normalized lag for lag index $r$, timestep $\Delta$, and memory horizon $\tau$; the memory output is $m_t=\sum_{r=1}^{L}k(z_r)v_{t-r}$, where $v_{t-r}\in\mathbb{R}^d$ is a past value vector and $L$ is the retained context length.

## Key formulas

- $$\mathcal{T}[u](t,\mathbf{x})=\mathcal{N}[u](t,\mathbf{x})+\int_{0}^{t}\int_{\Omega}K(t-s,\mathbf{x}-\mathbf{y})\,\mathcal{F}[u](s,\mathbf{y})\,d\mathbf{y}\,ds+S(t,\mathbf{x})$$
- $$k(z)=\sum_{j=0}^{m}c_j B_{j,m}(z),\qquad B_{j,m}(z)=\binom{m}{j}z^j(1-z)^{m-j},\qquad z\in[0,1]$$
- $$c_j\geq0,\qquad c_j-c_{j+1}\geq0,\qquad c_j-2c_{j+1}+c_{j+2}\geq0$$
- $$m_t=\sum_{r=1}^{L}\frac{k\!\left(\min(r\Delta/\tau,1)\right)}{\sum_{q=1}^{L}k\!\left(\min(q\Delta/\tau,1)\right)+\varepsilon}\,v_{t-r},\qquad h_t^{\mathrm{out}}=h_t+g\,W_m m_t$$

## Implementation notes

(1) Integration point: insert the module after the value projection in each Transformer block, or beside the state update in an SSM. For hidden states $h_{1:T}\in\mathbb{R}^{T\times d}$, compute values $v_t=W_vh_t$, construct the causal memory vector $m_t$, and add it through a learned projection and scalar gate as $h_t^{\mathrm{out}}=h_t+gW_mm_t$. Start with one shared kernel per layer; optionally use one kernel per attention head.

(2) Pseudocode:
```text
input h[1:T]
v = h @ Wv
raw = softplus(theta[m+1])
q = raw
for j = m-1 down to 0:
    d[j] = sum(q[j:m])
c[m] = softplus(a)
for j = m-1 down to 0:
    c[j] = c[j+1] + d[j]
for z in lag_grid:
    k[z] = sum_j c[j] * binom(m,j) * z**j * (1-z)**(m-j)
w = k(lag_grid) / (sum(k(lag_grid)) + eps)
for t in 1..T:
    memory[t] = sum_{r=1}^{min(L,t-1)} w[r] * v[t-r]
out = h + sigmoid(g) * (memory @ Wm)
```
The nested cumulative sums enforce positivity, monotone decrease, and convexity without penalty losses. Use `torch.flip` and cumulative sums to vectorize coefficient construction, and use a causal depthwise convolution or FFT convolution for long contexts.

(3) Computed from the paper's mathematics: the Bernstein basis, coefficient-difference constraints, and the causal kernel convolution. Estimated empirically: the degree $m$, memory horizon $\tau$, gate $g$, projection matrices, and whether one shared kernel or per-head kernels are preferable. If $\tau$ is learned, parameterize it as `softplus(tau_raw)` and clip it to a practical range. Monitor the discrete first differences $k_r-k_{r+1}$ and second differences $k_r-2k_{r+1}+k_{r+2}$ to verify the implementation.

(4) First cheap experiment: train a 4-layer, 8-head Transformer on synthetic autoregressive sequences generated by a known mixture of exponential and power-law memory, with random input corruption and only 10-20% observed training tokens. Compare the constrained head against standard attention, an unconstrained learned depthwise lag convolution with the same number of parameters, and a single exponential kernel. Evaluate validation MSE, stability on sequences 4x longer than training, and sensitivity to noise. Success is lower extrapolation error and fewer exploding or oscillating runs at equal FLOPs, while the learned kernel shape remains stable across random seeds.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a hard-constrained Bernstein memory kernel with coefficients parameterized to enforce positivity, monotone decrease, and convexity, plus normalized causal lag aggregation. All three mechanism predictions were confirmed: zero shape violations across 100 random parameterizations, first- and second-difference margins scaled with observed slope 1.0000000000000007 versus predicted 1.0, and the saturation lag exactly matched ceil(tau/dt) for all five tested horizons. In the small regression test, the constrained model achieved MSE 0.03887 versus 0.04552 for an unconstrained ridge lag model, but this comparison is favorable partly because the constrained model uses only 7 parameters versus 32 and does not establish a Transformer-level win.

### Mechanism check

- Verdict: Built a hard-constrained Bernstein memory kernel with coefficients parameterized to enforce positivity, monotone decrease, and convexity, plus normalized causal lag aggregation. All three mechanism predictions were confirmed: zero shape violations across 100 random parameterizations, first- and second-difference margins scaled with observed slope 1.0000000000000007 versus predicted 1.0, and the saturation lag exactly matched ceil(tau/dt) for all five tested horizons. In the small regression test, the constrained model achieved MSE 0.03887 versus 0.04552 for an unconstrained ridge lag model, but this comparison is favorable partly because the constrained model uses only 7 parameters versus 32 and does not establish a Transformer-level win.
- Confidence: 9/10
- Limitations: Did not test a Transformer or SSM, long-sequence extrapolation, multiple random training seeds, equal-parameter baseline, learned tau, or FLOP-normalized training. The toy comparison uses a fixed horizon and a linear regression task; CUDA was used for the constrained fit.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.55595
- Paired wins: 6/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/308/bench_report.json)
- [experiment.py](https://synthcore.org/code/308/experiment.py)
- [report.md](https://synthcore.org/code/308/report.md)
- [report_bench_2026-08-31T004750.md](https://synthcore.org/code/308/report_bench_2026-08-31T004750.md)
- [results.json](https://synthcore.org/code/308/results.json)
- [stage2_bernstein_bench.py](https://synthcore.org/code/308/stage2_bernstein_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/308)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.