# Orthogonal-Rank Contextual Memory

- ID: 309
- Canonical URL: https://synthcore.org/idea/309/orthogonal-rank-contextual-memory
- API JSON: https://synthcore.org/api/idea/309.json
- API Markdown: https://synthcore.org/api/idea/309.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.00507](https://arxiv.org/abs/2607.00507)
- Category: architecture
- Solves: memory, scalability, accuracy
- ML areas: rnn, ssm, graph-nn, embedding
- Math tags: graph-theory, geometry, linear-algebra, combinatorics
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Replace a discrete or one-hot recurrent state table with a low-dimensional vector memory whose event embeddings are orthogonal whenever the corresponding events are mutually exclusive in an input exclusivity graph. The module uses continuous state vectors and can therefore target dimension \(d=\xi(G)\), whereas a discrete state encoding is lower-bounded by \(N\geq\chi(G)\). This should be tested on graph-defined formal-language recognition tasks, where the graph is known and the claimed advantage can be measured directly.

## Mathematical statement

The paper represents each graph vertex \(v_i\in V\) by a rank-1 projector \(\hat P_i=|v_i\rangle\langle v_i|\), with adjacent vertices mapped to orthogonal projectors. Here \(G=(V,E)\) is an exclusivity graph, \(E\) contains pairs of mutually exclusive events, and \(|v_i\rangle\in\mathbb{R}^d\) or \(\mathbb{C}^d\) is a unit vector. The orthogonal rank is \(\xi(G)=\min\{d:\exists\{|v_i\rangle\}_{i\in V},\ \|v_i\|_2=1,\ \langle v_i,v_j\rangle=0\ \forall(i,j)\in E\}\). The classical memory requirement is \(N\geq\chi(G)\), where \(\chi(G)\) is the minimum number of colors assigning different colors to adjacent vertices; the quantum-style memory dimension is \(d=\xi(G)\). The transferable property is that dense incompatibility constraints can require many discrete colors while admitting a much smaller continuous orthogonal representation. In a neural implementation, exact orthogonality is relaxed with an edge penalty, while the recurrent state update learns how to move between event vectors.

## Key formulas

- $$\hat{P}_{i}=\ket{v_i}\bra{v_i},\qquad (i,j)\in E\ \Longrightarrow\ \hat{P}_{i}\hat{P}_{j}=0\ \Longleftrightarrow\ \langle v_i,v_j\rangle=0.$$
- $$\xi(G)=\min\left\{d:\exists\ \{|v_i\rangle\in\mathbb{R}^{d}\}_{i\in V},\ \|v_i\|_2=1,\ \langle v_i,v_j\rangle=0\ \forall(i,j)\in E\right\}.$$
- $$N\geq\chi(G),\qquad d=\xi(G).$$
- $$\mathcal{L}=\mathcal{L}_{\mathrm{task}}+\lambda_{\mathrm{orth}}\sum_{(i,j)\in E}\langle v_i,v_j\rangle^2+\lambda_{\mathrm{norm}}\sum_{i\in V}(\|v_i\|_2^2-1)^2.$$

## Implementation notes

1. Integration point: implement a GRU-like recurrent recognizer for graph-generated symbolic sequences. Let the alphabet contain one token per graph vertex, let \(e_i\in\mathbb{R}^d\) be a learned event embedding, and maintain hidden state \(h_t\in\mathbb{R}^d\). Normalize embeddings by \(v_i=e_i/(\|e_i\|_2+\epsilon)\). Feed the token and previous state into a gated transition, using the same dimension \(d\) for the hidden memory rather than a one-hot state table. 2. Pseudocode: initialize \(e_i\) and recurrent parameters; for each minibatch sequence \(x_{1:T}\), compute \(v_i=e_i/(\|e_i\|_2+\epsilon)\), set \(h_0=0\), and for \(t=1,\ldots,T\) compute \(h_t=\mathrm{GRU}(v_{x_t},h_{t-1})\), then predict the accept/reject label from \(h_t\). Add \(\lambda_{\mathrm{orth}}\sum_{(i,j)\in E}(v_i^\top v_j)^2\) and \(\lambda_{\mathrm{norm}}\sum_i(\|e_i\|_2^2-1)^2\) to cross-entropy. Optionally project the hidden state after each step onto the span of active event vectors. 3. Mathematics versus estimation: compute \(\chi(G)\) for small graphs with an integer-programming coloring solver. Estimate \(\xi(G)\) by optimizing the orthogonality objective over random embeddings for candidate dimensions \(d=1,2,\ldots\); the smallest dimension with near-zero edge violation is the empirical orthogonal rank. The network does not receive these values; they determine the width sweep and predicted classical/continuous memory comparison. Use real vectors first, then test complex-valued embeddings only if needed. 4. First experiment: generate promise languages from graph walks, where a sequence is a valid walk under graph-dependent transitions and the label depends on whether its final event is compatible with a designated start context. Compare a discrete-state RNN with width \(\chi(G)\), a standard GRU with width \(\xi(G)\), and the same GRU without the orthogonality penalty. Use 5--10 small graphs, including odd cycles and Boolean-orthogonality-derived graphs, sequence lengths 32--256, and equal parameter/FLOP budgets. Success means matching accuracy with substantially smaller hidden dimension, retaining accuracy on longer sequences, and showing lower validation loss or fewer parameters at matched accuracy. Report the Gram edge violation \(\max_{(i,j)\in E}|v_i^\top v_j|\) to verify that gains come from the intended structure rather than ordinary embedding capacity.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a reproducible orthogonal-rank memory MVP and verified the core graph claim: the 13-vertex graph has chi(G)=4 while an explicit 3D real orthogonal representation exists, with a 3-clique proving xi(G)>=3. The width-3 GRU used 35% fewer parameters than width 4 and matched its toy-task accuracy (72.3%), while the orthogonality penalty reduced edge violation from 0.99 to 0.007. However, the task was weak and the penalty did not improve accuracy or loss, so this is evidence for structural compression rather than a demonstrated neural performance win.

### Mechanism check

- Verdict: Built a reproducible orthogonal-rank memory MVP and verified the core graph claim: the 13-vertex graph has chi(G)=4 while an explicit 3D real orthogonal representation exists, with a 3-clique proving xi(G)>=3. The width-3 GRU used 35% fewer parameters than width 4 and matched its toy-task accuracy (72.3%), while the orthogonality penalty reduced edge violation from 0.99 to 0.007. However, the task was weak and the penalty did not improve accuracy or loss, so this is evidence for structural compression rather than a demonstrated neural performance win.
- Confidence: 7/10
- Limitations: Only one 13-vertex graph and one simple compatibility task were tested; the task has class imbalance and observed accuracy is close to the majority-class baseline. No longer sequence sweep, multiple seeds, FLOP-matched training, exact integer-programming coloring, or complex-valued representations were tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0169
- Paired wins: 1/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_orthogonal_memory.py](https://synthcore.org/code/1263/bench_orthogonal_memory.py)
- [bench_report.json](https://synthcore.org/code/1263/bench_report.json)
- [experiment.py](https://synthcore.org/code/1263/experiment.py)
- [report.md](https://synthcore.org/code/1263/report.md)
- [report_bench_2026-09-03T233717.md](https://synthcore.org/code/1263/report_bench_2026-09-03T233717.md)
- [results.json](https://synthcore.org/code/1263/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1263)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.