{
 "artifacts": null,
 "category": "architecture",
 "description": "Make a compact hidden representation explicitly decorrelated under the empirical data distribution while retaining a supervised linear readout. This creates a spectral-style bottleneck whose coordinates cannot redundantly encode the same feature, potentially improving small embeddings and making downstream linear decoding more effective.",
 "formulas_latex": [
  "$$\\langle f_{1},f_{2}\\rangle_{0}=\\int f_{1}(\\mathbf{x})f_{2}(\\mathbf{x})p(\\mathbf{x})d\\mathbf{x}$$",
  "$$C=\\frac{1}{B}\\widetilde Z^{\\mathsf T}\\widetilde Z,\\qquad \\widetilde Z=Z-\\mathbf{1}\\bar z^{\\mathsf T},\\qquad L_{\\mathrm{gram}}=\\|C-I_k\\|_{F}^{2}$$",
  "$$L(\\theta,W)=L_{\\mathrm{task}}(WZ)+\\lambda_{\\mathrm{gram}}\\left\\|\\frac{1}{B}\\widetilde Z^{\\mathsf T}\\widetilde Z-I_k\\right\\|_{F}^{2}+\\lambda_{\\mathrm{dir}}\\frac{1}{B}\\sum_{n=1}^{B}\\left\\|J_z(x_n)A(x_n)^{\\mathsf T}\\right\\|_{F}^{2}$$",
  "$$\\min_{\\theta,W}\\;L_{\\mathrm{task}}(Wf_\\theta(x))+\\lambda_{\\mathrm{gram}}L_{\\mathrm{gram}}+\\lambda_{\\mathrm{dir}}L_{\\mathrm{dir}},\\qquad W\\in\\mathbb{R}^{c\\times k}$$"
 ],
 "id": 246,
 "implementation": "Insert a projection head \\(z=f_\\theta(x)\\in\\mathbb{R}^k\\) before an existing classifier or encoder output, choosing \\(k\\) much smaller than the original hidden width. On every training step, compute \\(Z=f_\\theta(X)\\), the batch mean \\(\\bar z=B^{-1}\\sum_nz_n\\), centered features \\(\\widetilde Z=Z-\\mathbf{1}\\bar z^T\\), and \\(C=B^{-1}\\widetilde Z^T\\widetilde Z\\). Add \\(\\lambda_{gram}\\|C-I_k\\|_F^2\\) to the task loss. Normalize the representation with LayerNorm before the projection, or otherwise control feature scale, because unconstrained scaling can interact with the identity target. Keep the task head explicitly linear, \\(\\mathrm{logits}=WZ+b\\), matching the supervised readout in the construction. The core pseudocode is: `Z=proj(encoder(X)); Zc=Z-Z.mean(0); C=Zc.T@Zc/B; loss=CE(W@Z+b,y)+lam_gram*((C-I)**2).sum(); loss.backward(); optimizer.step()`. The identity Gram matrix is the mathematical target; minibatch covariance is its empirical estimator. Track an exponential moving average of \\(C\\) only for diagnostics, unless a large-batch variant is desired. First test CIFAR-10 with a small CNN or ViT and a 64-dimensional projection, comparing cross-entropy alone, ordinary covariance decorrelation, Barlow-Twins-style off-diagonal loss, and the full identity-Gram penalty. Evaluate accuracy versus projection dimension, linear-probe accuracy, covariance off-diagonal energy, and representation memory. Success means matching baseline accuracy with a 2x--4x smaller representation and lower coordinate redundancy without materially slower optimization.",
 "math_summary": "Use the paper's population inner product \\(\\langle f_1,f_2\\rangle_0=\\int f_1(x)f_2(x)p(x)dx\\) as the target notion of coordinate orthogonality. For a minibatch representation matrix \\(Z\\in\\mathbb{R}^{B\\times k}\\), with row \\(z_n=f_\\theta(x_n)\\), center the features as \\(\\widetilde Z=Z-\\mathbf{1}\\bar z^{\\mathsf T}\\), where \\(\\bar z=B^{-1}\\sum_n z_n\\), and form the empirical Gram matrix \\(C=B^{-1}\\widetilde Z^{\\mathsf T}\\widetilde Z\\). Penalize deviation from identity with \\(L_{gram}=\\|C-I_k\\|_F^2\\), a batch approximation to \\(\\langle z_i,z_j\\rangle_0=\\delta_{ij}\\). Couple this with a supervised linear readout \\(g(x)=Wz(x)+b\\), where \\(W\\in\\mathbb{R}^{c\\times k}\\) maps the \\(k\\)-dimensional representation to \\(c\\) classes, and optionally with the metric-weighted Dirichlet energy \\(L_{dir}=B^{-1}\\sum_n\\|J_z(x_n)A(x_n)^T\\|_F^2\\). The identity Gram target fixes coordinate scale and discourages collapse, while the linear readout ensures that the coordinates remain predictive rather than merely diverse.",
 "math_tags": [
  "functional-analysis",
  "linear-algebra",
  "spectral-theory",
  "regularization"
 ],
 "ml_areas": [
  "embedding",
  "mlp",
  "loss",
  "memory"
 ],
 "paper": {
  "arxiv_id": "2607.03692",
  "arxiv_url": "https://arxiv.org/abs/2607.03692",
  "summary_what_math_gives_to_ml": "PIEFS combines a supervised coordinate map with a Dirichlet-energy penalty whose local metric is learned rather than fixed. The transferable asset is a differentiable anisotropic smoothness prior: the model can penalize variation strongly in irrelevant directions while preserving rapid variation along task-relevant directions, with rotations represented compactly by Givens factors. A second useful component is the explicit coupling of batch Gram orthogonality, a linear readout, and the metric-weighted Jacobian penalty, which can turn a neural hidden layer into a compact, decorrelated spectral-style representation.",
  "title": "PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 3,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "generalization",
  "memory",
  "accuracy"
 ],
 "title": "Orthogonal supervised coordinate bottleneck",
 "url": "https://synthcore.org/idea/246/orthogonal-supervised-coordinate-bottleneck",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}