Shared Symbolic Mechanism Bottleneck

Integrate this as the final shared representation and output head for a small multi-task regression model. (1) For a batch \(X\in\mathbb{R}^{B\times d}\), construct a numerically safe feature bank: standardized raw inputs, clipped reciprocals \(1/\operatorname{clip}(x,\epsilon,\epsilon^{-1})\), and \(\log(|x|+\epsilon)\). Create \(H\) units with learnable \(w_h,b_h\), continuous feature gates \(g_{hk}=\operatorname{sigmoid}(u_{hk})\), and operator logits \(o_{hk}\). Compute operator probabilities \(p_{hk}=\operatorname{softmax}(o_h/\tau)_k\), then evaluate \(z_h=\sum_k p_{hk}\sigma_k(w_h^\top(g_h\odot\phi(X))+b_h)\). Use safe implementations for reciprocal, logarithm, square root, and exponential with clipping. (2) Give output \(j\) coefficients \(a_{jh}\), bias \(c_j\), and read-out gates \(q_{jh}=\operatorname{sigmoid}(v_{jh})\), computing \(\hat y_j=c_j+\sum_h q_{jh}a_{jh}z_h\). (3) Optimize prediction loss plus \(\lambda_r\sum_{j,h}|q_{jh}a_{jh}|\), \(\lambda_s\sum_{h,k}g_{hk}\), and operator entropy \(\lambda_o\sum_{h,k}p_{hk}\log p_{hk}\). Anneal \(\tau\) from 1 to 0.05. After training, set gates above 0.5 to one, choose each operator by argmax, prune inactive units, and fine-tune the remaining continuous parameters. Optionally mutate one operator or gate at a time and retain a child only when validation loss plus a complexity penalty improves. The first cheap experiment should use a 3-output synthetic kinetics model with a shared denominator, such as \(y_j=\alpha_jx_j/(1+K_1x_1+K_2x_2)\), with 1,000 samples and 0% or 1% noise. Compare against independent MLP heads and a standard shared MLP trunk at equal parameter count. Record validation error, extrapolation error, active-unit count, and whether all outputs recover the same denominator. Success is equal or lower in-range error, lower extrapolation error, and substantially more consistent recovery of the common factor.