Shared Symbolic Mechanism Bottleneck MVP

Implementation

experiment.py implements the proposed differentiable symbolic bank: standardized raw inputs, safe reciprocal/log features, sigmoid feature gates, soft operator mixtures over seven operators, sigmoid readout gates, sparse readout regularization, feature-gate regularization, and temperature annealing. It compares this model with a small two-hidden-layer shared MLP on a three-output synthetic kinetics problem.

Stage-1 quantitative mechanism checks

The checks were run with fixed seed 1450.

1. Operator softmax temperature law. For an operator-logit gap Delta=2, the predicted entropy is H = log(1+exp(-Delta/tau)) + (Delta/tau)/(1+exp(Delta/tau)). Observed versus predicted entropy:

   | tau | observed | predicted | |---:|---:|---:| | 1.00 | 0.3653338551 | 0.3653338551 | | 0.50 | 0.0900947678 | 0.0900947678 | | 0.25 | 0.0030182074 | 0.0030182074 | | 0.10 | 4.3284e-08 | 4.3284e-08 | | 0.05 | 1.6993e-16 | 1.7418e-16 |

   This confirms the expected rapid concentration of the differentiable operator choice as temperature decreases.

2. Shared latent readout prediction. If every output is a coefficient times the same latent z, least-squares shared-factor reconstruction has zero residual in exact arithmetic. For 1,000 synthetic samples and three outputs, the measured residual was 3.13e-32.

3. Noise scaling prediction. For additive zero-mean noise, expected irreducible MSE is sigma^2. Measured versus predicted MSE:

   | sigma | measured | sigma^2 | |---:|---:|---:| | 0.00 | 0.000000 | 0.000000 | | 0.01 | 0.00010046 | 0.00010000 | | 0.03 | 0.00090141 | 0.00090000 | | 0.10 | 0.01002553 | 0.01000000 |

All three elementary mathematical predictions were numerically confirmed. The second and third are mechanism sanity checks rather than evidence that optimization discovers the factor.

Mini-experiment

Training used 1,000 samples from x in [0.15,5], 1% noise, 500 Adam epochs, and fixed initialization. Extrapolation used noiseless samples from [5,10].

| model | in-range MSE | extrapolation MSE | |---|---:|---:| | symbolic bottleneck | 0.0800653 | 0.0449622 | | shared MLP | 0.000284852 | 0.00317361 |

The symbolic model retained 5 active readout units after the simple activity threshold. It did not recover a single interpretable common denominator; its selected operator indices were [0,2,5,6,0,1,1,0].

Verdict

The math sanity checks passed and the symbolic model showed a useful extrapolation signal, but it did not achieve equal in-range error and did not recover the shared denominator. In particular, the stated output head is additive in shared units, while the kinetics target requires output-specific factors x_j multiplied by the common reciprocal denominator; this mismatch likely makes the target unnecessarily difficult for the proposed bank. Therefore this MVP does not establish the claimed overall win.

Run with:

/home/maxwelhelp/main/bin/python3 experiment.py

Results are also saved in results.json.