Dual-Ensemble Latent Transition Model MVP

What was implemented

dual_ensemble_experiment.py builds an eight-state reversible equilibrium Markov chain with adjacent microscopic states merged into four coarse states. Its equilibrium coarse occupancy is exactly

(18, 12, 6, 1) / 37 = (0.486486, 0.324324, 0.162162, 0.027027).

A source-sink recycling operator Q resets the sink states (6, 7) to the source distribution over states (0, 1). The script estimates stationary vectors, stationary-weighted coarse operators, flux-related quantities, MFPTs, and finite-sample transition matrices.

Core math verification

• Equilibrium stationarity error: max|pi P - pi| = 2.776e-17.
• Recycling stationarity error: max|pi_Q Q - pi_Q| = 1.665e-16.
• Recycling MFPT from the source to the sink:
  • direct first-passage calculation: 837.500000
  • Hill identity 1/pi_Q(B) - 1: 837.500000
  • absolute discrepancy: 3.786e-11.

Quantitative predictions and sweeps

Prediction 1: matched stationary distributions are independent of lag

Predicted error: numerical zero for every lag when the equilibrium distribution is paired with P and the recycling distribution with Q.

Maximum coarse stationary error for lags 1, 2, 4, 8, 16, 32:

• equilibrium: 1.554e-15, 4.441e-16, 3.608e-16, 3.886e-16, 4.441e-16, 2.776e-16
• recycling: 2.859e-15, 6.439e-15, 4.885e-15, 5.107e-15, 5.884e-15, 6.106e-15

This prediction is confirmed to floating-point precision.

Prediction 2: ensemble matching removes source-sink MFPT bias

Prediction: the dual recycling operator has zero model bias for recycling MFPT, while an equilibrium-only operator remains biased; the discrepancy should increase with the difference between equilibrium and recycling ensembles.

For reset mixtures from state 0 to state 1 of 0, 0.25, 0.5, 0.75, and 1.0, respectively:

• dual MFPT absolute error: 0.000000 for every mixture (within numerical precision)
• single equilibrium-operator MFPT errors: 805.666667, 803.583333, 801.500000, 799.416667, 797.333333

The effect is very large and the dual prediction is exact in this finite-state construction.

Prediction 3: statistical error decreases approximately as N^-1/2

Predicted log-log slope: -0.5.

Dual recycling occupancy RMS error over 30 replications:

• N=400: 0.109303
• N=1600: 0.079340
• N=6400: 0.049304
• N=25600: 0.023440

Fitted slope: -0.368. This has the predicted decreasing trend but is slower than ideal asymptotic N^-1/2, likely because the chain has a very rare sink state and the estimator uses finite-count smoothing. I count this as a promising but weaker confirmation, not an exact scaling confirmation.

The P-only estimator's NESS occupancy errors at the same sample sizes were 0.134357, 0.119171, 0.093724, 0.079484, consistently worse than the matched dual estimator.

Reproduction

/home/maxwelhelp/main/bin/python3 dual_ensemble_experiment.py

Limitations

This is a synthetic discrete Markov-chain verification, not a learned neural latent model. It does not test encoder learning, soft memberships, optimization stability, wall-clock speed, uncertainty calibration, grid-world data, MuJoCo, or a realistic fixed-lag world-model benchmark. The source-sink operator is supplied from exact recycling dynamics rather than learned jointly from raw trajectories. The N^-1/2 result is based on only four sample sizes and is affected by rare-sink finite-sample effects.