Thermodynamics of Learning: A Typed Four-Component Accounting of Memory, Fit, and Value
arXiv:2608.12791
2026
Training
2 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper’s central transferable mechanism is a deletion-counterfactual definition of learned value: a memory state is useful only when an informed agent with read access outperforms a blind agent that has the read port removed and re-optimizes under the same task, access, and compute budget. This separates training fit and record correlation from operational future-task utility, exposing cases where memorization increases while downstream value remains exactly zero. A second mechanism is an information-to-value capitalization ledger, which bounds value gain per unit of update-side information acquisition in a specified no-discarded-correlation regime. Neural-network implementations should turn these ideas into ablation-based objectives and diagnostics for fine-tuning, optimizer memory, and task-shift robustness, with explicit predictions about retention and nuisance memorization.
Ideas from this paper
✗ Failed on benchmark
2026
Evaluate a neural network’s learned state by comparing its normal future-task performance with a matched blind counterfactual in which the stored representation, adapter, optimizer state, or memory slots are inaccessible and the model must re-optimize from the same compute budget. Train or select models to maximize this operational value rather than training loss or mutual information with the training data. The method should suppress nuisance memorization because information that cannot…
Useful8/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Monitor learning as the ratio of future-task value gained to information irreversibly acquired by an update, rather than treating every reduction in training loss as equally productive. Penalize updates that absorb substantial data-specific information without increasing deletion-counterfactual value, and use the ratio to stop, trust-region, or schedule updates. This creates a falsifiable diagnostic for overfitting without assuming that overfitting and low efficiency are monotonically related.
Useful7/10
Difficulty6/10
Novelty6/10