✗ Failed on benchmark 2026

Read-Port Capital Value

Implementation & benchmark of arXiv:2608.12791 — Thermodynamics of Learning: A Typed Four-Component Accounting of Memory, Fit, and Value

Usefulness8/10

Difficulty5/10

Novelty7/10

Training Generalization Stability Sample efficiency Training dynamics Fine tuning Regularization Information theory Optimization Statistics

Source paper: Thermodynamics of Learning: A Typed Four-Component Accounting of Memory, Fit, and Value arXiv:2608.12791 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Evaluate a neural network’s learned state by comparing its normal future-task performance with a matched blind counterfactual in which the stored representation, adapter, optimizer state, or memory slots are inaccessible and the model must re-optimize from the same compute budget. Train or select models to maximize this operational value rather than training loss or mutual information with the training data. The method should suppress nuisance memorization because information that cannot improve future-task work produces a zero deletion gap.

Formulas

$$V(M;T,b)=W_{\mathrm{informed}}(M;T,b)-W_{\mathrm{blind}}(T,b),$$

$$J_D(M)=I(M;D),$$

$$\widehat V(M;T,b)=\frac{1}{|T|}\sum_{\tau\in T}\left[U\!\left(f_{\theta,M}^{\mathrm{adapt}(b)};\tau\right)-U\!\left(f_{\theta,\varnothing}^{\mathrm{adapt}(b)};\tau\right)\right].$$

Mathematical statement

Let $M$ denote the learned neural-network memory state, such as weights, a frozen representation, optimizer state, or external memory; let $T$ be a distribution over future tasks; let $b$ be the per-run adaptation or inference-work budget; and let $W_{\mathrm{informed}}(M;T,b)$ and $W_{\mathrm{blind}}(T,b)$ be the optimal expected task utility with and without read access to $M$, respectively. The paper defines capital value as $V(M;T,b)=W_{\mathrm{informed}}(M;T,b)-W_{\mathrm{blind}}(T,b)$. The transferable distinction is that $V$ need not increase with training fit or record correlation $J_D(M)=I(M;D)$: a model can accumulate record information while having exactly zero capital gain. Use the empirical deletion gap as a task-conditioned value functional.

Implementation notes

Integrate this as a validation-time objective for a continual task model with a shared encoder and a memory state $M$. In the informed branch, retain the learned encoder, adapter, recurrent state, retrieval table, or optimizer state and allow at most $b$ gradient steps on each future task. In the blind branch, delete or mask the selected memory read port, reinitialize the task adapter or hidden state exactly as specified, and allow the same $b$ gradient steps, batch sequence, and compute. Pseudocode: train on source data; checkpoint $M$; for each held-out task $\tau$, run informed = adapt(model, read(M), tau, b) and blind = adapt(model, read_zero_or_reinitialized_memory, tau, b); compute V_hat = mean(U(informed)-U(blind)); optionally optimize a differentiable approximation to this gap using soft masks and short inner loops. Compute the deletion gap directly; estimate $I(M;D)$ only as a diagnostic using a variational classifier or k-nearest-neighbor estimator. The first cheap experiment is a small MLP on permuted-MNIST or task-disjoint CIFAR-100, comparing ordinary training, weight decay, and value-aware checkpoint selection. Prediction: increasing a nuisance memory independent of future-task labels can increase estimated $I(M;D)$ while leaving $\widehat V$ statistically indistinguishable from zero. A reusable memory should show a positive gap that decreases with task-shift severity and collapses near the adaptation budget at which blind adaptation catches up.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a matched informed/blind adaptation toy implementing the empirical capital-value deletion gap. The mechanism manifested: zero task-memory alignment produced an observed gap of -0.0016, zero-budget alignment matched the Gaussian prediction asin(rho)/pi with maximum absolute error 0.0045, and at rho=0.75 the gap decreased from 0.2709 to 0.0313 as the adaptation budget increased from 0 to 100. The direct per-task gap identity matched the difference of means to below 8e-16, and an orthogonal memory relabeling changed the estimate by only 0.00052.

Agent confidence: 8/10

Baseline: Blind zero-initialized logistic adaptation: at rho=0.75, held-out accuracy was 0.5000 at budget 0, 0.6929 at 1 step, 0.6981 at 5, 0.7106 at 20, and 0.7156 at 100.
Idea: Read-port informed adaptation: at rho=0.75, held-out accuracy was 0.7709 at budget 0, 0.7910 at 1 step, 0.7967 at 5, 0.7706 at 20, and 0.7469 at 100; deletion gaps were 0.2709, 0.0981, 0.0986, 0.0600, and 0.0313 respectively. At zero budget, observed gaps for rho=[0,0.25,0.5,0.75,1] were [-0.0016,0.0759,0.1689,0.2709,0.5000], versus predictions [0,0.0804,0.1667,0.2699,0.5000].

Limitations: This is a synthetic Gaussian halfspace experiment, not permuted-MNIST or CIFAR-100. It does not estimate mutual information I(M;D), test learned neural-network representations, optimize the deletion gap during training, measure compute/FLOPs, or establish nuisance-memory independence with a separately trained nuisance read port. Budget decay was clear in endpoint comparisons but not strictly monotone at every noisy sweep point.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.12791) Sep 2, 2026

Ahead of the baseline (−0.3%) but not statistically significant (p=0.8099, wins 3 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 9.9834
Idea mean: 9.9527
Effect (Δ): -0.0307 (−0.3%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.8099 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±2.7%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the registered tabular track was tested. The intervention was checkpoint selection rather than differentiable capital-value optimization; mutual information was not estimated, and the adaptation-gap implementation showed unstable scale and failed the predicted monotonic budget behavior.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Read-Port Capital Value as validation-time checkpoint selection on the registered tabular track with matched mlp_tiny systems. Across 8 paired seeds, idea mean test MSE was 9.952694 versus baseline 9.983420, but the paired permutation test was not significant (delta=-0.030726, p=0.80995), so there was no demonstrated win. The NN-scale deletion-gap signature was not confirmed.

Baseline: Best lr=0.003, epochs=30; full mean MSE=9.983420, std=0.869635.
Idea: Best lr=0.003, epochs=30; full mean MSE=9.952694, std=1.022854; paired delta=-0.030726; p=0.80995; 3/8 wins.

Benchmark result