Joint Modeling for Stochastic Interventions / report.md
Mechanism confirmed, baseline not beaten
Эксперимент: Joint Modeling for Stochastic Interventions (#1342)
{ "worked": true, "confidence": 8, "verdict": "Built a synthetic SCM MVP with an analytic sanity check and neural joint-vs-mediator-only comparison. The math check reproduced mediator selection bias: estimated E[Y|M≈1] shifted from 1.50 under intervention variance 1 to 1.76 under variance 4, close to the analytic 1.8. On shifted-intervention test data, the joint model achieved NLL 1.647 versus 1.968 for the baseline, a 0.320 improvement; the effect is promising, though the experiment is small and the marginal prediction evaluation is limited.", "metrics": { "baseline": "Shifted sigma=2 outcome NLL: 1.9675; mediator-only predicted selected-M mean: 1.6589", "idea": "Shifted sigma=2 outcome NLL: 1.6475; joint predicted selected-M mean: 2.0122; sampled marginal Y variance: 13.315 versus analytic target variance 18" }, "how_to_run": "/home/maxwelhelp/main/bin/python3 experiment.py", "files": [ "experiment.py", "results.json" ], "limitations": "Only one fixed seed and one synthetic SCM were tested. The reported NLL uses observed shifted intervention values rather than a full held-out marginal likelihood; the learned q intervention head was not separately calibrated, and the sampled marginal variance was under-estimated relative to the analytic value. No repeated trials, richer contexts, categorical outcomes, or real offline-RL/world-model data were evaluated." }