# Эксперимент: Conformal Early-Rejection for Diffusion Architecture Search (#1090)

{
  "worked": false,
  "confidence": 9,
  "verdict": "Built a synthetic diffusion/NAS trajectory simulator with checkpoint-wise split-conformal thresholds, authoritative final labels, exchangeability-shift tests, and baseline comparisons. The finite-sample conformal score-exceedance prediction was confirmed closely across calibration sizes. However, the core conditional failure claim was not confirmed: at alpha=0.10, conformal rejection saved 17.13% of evaluations but accepted-candidate failure remained 39.98%, far above 10%, and increasing score separability produced no meaningful monotonic improvement.",
  "metrics": {
    "baseline": "No rejection: 1000.0 full evaluations per 1000 candidates, accepted failure 50.25%. Uncalibrated threshold: 357.55 evaluations, accepted failure 2.13%, but it rejected 64.25% and was not calibrated.",
    "idea": "Split-conformal alpha=0.10: 828.75 full evaluations per 1000, 17.13% compute saved, accepted failure 39.98%. Alpha=0.05: 905.14 evaluations, 9.49% saved, accepted failure 45.03%. At alpha=0.10, observed calibration exceedance for ncal 20, 50, 100, 500 was 0.0938, 0.0993, 0.0989, 0.0999 versus predictions 0.0952, 0.0980, 0.0990, 0.0998. Under test shifts 0, -0.25, -0.5, and -1.0, accepted failure was 39.92%, 47.99%, 55.35%, and 68.73%."
  },
  "how_to_run": "/home/maxwelhelp/main/bin/python3 conformal_early_rejection.py",
  "files": [
    "conformal_early_rejection.py",
    "results.json",
    "experiment_report.json",
    "run_output.txt"
  ],
  "limitations": "This is a synthetic trajectory checker rather than NAS-Bench-201 or a real diffusion architecture mutator. It does not measure GPU training, compilation, real latency, or wall-clock savings, and it uses a simple hand-designed risk score rather than a trained neural risk head."
}