# Zero-Augmented Double-Scoring

- ID: 1363
- Canonical URL: https://synthcore.org/idea/1363/zero-augmented-double-scoring
- API JSON: https://synthcore.org/api/idea/1363.json
- API Markdown: https://synthcore.org/api/idea/1363.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.20555](https://arxiv.org/abs/2607.20555)
- Category: architecture
- Solves: accuracy, stability, scalability
- ML areas: initialization, training, pruning
- Math tags: linear-algebra, optimization, combinatorics
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

For each frozen weight tensor, append a second tensor of identically shaped zero weights and assign trainable scores to both the real and dummy edges. Select a fixed number of candidates by top-k score in the doubled space; real edges selected by the competition remain active, while selected dummy edges consume the quota without changing the network. The resulting number of active original edges is learned rather than imposed by a separate layerwise sparsity search.

## Mathematical statement

The network is a composition of affine maps and activations, $$f_W(x)=W_{\\ell}\\sigma_{\\ell-1}\\big(\\cdots\\sigma_1(W_1x+b_1)\\cdots\\big)+b_{\\ell},$$ with frozen weights W_i. A binary mask H produces the subnetwork f_{W\\odot H}. The paper notes that masks are not unique when weights contain zeros: if W_{1,1}=0, changing the corresponding mask entry leaves W\\odot H unchanged, so distinct masks can represent the same masked tensor. Double-scoring exploits this redundancy by constructing an augmented candidate tensor \\widetilde W=[W\\;0] and score tensor \\widetilde S=[S^{\\mathrm{real}}\\;S^{\\mathrm{dummy}}]. For a layer containing M real edges, choose exactly K augmented coordinates using \\operatorname{TopK}_K(\\widetilde S). The realized original mask is H_j=1 exactly when selected coordinate j is a real edge; dummy selections contribute zero because their augmented weights are zero. Thus the effective real density is r/M, where r=\\sum_{j=1}^{M}H_j can vary between 0 and min(K,M), even though the augmented density K/(2M) is fixed. Any original mask with at most K active real edges can be represented by selecting those real coordinates plus K-r dummy coordinates. During score training, the forward pass is f_{\\widetilde W\\odot\\widetilde H} and only scores are updated; W remains fixed.

## Key formulas

- $$f_W(x)=W_{\ell}\sigma_{\ell-1}\big(\cdots\sigma_1(W_1x+b_1)\cdots\big)+b_{\ell},$$
- $$f_{W\odot H}(x),\qquad (W\odot H)_{ab}=W_{ab}H_{ab},\quad H_{ab}\in\{0,1\},$$
- $$\widetilde W=[W\;0],\qquad \widetilde S=[S^{\mathrm{real}}\;S^{\mathrm{dummy}}],\qquad \widetilde H=\operatorname{TopK}_{K}(\widetilde S),$$
- $$H_j=\\widetilde H_j\quad(1\\le j\\le M),\\qquad r=\\sum_{j=1}^{M}H_j,\\qquad \\rho_{\\mathrm{eff}}=\\frac{r}{M}\\leq\\frac{K}{M}.$$

## Implementation notes

Integrate this at every prunable linear or convolutional layer during score-only ticket extraction. Keep the usual randomly initialized weight tensor W frozen. For a layer with M scalar edges, create trainable real scores S_real with the same shape as W and trainable dummy scores S_dummy with the same shape; do not allocate or multiply a second dense weight tensor, because dummy values are mathematically zero. At each forward pass: (1) concatenate scores into S_aug = concat(S_real, S_dummy); (2) compute H_aug as a hard top-K indicator over all 2M scores, using K = floor(rho_aug times 2M); (3) form the effective real mask H = H_aug[..., :M]; and (4) execute the layer with W multiplied elementwise by H. Use a straight-through estimator, H_aug = stop_gradient(hard_topk(S_aug) - S_aug) + S_aug, or the same differentiable score surrogate used by the baseline edge-popup implementation. Update only S_real and S_dummy with SGD or Adam; after extraction, discard dummy coordinates and retain H. The paper supplies the zero-augmentation and fixed-density representation principle, so no additional mathematical quantity must be estimated. Log r/M independently for every layer and also log the fraction of selected candidates that are dummy. First run a cheap experiment on CIFAR-10 with a six-layer frozen random ReLU MLP or small CNN, comparing ordinary edge-popup at densities 0.1, 0.2, 0.4, and 0.6 against double-scoring at one fixed augmented density, using equal score-update steps and identical initialization. Evaluate test accuracy before weight training, accuracy after rewinding and training the selected topology, and variance across five seeds. Success means higher strong-ticket accuracy at comparable effective density, a flatter accuracy-versus-density curve, and substantially lower variance or sensitivity to the chosen density.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a readable zero-augmented double-scoring MVP with frozen weights, trainable real/dummy scores, hard Top-K straight-through selection, exhaustive representation checking, and a five-seed toy comparison. Across M=20,50,100 and augmented densities 0.2,0.5,0.8, observed real-count means and standard deviations matched the exact hypergeometric predictions, and dummy fractions were approximately 0.5; all masks with r≤K were represented exactly. The toy double-scoring accuracy was 0.700 versus 0.683 for baseline, but its effective density averaged 0.34 versus the baseline's fixed 0.30, so this is only a weak secondary signal rather than evidence of a clear accuracy win.

### Mechanism check

- Verdict: Built a readable zero-augmented double-scoring MVP with frozen weights, trainable real/dummy scores, hard Top-K straight-through selection, exhaustive representation checking, and a five-seed toy comparison. Across M=20,50,100 and augmented densities 0.2,0.5,0.8, observed real-count means and standard deviations matched the exact hypergeometric predictions, and dummy fractions were approximately 0.5; all masks with r≤K were represented exactly. The toy double-scoring accuracy was 0.700 versus 0.683 for baseline, but its effective density averaged 0.34 versus the baseline's fixed 0.30, so this is only a weak secondary signal rather than evidence of a clear accuracy win.
- Confidence: 9/10
- Limitations: The planned CIFAR-10 six-layer MLP/CNN, rewinding and post-selection weight training, layerwise density curves, FLOP-matched comparisons, and five-seed trained-ticket evaluation were not tested. The toy comparison uses a single frozen linear layer and differs in effective density, so it cannot establish a practical strong-ticket improvement.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.16705
- Paired wins: 2/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/533/bench_report.json)
- [report.md](https://synthcore.org/code/533/report.md)
- [report_bench_2026-08-31T170403.md](https://synthcore.org/code/533/report_bench_2026-08-31T170403.md)
- [results.json](https://synthcore.org/code/533/results.json)
- [run_bench.py](https://synthcore.org/code/533/run_bench.py)
- [run_experiment.py](https://synthcore.org/code/533/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/533)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.