# Two-dimensional checkpoint repair

- ID: 140
- Canonical URL: https://synthcore.org/idea/140/two-dimensional-checkpoint-repair
- API JSON: https://synthcore.org/api/idea/140.json
- API Markdown: https://synthcore.org/api/idea/140.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.24636](https://arxiv.org/abs/2608.24636)
- Category: memory
- Solves: memory, scalability, stability
- ML areas: training, optimizer, memory
- Math tags: linear-algebra, combinatorics, information-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Encode a neural-network checkpoint into a k by k matrix with k=n-t, and assign worker i both a row fragment and a column fragment. When a worker fails, a replacement obtains only the row and column fragments needed to reconstruct its assigned state, instead of downloading the complete checkpoint from all workers.

## Mathematical statement

The paper embeds the data item m, of bit length |m|, into a matrix M whose payload occupies k^2=(n-t)^2 elements, where n is the number of storage workers and t<n/3 is the Byzantine-fault bound. Worker v_i stores a row vector r_i and a column vector c_i, each containing k=n-t elements, plus commitments or proofs for its 2k fragments. Since one matrix element has size |m|/k^2, the aggregate payload storage is 2nk|m|/k^2=2n|m|/(n-t)<3|m|. The paper’s communication theorem gives dispersal 6|m|+O(n^2|pi|), retrieval 3|m|/2+O(n^2|pi|), and replacement recovery (9/2)|m|/n+O(n|pi|), where pi is a fragment-commitment proof. The neural-network adaptation applies this layout to a serialized checkpoint m, including parameters, optimizer moments, and scheduler state; a replacement worker reconstructs its row and column assignments after verifying fragment commitments.

## Key formulas

- $$k=n-t,\qquad |M_{ij}|=\frac{|m|}{(n-t)^2}=\frac{|m|}{k^2}.$$
- $$\text{aggregate payload storage}=2n(n-t)\frac{|m|}{(n-t)^2}=\frac{2n}{n-t}|m|<3|m|\qquad (t<n/3).$$
- $$C_{\mathrm{disperse}}=6|m|+O(n^2|\pi|),\qquad C_{\mathrm{retrieve}}=\frac{3}{2}|m|+O(n^2|\pi|),\qquad C_{\mathrm{recover}}=\frac{9}{2}\frac{|m|}{n}+O(n|\pi|).$$
- $$M=\operatorname{reshape}(m,k,k),\qquad S_i=(r_i,c_i)=\bigl(M_{i,:},M_{:,i}\bigr),\qquad \widehat{S}_i=\operatorname{verify\_and\_assemble}(\{r_i,c_i\}).$$

## Implementation notes

Integrate this at the distributed-training checkpoint layer, not inside the forward or backward pass. First flatten model parameters, Adam first and second moments, gradient-scaler state, and scheduler state into one byte string m. Choose n checkpoint or storage workers and a tolerated fault count t, set k=n-t, pad m to exactly k squared equal-sized blocks, and reshape those blocks into M. For worker i up to k, store row M[i,:] and column M[:,i]; for extra workers, use the paper’s dispersal assignment or duplicate selected row and column shares. Attach a cryptographic hash or Merkle proof pi to every block and publish a root commitment. Pseudocode: blocks=pad_and_split(checkpoint,k*k); M=reshape(blocks,k,k); root=commit(M); for each worker i, send row=M[i,:], col=M[:,i], and proofs to worker i. On failure of worker j, the replacement requests only the row and column blocks assigned to j, verifies every proof against root, rejects inconsistent responses, assembles the required state, and resumes training. If protection against missing as well as corrupted shares is required, add a Reed-Solomon erasure layer independently to rows and columns and measure its redundancy separately. Compute the predicted storage factor 2n/(n-t) and recovery traffic 4.5|m|/n, then measure actual bytes, recovery wall-clock time, and time to resume after killing one worker. Start with a 4- or 8-GPU data-parallel ResNet-50 or small Transformer, checkpointing parameters plus Adam states every few hundred steps. Compare conventional replicated checkpoints and a central-server shard baseline at equal fault tolerance. Success means recovery traffic scales toward 4.5|m|/n, replacement time scales with one shard rather than the full checkpoint, and post-recovery loss or perplexity matches the baseline.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a byte-level 2-D checkpoint encoder with SHA-256 commitment checking, storage accounting, failure simulation, and a minimal diagonal-redundancy variant. The storage formula matched 2n/(n-t) across n=4..64, and repair traffic was below full-checkpoint replication, but the literal row-plus-column assignment could repair only 2/4, 4/8, 10/16, 20/32, and 42/64 worker failures because the failed assignment's diagonal block is unavailable unless another worker duplicates that assignment. Therefore the claimed general replacement-recovery effect was not demonstrated; the repairable variant requires extra redundancy not specified in the literal layout.

### Mechanism check

- Verdict: Built a byte-level 2-D checkpoint encoder with SHA-256 commitment checking, storage accounting, failure simulation, and a minimal diagonal-redundancy variant. The storage formula matched 2n/(n-t) across n=4..64, and repair traffic was below full-checkpoint replication, but the literal row-plus-column assignment could repair only 2/4, 4/8, 10/16, 20/32, and 42/64 worker failures because the failed assignment's diagonal block is unavailable unless another worker duplicates that assignment. Therefore the claimed general replacement-recovery effect was not demonstrated; the repairable variant requires extra redundancy not specified in the literal layout.
- Confidence: 9/10
- Limitations: This is a toy byte-array simulation, not distributed multi-GPU training; it does not measure post-recovery loss, optimizer/scheduler continuity, network wall-clock transfer, Byzantine proof protocols, Reed-Solomon erasure coding, or a real checkpoint server. The commitment is a simple ordered hash of blocks rather than a Merkle proof system.

## Artifacts

- [checkpoint_repair.py](https://synthcore.org/code/1185/checkpoint_repair.py)
- [report.md](https://synthcore.org/code/1185/report.md)
- [Download all files as ZIP](https://synthcore.org/download/1185)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.