Two-dimensional checkpoint repair

Integrate this at the distributed-training checkpoint layer, not inside the forward or backward pass. First flatten model parameters, Adam first and second moments, gradient-scaler state, and scheduler state into one byte string m. Choose n checkpoint or storage workers and a tolerated fault count t, set k=n-t, pad m to exactly k squared equal-sized blocks, and reshape those blocks into M. For worker i up to k, store row M[i,:] and column M[:,i]; for extra workers, use the paper’s dispersal assignment or duplicate selected row and column shares. Attach a cryptographic hash or Merkle proof pi to every block and publish a root commitment. Pseudocode: blocks=pad_and_split(checkpoint,k*k); M=reshape(blocks,k,k); root=commit(M); for each worker i, send row=M[i,:], col=M[:,i], and proofs to worker i. On failure of worker j, the replacement requests only the row and column blocks assigned to j, verifies every proof against root, rejects inconsistent responses, assembles the required state, and resumes training. If protection against missing as well as corrupted shares is required, add a Reed-Solomon erasure layer independently to rows and columns and measure its redundancy separately. Compute the predicted storage factor 2n/(n-t) and recovery traffic 4.5|m|/n, then measure actual bytes, recovery wall-clock time, and time to resume after killing one worker. Start with a 4- or 8-GPU data-parallel ResNet-50 or small Transformer, checkpointing parameters plus Adam states every few hundred steps. Compare conventional replicated checkpoints and a central-server shard baseline at equal fault tolerance. Success means recovery traffic scales toward 4.5|m|/n, replacement time scales with one shard rather than the full checkpoint, and post-recovery loss or perplexity matches the baseline.