Reversal-Defect Adaptive Rank and Checkpointing

(1) Exact integration point: wrap the controller around a reversible low-rank recurrent or Neural ODE block. The controller operates on factor checkpoints, not on model weights. A block contains B low-rank time steps, and the default policy saves one factor triple every C steps.

(2) Pseudocode:

for each block beginning at checkpoint F_k:
    run B forward low-rank steps to obtain F_end
    probe = reverse_one_block(F_end, step = -h)
    delta = ||probe - F_k||_F / (||F_k||_F + eps)
    rho = sigma_min(S_probe) / (sigma_max(S_probe) + eps)
    if delta > tau_warn:
        save an additional midpoint checkpoint
        C = max(1, C // 2)
    if delta > tau_rank or rho < tau_svd:
        increase rank by Delta_r using residual enrichment
    during backward:
        reverse from the nearest checkpoint
        if local delta > tau_grad, reload a checkpoint instead

When increasing rank, form the tangent-space residual R = G - P_F G, compute a randomized range basis for R, append Delta_r orthonormal columns to X and V, and initialize the new diagonal entries of S to a small positive alpha. Do not pad with exact zeros, because that recreates the rank-deficient case.

(3) The reversal formula supplies the defect test; singular values of S and the residual range are computed numerically. The thresholds tau_warn, tau_rank, tau_svd, tau_safe, and tau_grad are hyperparameters. The reference-gradient cosine is computed only on occasional small minibatches, using dense autograd or a much denser checkpoint schedule. Log rank, checkpoint count, reversal defect, and gradient cosine throughout training.

(4) First cheap experiment: use a 6-layer low-rank Neural ODE on permuted Sequential MNIST, with maximum rank 32 and initial checkpoint intervals C equal to 4, 8, and 16. Compare fixed-rank/fixed-checkpoint training with the adaptive controller under equal peak-memory budgets. Evaluate peak memory, average rank, number of saved checkpoints, reversal defects, gradient cosine similarity, loss variance, and accuracy. Success means fewer checkpoints or lower average rank at the same accuracy, gradient cosine above 0.98, and no loss spikes caused by inaccurate state reconstruction.