Singular-Value-Robust Projector-Splitting LoRA MVP

Run with:

/home/maxwelhelp/main/bin/python3 projector_splitting_mvp.py

The complete machine-readable output is in results.json.

Implementation

projector_splitting_mvp.py implements the paper's three-substep QR projector-splitting retraction and the common-base explicit midpoint method. It also includes a small matrix-fitting comparison against independently updated low-rank factors. The comparison baseline is plain factor gradient descent (not Adam), so it is only a directional toy comparison.

Stage-1 quantitative predictions

The toy verification uses an exactly rank-2 rotating matrix curve with a known full matrix derivative. Three predictions were tested:

1. QR orthogonality/rank preservation. Thin QR should keep U and V orthonormal and maintain a rank-r representation without inverse-S operations. For sigma-min values 1e-2, 1e-6, and 1e-10, U orthogonality errors were 8.9e-16, 2.9e-16, and 4.7e-16; V errors were 8.1e-16, 6.7e-16, and 5.1e-16.

2. Second-order convergence. The common-base explicit midpoint construction predicts global error proportional to h^2. With h values 0.2, 0.1, 0.05, 0.025, observed orders from the final two refinements were 1.99897, 1.99897, and 1.99897 for sigma-min 1e-2, 1e-6, and 1e-10, respectively. This agrees with the predicted order 2.

3. Robustness as a singular value collapses. The error constant should remain essentially independent of the smallest retained singular value. At h=0.025, errors were 1.00198e-4, 1.00193e-4, and 1.00193e-4 for sigma-min 1e-2, 1e-6, and 1e-10; the maximum/minimum ratio was about 1.00005. Factor norms also remained finite and similar (5.756, 5.744, 5.696).

The math_checks reconstruction residuals around 0.667 are expected: a rank-r projector-splitting step approximates the generally full-rank Euler matrix Y+hZ; it is not supposed to reproduce that full-rank increment exactly. The meaningful structural checks are orthogonality, finite factors, and convergence to the rank-r exact curve.

Toy optimization comparison

On a 20x16 matrix-fitting task with rank 4 and initial core singular values ending at 1e-6, after 120 steps at step size 0.08:

• Factor gradient-descent baseline final loss: 0.226712
• Projector-splitting midpoint final loss: 0.106228
• NaNs: neither method
• CPU time: baseline 0.0172 s, idea 0.1048 s

Thus the idea showed a promising lower-loss signal in this tiny setup, but QR and the two-stage midpoint evaluation were approximately 6.1x slower in this unoptimized NumPy implementation. This is not evidence of a general training win over production Adam or GPU kernels.

Verdict

The requested mechanism manifested clearly: second-order convergence and stable error constants persisted while sigma-min decreased from 1e-2 to 1e-10, with no inverse singular-value operation. The toy optimization result favors projector splitting at matched steps, but the baseline and problem are too small to establish a practical LoRA advantage; the strongest supported claim is singular-value robustness.