Hybrid Least Squares/Gradient Descent Methods for MIONets

What the math gives to ML

The paper exposes a useful multilinear structure in MIONets: once all but one branch's final-layer parameters are fixed, the network output is linear in the remaining parameters. This permits exact or ridge-regularized block least-squares updates instead of noisy gradient steps, while retaining gradient descent for nonlinear hidden layers. The transferable asset is alternating optimization over multilinear factors together with matrix-free Kronecker/Khatri-Rao implementations that avoid constructing enormous design matrices. A practical first test is a hybrid optimizer for tensor-factorized MLPs or multi-input operator networks, measuring loss decrease and wall-clock time at equal computational cost.