Rank-Adaptive Tensor-Train MLP

(1) Integration point: replace one or more large Linear layers in an MLP, coordinate neural field, or neural operator with a TTLinear module. Factor the input width as n_1...n_d and output width as m_1...m_d. Reshape each input vector into a d-way tensor, contract it successively with the TT cores, then reshape the result back to a vector. Apply the normal bias and GELU or ReLU after the full contraction.

(2) Pseudocode:

initialize W[k] with shape [r[k-1], m[k], n[k], r[k]]
for minibatch x:
    X = reshape(x, [batch,n[1],...,n[d]])
    Y = sequential_tt_contract(W, X)
    y = reshape(Y, [batch,prod(m)]) + bias
    loss = criterion(activation(y), target)
    backprop(loss); optimizer.step()
    if step % R == 0:
        for k in 1,...,d-1:
            form kth TT unfolding
            compute singular values sigma[k,q]
            choose smallest r[k] satisfying
                sum(q > r[k]) sigma[k,q]^2 <= epsilon[k]^2
        replace W by TT-SVD-rounded cores

Use both a relative tolerance and a maximum rank cap. If rounding changes the output too much, increase the tolerance or round less frequently. A dense bias can be retained initially so that only the weight compression is tested.

(3) Computed directly from the mathematics: TT contractions, unfolding SVDs, rank selection, TT parameter counts, and the Frobenius truncation budget. Estimated empirically: rank growth during training, output perturbation caused by rounding, backward cost, wall-clock time, peak memory, and validation error. Log every TT bond rank and compare the measured output error against the SVD tolerance.

(4) First cheap experiment: train a 6-layer MLP on CIFAR-10 with one 1024-by-1024 hidden layer replaced by TTLinear. Factor both dimensions into five modes of size 4, test TT ranks 4, 8, and 16, and round every 50 steps with relative tolerances 1e-4, 1e-3, and 1e-2. Compare against the dense model and a conventional low-rank matrix-factorized layer at matched parameter counts. Measure parameters, peak GPU memory, forward/backward time, validation accuracy, and loss at equal FLOPs. Success means at least 4x lower layer memory with less than 1 percentage-point accuracy loss and no severe slowdown; failure is rank saturation, rounding overhead exceeding contraction savings, or materially worse optimization.