Entropy-Neutral Stream Mixing for Rate-Aware Networks

Use a learned image or language autoencoder with an explicit latent bottleneck. Let Z have shape [batch, latent_positions, S, d], apply A across the S axis immediately before scalar quantization, and use the same entropy-model architecture for the mixed latent Y and the baseline latent Z. Build A from Sinkhorn-normalized logits and initialize it near the identity so the rate comparison is not confounded by a large initial representation change. During training replace rounding with additive uniform noise: y_soft=Y+U, with each component of U sampled independently from Uniform(-0.5,0.5). At validation use round(Y), estimate discrete likelihood by integrating the learned density over each quantization bin, and optionally arithmetic-code the result. Minimize task reconstruction or classification loss plus beta times the summed negative log2 likelihood from the rate formula. Sweep beta across at least five values and report rate-distortion or rate-accuracy curves. Compute the ideal entropy change analytically as d*log(abs(det(A))+epsilon), and separately measure empirical differential entropy with a k-nearest-neighbor estimator and actual cross-entropy under p_psi; do not assume the theorem implies lower finite-model rate. Compare no mixing, fixed random permutation mixing, unconstrained learned mixing, and doubly stochastic mixing. Start with a CIFAR-10 convolutional autoencoder or a 2-layer transformer bottleneck targeting 0.1-1.0 bits per latent scalar. Success is lower measured bits at equal reconstruction PSNR or classification accuracy, or higher accuracy at equal bits, with no increase in latent power.