Prune redundant attention heads using separate similarity scores for sink behavior and content routing. Two heads are considered safely redundant only when their normalized content compositions are close in Aitchison distance and their sink-mass trajectories are also close, avoiding pruning decisions dominated by a shared sink token.
Use trajectory sensitivities to remove neural units or parameter groups whose effects are redundant over the available data support. A parameter group is pruned when its Fisher contribution is small or its sensitivity is nearly collinear with other groups, producing a compact neural ODE without relying only on parameter magnitude.
Replace magnitude pruning in a trained recurrent network with stochastic pruning probabilities computed from weight magnitudes and the covariance of neuron activities under injected noise. Connections whose endpoints fluctuate in a sign-compatible way receive higher retention probability, while retained weights are rescaled to preserve average recurrent strength. The method uses local weights and activity covariance, avoiding Hessian construction and expensive global saliency optimization.
Replace fixed PCA-rank selection in a hidden layer with a renormalization-group-inspired gate over covariance eigenvalue bands. The gate retains modes whose effective quartic interaction remains unstable or strongly scale-dependent, while pruning bands that flow toward the Gaussian noise fixed point. Unlike top-eigenvalue truncation, this is designed for extensive-rank signal distributed throughout the bulk spectrum.
Replace parameter-count or validation-loss-only model selection with a singular-complexity score based on the evidence scaling of each candidate neural network. Select or prune architectures using \(n\widehat L_n+\widehat\lambda\log n-(\widehat m-1)\log\log n\), which can prefer overparameterized but highly redundant networks when their effective singular complexity is lower.
Replace a collection of dense task-specific linear layers with a common sparse structural matrix and task-specific edge strengths. All tasks share the same learned connectivity pattern, but retain independent values on active connections, allowing parameter sharing without forcing identical interactions.
Replace a neural layer's dense parameters by a real parameter tensor and a binary activity mask, then update the mask using importance scores measured across multiple perturbation intervals. Gates that repeatedly occur in non-dominated accuracy–cost configurations receive larger scores and are more likely to be retained or activated, avoiding uniformly random sparse rewiring.