Elimination-compensation pruning for fully-connected neural networks

23/2026

Elimination-compensation pruning for fully-connected neural networks

Wednesday 4th March 2026

Ballini, E.; Muscarnera, L.; Fumagalli, A.; Scotti, A.; Regazzoni, F.

The unmatched ability of deep neural networks to capture complex patterns in large and noisy datasets is often associated with their large hypothesis space and the vast number of parameters characterizing modern architectures. Pruning techniques have emerged as effective tools to extract sparse representations of neural network parameters while preserving accuracy. However, a fundamental assumption behind pruning is that expendable weights have a small impact on the network error, whereas highly important weights exert a larger influence on inference. We argue that this idea could be generalized; what if a weight is not simply removed but also compensated with a perturbation of the adjacent bias, which does not contribute to the network sparsity? Our work introduces a novel pruning method in which the importance measure of each weight is computed considering the output behavior after an optimal perturbation of its adjacent bias. These perturbations can be then applied directly after the removal of each weight, independently of each other. After deriving analytical expressions for the aforementioned quantities, numerical experiments are conducted to benchmark this technique against some of the most popular pruning strategies, demonstrating an intrinsic efficiency of the proposed approach in very diverse machine learning scenarios.