A new MOX Report entitled “Elimination-compensation pruning for fully-connected neural networks” by Ballini, E.; Muscarnera, L.; Fumagalli, A.; Scotti, A.; Regazzoni, F. has appeared in the MOX Report Collection.
Check it out here: https://www.mate.polimi.it/biblioteca/add/qmox/23-2026.pdf
Abstract: The unmatched ability of deep neural networks to capture complex patterns in large and noisy datasets is often associated with their large hypothesis space and the vast number of parameters characterizing modern architectures. Pruning techniques have emerged as effective tools to extract sparse representations of neural network parameters while preserving accuracy. However, a fundamental assumption behind pruning is that expendable weights have a small impact on the network error, whereas highly important weights exert a larger influence on inference. We argue that this idea could be generalized; what if a weight is not simply removed but also compensated with a perturbation of the adjacent bias, which does not contribute to the network sparsity? Our work introduces a novel pruning method in which the importance measure of each weight is computed considering the output behavior after an optimal perturbation of its adjacent bias. These perturbations can be then applied directly after the removal of each weight, independently of each other. After deriving analytical expressions for the aforementioned quantities, numerical experiments are conducted to benchmark this technique against some of the most popular pruning strategies, demonstrating an intrinsic efficiency of the proposed approach in very diverse machine learning scenarios.
