Denoising Noisy Neural Networks: A Bayesian Approach With Compensation

This article studies a fundamental problem of NoisyNNs: how to reconstruct the DNN weights from their noisy manifestations. While prior works relied exclusively on the maximum likelihood (ML) estimation, this article puts forth a denoising approach to reconstruct DNNs with the aim of maximizing the inference accuracy of the reconstructed models. The superiority of our denoiser is rigorously proven in two small-scale problems, wherein we consider a quadratic neural network function and a shallow feedforward neural network, respectively. When applied to advanced learning tasks with modern DNN architectures, our denoiser exhibits significantly better performance than the ML estimator. Consider the average test accuracy of the denoised DNN model versus the weight variance to noise power ratio (WNR) performance. When denoising a noisy ResNet34 model arising from noisy inference, our denoiser outperforms ML estimation by up to 4.1 dB to achieve a test accuracy of 60%. When denoising a noisy ResNet18 model arising from noisy training, our denoiser outperforms ML estimation by 13.4 dB and 8.3 dB to achieve test accuracies of 60% and 80%, respectively.
Source: IEEE Transactions on Signal Processing - Category: Biomedical Engineering Source Type: research