A deterministic algorithm that emulates learning with random weights

Abstract

The expectation of a function of random variables can be modeled as the value of the function in the mean value of the variables plus a penalty term. Here, this penalty term is calculated exactly, and the properties of different approximations are analyzed. Then, a deterministic algorithm for minimizing the expected error of a feedforward network of random weights is presented. Given a particular feedforward network architecture and a training set, this algorithm accurately finds the weight configuration that makes the network response most resistant to a class of weight perturbations. Finally, the study of the most stable configurations of a network unravels some undesirable properties of networks with asymmetric activation functions.