Averaging over networks: properties, evaluation, and minimization

Abstract

The expectation of random variables is modelled as the value of the function in the mean value of the variables plus a penalty term. This penalty term is calculated exactly, and the properties of different approximations are analyzed. In particular, for quadratic functions, the penalty term is shown to have a surprised and an unsupervised part. Then, two algorithms for minimizing the expected error of a feedfoward network of random weights are compared. One of them is stochastical, while the other is deterministic and more flexible. give a particular feedforward network architecture and training set, the deterministic algorithm here presented accurately finds the weight configuration that makes the network reponse most resistant to a class of weight perturbations. Finally, the study of most stable configurations of a network unravels some undesirable properties of network with asymmetric activation functions.