Part of Advances in Neural Information Processing Systems 9 (NIPS 1996)
Volker Tresp, Ralph Neuneier, Hans-Georg Zimmermann
Optimal Brain Damage (OBD) is a method for reducing the num(cid:173) ber of weights in a neural network. OBD estimates the increase in cost function if weights are pruned and is a valid approximation if the learning algorithm has converged into a local minimum. On the other hand it is often desirable to terminate the learning pro(cid:173) cess before a local minimum is reached (early stopping). In this paper we show that OBD estimates the increase in cost function incorrectly if the network is not in a local minimum. We also show how OBD can be extended such that it can be used in connec(cid:173) tion with early stopping. We call this new approach Early Brain Damage, EBD. EBD also allows to revive already pruned weights. We demonstrate the improvements achieved by EBD using three publicly available data sets.