The organization of this book is based upon Marr's (1982) theory of complex information processing systems. Marr (1982) suggested that any complex information processing system could be described using at least three levels of description. The implementational level which describes the mechanisms. The algorithmic level which describes the algorithms which are instantiated by the mechanisms. And the computational level which describes the computational goals (and justifies those goals) of the algorithms.
Implementational Level: Dynamical Systems Theory. Chapters 2, 3, and 4 of the book are concerned with mathematical tools relevant for understanding the implementational level of description of artificial neural networks. These chapters describe LaSalle's Invariant Set Theorem for the analysis of high-dimensional nonlinear dynamical systems and Stochastic Approximation Theory for analyzing high-dimensional nonlinear stochastic dynamical systems.
Algorithmic Level: Optimization Theory. Chapter 5 (Optimization Theory) is concerned with understanding the algorithmic level of description of artificial neural networks. In Chapter 5, classical and powerful methods from the field of nonlinear optimization theory are used to compare, contrast, analyze, and design high-dimensional nonlinear dynamical systems which have the computational goal of optimizing some performance functoin.
Computational Level: Rational Inference Measures and Statistical Pattern Recognition. In Chapter 6, the mathematical theories of evidence and rational decision making were shown to be useful for understanding in what sense computing the minimum value of an expected risk function is a rational decision which yields appropriate generalizations. Chapter 7 shows how probabilistic interpretations of artificial neural networks can be constructed for most artificial neural networks. Chapter 7 also shows how such interpretations may be used to analyze and design good learning algorithms for artificial neural networks. Chapter 8 shows how to characterize precisely the generalization errors which will be made by a given artificial neural network in a particular statistical environment. Chapter 8 also provides principled methods for selecting the best neural network architecture for a given statistical environment.