... vector and matrixgradients are reviewed first, followed by the most typical ways to solve unconstrained and constrained optimization problems with gradient-type learning algorithms.3.1 VECTOR AND ... constraints on thesolutions, is the subject of optimization theory. In this chapter, we discuss sometypical iterative optimization algorithms and their properties. Mostly, the algorithmsare ... possible ICA cost functions willbe given and discussed in detail in Parts II and III of this book. In general, statisticalestimation is largely based on optimization of cost or objective functions,...