13 Practical Considerations In the preceding chapters, we presented several approaches for the estimation of the independent component analysis (ICA) model. In particular, several algorithms were proposed for the estimation of the basic version of the model, which has a square mixing matrix and no noise. Now we are, in principle, ready to apply those algorithms on real data sets. Many such applications will be discussed in Part IV. However, when applying the ICA algorithms to real data, some practical con- siderations arise and need to be taken into account. In this chapter, we discuss different problems that may arise, in particular, overlearning and noise in the data. We also propose some preprocessing techniques (dimension reduction by principal component analysis, time filtering) that may be useful and even necessary before the application of the ICA algorithms in practice. 13.1 PREPROCESSING BY TIME FILTERING The success of ICA for a given data set may depend crucially on performing some application-dependent preprocessing steps. In the basic methods discussed in the previous chapters, we always used centering in preprocessing, and often whitening was done as well. Here we discuss further preprocessing methods that are not necessary in theory, but are often very useful in practice. 263 Independent Component Analysis. Aapo Hyv ¨ arinen, Juha Karhunen, Erkki Oja Copyright  2001 John Wiley & Sons, Inc. ISBNs: 0-471-40540-X (Hardback); 0-471-22131-7 (Electronic) 264 PRACTICAL CONSIDERATIONS 13.1.1 Why time filtering is possible In many cases, the observed random variables are, in fact, time signals or time series, which means that they describe the time course of some phenomenon or system. Thus the sample index in is a time index. In such a case, it may be very useful to filter the signals. In other words, this means taking moving averages of the time series. Of course, in the ICA model no time structure is assumed, so filtering is not always possible: If the sample points cannot be ordered in any meaningful way with respect to , filtering is not meaningful, either. For time series, any linear filtering of the signals is allowed, since it does not change the ICA model. In fact, if we filter linearly the observed signals to obtain new signals, say , the ICA model still holds for , with the same mixing matrix. This can be seen as follows. Denote by the matrix that contains the observations as its columns, and similarly for . Then the ICA model can be expressed as: (13.1) Now, time filtering of corresponds to multiplying from the right by a matrix, let us call it .Thisgives (13.2) which shows that the ICA model still remains valid. The independent components are filtered by the same filtering that was applied on the mixtures. They are not mixed with each other in because the matrix is by definition a component-wise filtering matrix. Since the mixing matrix remains unchanged, we can use the filtered data in the ICA estimating method only. After estimating the mixing matrix, we can apply the same mixing matrix on the original data to obtain the independent components. The question then arises what kind of filtering could be useful. In the following, we consider three different kinds of filtering: high-pass and low-pass filtering, as well as their compromise. PREPROCESSING BY TIME FILTERING 265 13.1.2 Low-pass filtering Basically, low-pass filtering means that every sample point is replaced by a weighted average of that point and the points immediately before it. 1 This is a form of smoothing the data. Then the matrix in (13.2) would be something like . . . . . . (13.3) Low-pass filtering is often used because it tends to reduce noise. This is a well- known property in signal processing that is explained in most basic signal processing textbooks. In the basic ICA model, the effect of noise is more or less neglected; see Chapter 15 for a detailed discussion. Thus basic ICA methods work much better with data that does not have much noise, and reducing noise is thus useful and sometimes even necessary. A possible problem with low-pass filtering is that it reduces the information in the data, since the fast-changing, high-frequency features of the data are lost. It often happens that this leads to a reduction of independence as well (see next section). 13.1.3 High-pass filtering and innovations High-pass filtering is the opposite of low-pass filtering. The point is to remove slowly changing trends from the data. Thus a low-pass filtered version is subtracted from the signal. A classic way of doing high-pass filtering is differencing, which means replacing every sample point by the d ifference between the value at that point and the value at the preceding point. Thus, the matrix in (13.2) would be . . . . . . (13.4) 1 To have a causal filter, points after the current point may be left out of the averaging. 266 PRACTICAL CONSIDERATIONS High-pass filtering may be useful in ICA because in certain cases it increases the independence of the components. It often happens in practice that the compo- nents have slowly changing trends or fluctuations, in which case they are not very independent. If these slow fluctuations are removed b y high-pass filtering the fil- tered components are often much more independent. A more principled approach to high-pass filtering is to consider it in the light of innovation processes. Innovation processes Given a stochastic process , we define its innovation process as the error of the best prediction of , g iven its past. Such a best prediction is given by the conditional expectation of given its past, because it is the expected value of the conditional distribution of given its past. Thus the innovation process of is defined by (13.5) The expression “innovation” describes the fact that contains all the new infor- mation about the process that can be obtained at time by observing . The concept of innovations can be utilized in the estimation of the ICA model due to the following property: Theorem 13.1 If and follow the basic ICA model, then the innovation processes and follow the ICA model as well. In particular, the components are independent from each other. On the other hand, independence of the innovations does not imply the indepen- dence of the . Thus, the innovations are more often independent from each other than the original processes. Moreover, one could argue that the innovations are usually more nongaussian than the original processes. This is because the is a kind of moving average of the innovation process, and sums tend to be more gaussian than the original variable. Together these mean that the innovation process is more susceptible to be independent and nongaussian, and thus to fulfill the basic assumptions in ICA. Innovation processes were discussed in more detail in [194], where it was also shown that using innovations, it is possible to separate signals (images of faces) that are otherwise strongly correlated and very difficult to separate. The connection between innovations and ordinary filtering techniques is that the computation of the innovation process is often rather similar to high-pass filtering. Thus, the arguments in favor of using innovation processes apply at least partly in favor of high-pass filtering. A possible problem with high-pass filtering, however, is that it may increase noise for the same reasons that low-pass filtering decreases noise. 13.1.4 Optimal filtering Both of the preceding types of filtering have their pros and cons. The optimum would be to find a filter that increases the independence of the components while reducing PREPROCESSING BY PCA 267 noise. To achieve this, some compromise between high- and low-pass filtering may be the b est solution. This leads to band-pass filtering, in which the highest and the lowest frequencies are filtered out, leaving a suitable frequency band in between. What this band should be depends on the data and general answers are impossible to give. In addition to simple low-pass/high-pass filtering, one might also u se more so- phisticated techniques. For example, one might take the (1-D) wavelet transforms of the data [102, 290, 17]. Other time-frequency decompositions could be used as well. 13.2 PREPROCESSING BY PCA A common preprocessing technique for multidimensional data is to reduce its dimen- sion by principal component analysis (PCA). PCA was explained in more detail in Chapter 6. Basically, the data is projected linearly onto a subspace (13.6) so that the maximum amount of information (in the least-squares sense) is preserved. Reducing dimension in this way has several benefits which we discuss in the next subsections. 13.2.1 Making the mixing matrix square First, let us consider the case where the the number of independent components is smaller than the number of mixtures, say . Performing ICA on the mixtures directly can cause big problems in such a case, since the basic ICA model does not hold anymore. Using PCA we can reduce the dimension of the data to . After such a reduction, the number of mixtures and I Cs are equal, the mixing matrix is square, and the basic ICA model holds. The question is whether PCA is able to find the subspace correctly, so that the ICs can be estimated from the reduced mixtures. This is not true in general, but in a special case it turns out to be the case. If the data consists of ICs only, with no noise added, the whole data is contained in an -dimensional subspace. Using PCA for dimension reduction clearly finds this -dimensional subspace, since the eigenvalues corresponding to that subspace, and only those eigenvalues, are nonzero. Thus reducing dimension with PCA works correctly. In practice, the data is usually not exactly contained in the subspace, due to noise and other factors, but if the noise level is low, PCA still finds approximately the right subspace; see Section 6.1.3. In the general case, some “weak” ICs may be lost in the dimension reduction process, but PCA may still be a good idea for optimal estimation of the “strong” ICs [313]. Performing first PCA and then ICA has an interesting interpretation in terms of factor analysis. In factor analysis, it is conventional that after finding the factor subspace, the actual basis vectors for that subspace are determined by some criteria 268 PRACTICAL CONSIDERATIONS that make the mixing matrix as simple as possible [166]. This is called factor rotation. Now, ICA can be interpreted as one method for determining this factor rotation, based on higher-order statistics instead of the structure o f the mixing matrix. 13.2.2 Reducing noise and preventing overlearning A well-known benefit of reducing the dimension of the data is that it reduces noise, as was already discussed in Chapter 6. Often, the dimensions that have been omitted consist mainly of noise. This is especially true in the case where the number of ICs is smaller than the number of mixtures. Another benefit of reducing dimensions is that it prevents overlearning, to which the rest of this subsection is devoted. Overlearning means that if the number of parameters in a statistical model is too large when compared to the number of available data points, the estimation of the parameters becomes difficult, maybe impossible. The estimation of the parameters is then too much determined by the available sample points, instead of the actual process that generated the data, which is what we are really interested in. Overlearning in ICA [214] typically produces estimates of the ICs that have a single spike or bump, and are practically zero everywhere else. This is because in the space of source signals of unit variance, nongaussianity is more or less maximized by such spike/bump signals. This becomes easily comprehensible if we consider theextremecasewherethesamplesize equals the dimension of the data ,and these are both equal to the number of independent components . Let us collect the realizations of as the columns of the matrix , and denote by the corresponding matrix of the realizations of , as in (13.1). Note that now all the matrices in (13.1) are square. This means that by changing the values of (and keeping fixed), we can give any values whatsoever to the elements of .Thisis a case of serious overlearning, not unlike the classic case of regression with equal numbers of data points and parameters. Thus it is clear that in this case, the estimate of that is obtained by ICA estimation depends little on the observed data. Let us assume that the densities of the source signals are known to be supergaussian (i.e., positively kurtotic). Then the ICA estimation basically consists of finding a separating matrix that maximizes a measure of the supergaussianities (or sparsities) of the estimates of the source signals. Intuitively, it is easy to see that sparsity is maximized when the source signals each have only one nonzero point. Thus we see that ICA estimation with an insufficient sample size leads to a form of overlearning that gives artifactual (spurious) source signals. Such source signals are characterized by larg e spikes. An important fact shown experimentally [214] is that a similar phenomenon is much more likely to occur if the source signals are not independently and identically distributed (i.i.d.) in time, but have strong time-dependencies. In such cases the sample size needed to get rid of overlearning is much larger, and the source signals are better characterized by bumps, i.e., low-pass filtered versions of spikes. An intuitive way of explaining this phenomenon is to consider such a signal as being constant on blocks of consecutive sample points. This means that the data can HOW MANY COMPONENTS SHOULD BE ESTIMATED? 269 be considered as really having only sample points; each sample point has simply been repeated times. Thus, in the case of overlearning, the estimation procedure gives “spikes” that have a width of time points, i.e., bumps. Here we illustrate the phenomenon by separation of artificial source signals. Three positively kurtotic signals, with 500 sample points each, were used in these simulations, and are depicted in Fig. 13.1 a. Five hundred mixtures were produced, and a very small amount of gaussian noise was added to each mixture separately. As an example of a successful ICA estimation, Fig. 13.1 b shows the result of applying the FastICA and maximum likelihood (ML) gradient ascent algorithms (denoted by “Bell-Sejnowski”) to the mixed signals. In both approaches, the prepro- cessing (whitening) stage included a dimension reduction of the data into the first three principal components. It is evident that both algorithms are able to extract all the i nitial signals. In contrast, when the whitening is made with very small dimension reduction (we took 400 dimensions), we see the emergence of spiky solutions (like Dirac functions), which is an extreme case of kurtosis maximization (Fig. 13.1 c). The algorithm used in FastICA was of a d eflationary type, from which we plot the first five components extracted. As for the ML gradient ascent, which was of a symmetric type, we show five representative solutions to the 400 extracted. Thus, we see here that without dimension reduction, we are not able to estimate the source signals. Fig. 13.1 d presents an intermediate stage of dimension reduction (from the original 500 mixtures we took 50 whitened vectors). We see that the actual source signals are revealed by both methods, even though each resulting vector is more noisy than the ones shown in Fig. 13.1 b. For the final example, in Fig. 13.1 e, we low-pass filtered the mixed signals, prior to the independent component analysis, using a 10 delay moving average filter. Taking the same amount of principal components as in d,wecanseethatweloseallthe original source signals: the decompositions show a bumpy structure corresponding to the low-pass filtering of the spiky outputs presented in c. Through low-pass filtering, we have reduced the information contained in the data, and so the estimation is rendered impossible even with this, not very weak, dimension reduction. Thus, we see that with this low-pass filtered data, a much stronger dimension reduction by PCA is necessary to prevent overlearning. In addition to PCA, some kind of prior information on the mixing matrix could be useful in preventing overlearning. This is considered in detail in Section 20.1.3. 13.3 HOW MANY COMPONENTS SHOULD BE ESTIMATED? Another problem that often arises in practice is to decide the number of ICs to be estimated. This problem does not arise if one simply estimates the same number of components as the dimension of the data. This may not always be a good idea, however. 270 PRACTICAL CONSIDERATIONS (a) (b) Fast ICA Bell−Sejnowski (c) (d) (e) Fig. 13.1 (From [214]) I llustration o f the importance of the de gree of dimension reduction and filtering in artificially generated data, using FastICA and a gradient algorithm for ML estimation. (a) Original positively kurtotic signals. (b) ICA decomposition in which the preprocessing includes a dimension reduction to the first 3 principal components. (c) Poor, i.e., too weak dimension reduction. (d) Decomposition using an intermediate dimension reduction ( 50 components r etained). (e) Same results as in (d) but using low-pass filtered mixtures CHOICE OF ALGORITHM 271 First, since dimension reduction by PCA is often necessary, one must choose the number of principal components to be retained. This is a classic p roblem; see Chapter 6. It is usually solved by choosing the minimum number of principal components that explain the data well enough, containing, for example, of the variance. Often, the dimension is actually chosen by trial and error with no theoretical guidelines. Second, for computational reasons we may prefer to estimate only a smaller number of ICs than the dimension of the data (after PCA preprocessing). This is the case when the dimension of the data is very large, and we do not want to reduce the dimension by PCA too much, since PCA always contains the risk of not including the ICs in the reduced data. Using FastICA and other algorithms that allow estimation of a smaller number of components, we can thus perform a kind o f dimension reduction by ICA. In fact, this is an idea somewhat similar to projection pursuit. Here, it is even more difficult to give any guidelines as to how many components should be estimated. Trial and error may be the only method applicable. Information-theoretic, Bayesian, and other criteria for determining the number of ICs are discussed in more detail in [231, 81, 385]. 13.4 CHOICE OF ALGORITHM Now we shall briefly discuss the choice of ICA algorithm from a practical viewpoint. As will be discussed in detail in Chapter 14, most estimation principles and objectiv e functions for ICA are equivalent, at least in theory. So, the main choice is reduced to a couple of points: One choice is between estimating all the independent components in parallel, or just estimating a few of them (possibly one-by-one). This corresponds to choosing between symmetric and hierarchical decorrelation. In most cases, symmetric decorrelation is recommended. Deflation is mainly useful in the case where we want to estimate only a very limited number of ICs, and other special cases. The disadvantage with deflationary orthogonalization is that the estimation errors in the components that are estimated first accumulate and increase the errors in the later components. One must also choose the nonlinearity used in the algorithms. It seems that the robust, nonpolynomial nonlinearities are to be preferred in most applications. Thesimplestthingtodoistojustusethe function as the nonlinearity . This is sufficient when using FastICA. (When using gradient algorithms, especially in the ML framework, a second function needs to be used as well; see Chapter 9.) Finally, there is the choice between on-line and batch algorithms. In most cases, the whole data set is available before the estimation, which is called in different contexts batch, block, or off-line estimation. This is the case where FastICA can be used, and it is the algorithm that we recommend. On- 272 PRACTICAL CONSIDERATIONS line or adaptive algorithms are needed in signal-processing applications where the mixing matrix may change on-line, and fast tracking is needed. In the on-line case, the recommended algorithms are those obtained by stochastic gradient methods. It should also be noted that in some cases, the FastICA algorithm may not converge well as Newton-type algorithms sometimes exhibit oscillatory behavior. This problem can be alleviated by using gradient methods, or combinations of the two (see [197]). 13.5 CONCLUDING REMARKS AND REFERENCES In this chapter, we considered some practical problems in ICA. When dealing with time signals, low-pass filtering of the data is useful to reduce noise. On the other hand, high-pass filtering, or computing innovation processes is useful to increase the independence and nongaussianity of the components. One of these, or their combi- nation may be very useful in practice. Another very useful thing to do is to reduce the dimension of the data by PCA. This reduces noise and prevents overlearning. It may also solve the problems with data that has a smaller number of ICs than mixtures. Problems 13.1 Take a Fourier transform on every observed signal . Does the ICA model still hold, and in what way? 13.2 Prove the theorem on innovations. Computer assignments 13.1 Take a gaussian white noise sequence. Low-pass filter it by a low-pass filter with coefficients ( ,0,0,1,1,1,1,1,0,0,0, ). What does the signal look like? 13.2 High-pass filter the gaussian white noise sequence. What does the signal look like? 13.3 Generate 100 samples of 100 independent components. Run FastICA on this data without any mixing. What do the estimated ICs look like? Is the estimate o f the mixing matrix close to identity? . signals, prior to the independent component analysis, using a 10 delay moving average filter. Taking the same amount of principal components as in d,wecanseethatweloseallthe original. not necessary in theory, but are often very useful in practice. 263 Independent Component Analysis. Aapo Hyv ¨ arinen, Juha Karhunen, Erkki Oja Copyright 