A. Murat Tekalp. “Image and Video Restoration.” 2000 CRC Press LLC. <http://www.engnetbase.com>. ImageandVideoRestoration A.MuratTekalp UniversityofRochester 53.1Introduction 53.2Modeling Intra-FrameObservationModel • MultispectralObserva- tionModel • MultiframeObservationModel • Regularization Models 53.3ModelParameterEstimation BlurIdentification • EstimationofRegularizationParameters • EstimationoftheNoiseVariance 53.4Intra-FrameRestoration BasicRegularizedRestorationMethods • RestorationofIm- agesRecordedbyNonlinearSensors • RestorationofImages DegradedbyRandomBlurs • AdaptiveRestorationforRing- ingReduction • BlindRestoration(Deconvolution) • Restora- tionofMultispectralImages • RestorationofSpace-Varying BlurredImages 53.5MultiframeRestorationandSuperresolution MultiframeRestoration • Superresolution • Superresolution withSpace-VaryingRestoration 53.6Conclusion References 53.1 Introduction Digitalimagesandvideo,acquiredbystillcameras,consumercamcorders,orevenbroadcast-quality videocameras,areusuallydegradedbysomeamountofblurandnoise.Inaddition,mostelectronic camerashavelimitedspatialresolutiondeterminedbythecharacteristicsofthesensorarray.Common causesofblurareout-of-focus,relativemotion,andatmosphericturbulence.Noisesourcesinclude filmgrain,thermal,electronic,andquantizationnoise.Further,manyimagesensorsandmediahave knownnonlinearinput-outputcharacteristicswhichcanberepresentedaspointnonlinearities.The goalofimageandvideo(imagesequence)restorationistoestimateeachimage(frameorfield)asit wouldappearwithoutanydegradations,byfirstmodelingthedegradationprocess,andthenapplying aninverseprocedure.Thisisdistinctfromimageenhancementtechniqueswhicharedesignedto manipulateanimageinordertoproducemorepleasingresultstoanobserverwithoutmaking useofparticulardegradationmodels.Ontheotherhand,superresolutionreferstoestimatingan imageataresolutionhigherthanthatoftheimagingsensor.Imagesequencefiltering(restoration andsuperresolution)becomesespeciallyimportantwhenstillimagesfromvideoaredesired.This isbecausetheblurandnoisecanbecomeratherobjectionablewhenobservinga“freeze-frame”, althoughtheymaynotbevisibletothehumaneyeattheusualframerates.Sincemanyvideosignals encounteredinpracticeareinterlaced,weaddressthecasesofbothprogressiveandinterlacedvideo. c  1999byCRCPressLLC The problemofimagerestorationhas sparked widespread interestin the signal processing commu- nity over the past 20 or 30 years. Because image restoration is essentially an ill-posed inverse problem which is also frequentlyencountered in various other disciplines such as geophysics, astronomy, med- ical imaging, and computer vision, the literature that is related to image restoration is abundant. A concise discussion of early results can be found in the books by Andrews and Hunt [1] and Gonzalez and Woods [2]. More recent developments are summarized in the book by Katsaggelos [3], and re- view papers by Meinel [4], Demoment [5], Sezan and Tekalp [6], and Kaufman and Tekalp [7]. Most recently, printing high-quality still images from video sources has become an important application for multi-frame restoration and superresolution methods. An in-depth coverage of video filtering methods can be found in the book Digital Video Processing by Tekalp [8]. This chapter summarizes key results in digital image and video restoration. 53.2 Modeling Every image restoration/superresolution algorithm is based on an observation model, which relates the observed degraded image(s) to the desired “ideal” image, and possibly a regularization model, which conveys the available a priori information about the ideal image. The success of image restora- tion and/or superresolution depends on how good the assumed mathematical models fit the actual application. 53.2.1 Intra-Frame Observation Model Let the observed and ideal images be sampled on the same 2-D lattice . Then, the observed blurred and noisy image can be modeled as g = s(Df ) + v (53.1) where g, f , and v denote vectors representinglexicographical ordering of the samples of the observed image, ideal image, and a particular realization of the additive (random) noise process, respectively. The operator D is called the blur operator. The response of the image sensor to light intensity is represented by the memoryless mapping s(·), which is, in general, nonlinear. (This nonlinearity has often been ignored in the literature for algorithm development.) The blur may be space-invariant or space-variant. For space-invariant blurs, D becomesaconvo- lution operator, which has block-Toeplitz structure; and Eq. (53.1) can be expressed, in scalar form, as g ( n 1 ,n 2 ) = s    (m 1 ,m 2 )∈ S d d ( m 1 ,m 2 ) f ( n 1 − m 1 ,n 2 − m 2 )   + v ( n 1 ,n 2 ) (53.2) where d(m 1 ,m 2 ) and S d denote the kernel and support of the operator D, respectively. The kernel d(m 1 ,m 2 ) is the impulse response of the blurring system, often called the point spread function (PSF). In case of space-variant blurs, the operator D does not have a particular structure; and the observation equation can be expressed as a superposition summation g ( n 1 ,n 2 ) = s    ( m 1 ,m 2 ) ∈ S d ( n 1 ,n 2 ) d ( n 1 ,n 2 ; m 1 ,m 2 ) f ( m 1 ,m 2 )   + v ( n 1 ,n 2 ) (53.3) where S d (n 1 ,n 2 ) denotes the support of the PSF at the pixel location (n 1 ,n 2 ). The noise is usually approximated by a zero-mean, white Gaussian random field which is additive and independent of the image signal. In fact, it has been generally accepted that more sophisticated noise models do not, in general, lead to significantly improved restorations. c  1999 by CRC Press LLC 53.2.2 Multispectral Observation Model Multispectral images refer to image data with multiple spectral bands that exhibit inter-band cor- relations. An important class of multispectral images are color images with three spectral bands. Suppose we have K spectral bands, each blurred by possibly a different PSF. Then, the vector-matrix model (53.1) can be extended to multispectral modeling as g = Df + v (53.4) where g . =    g 1 . . . g K    , f . =    f 1 . . . f K    , v . =    v 1 . . . v K    denote N 2 K × 1 vectors representing the multispectral observed, ideal, and noise data, respectively, stacked as composite vectors, and D . =    D 11 ··· D 1K . . . . . . . . . D K1 ··· D KK    is an N 2 K × N 2 K matrix representing the multispectral blur operator. In most applications, D is block diagonal, indicating no inter-band blurring. 53.2.3 Multiframe Observation Model Suppose a sequence of blurred and noisy images g k (n 1 ,n 2 ), k = 1, .,L, corresponding to multiple shots (from different angles) of a static scene sampled on a 2-D lattice or frames (fields) of video sampled (at different times) on a 3-D progressive (interlaced) lattice, is available. Then, we may be able to estimate a higher-resolution “ideal” still image f(m 1 ,m 2 ) (corresponding to one of the observed frames) sampled on a lattice, which has a higher sampling density than that of the input lattice. The main distinction between the multispectral and multiframe observation models is that here the observed images are subject to sub-pixel shifts (motion), possibly space-varying, which makes high-resolution reconstruction possible. In the case of video, we may also model blurring due to motion within the aperture time to further sharpen images. To this effect, each observed image (frame or field) can be related to the desired high-resolution ideal still-image through the superposition summation [8] g k ( n 1 ,n 2 ) = s    ( m 1 ,m 2 ) ∈ S d ( n 1 ,n 2 ;k ) d k ( n 1 ,n 2 ; m 1 ,m 2 ) f ( m 1 ,m 2 )   + v k ( n 1 ,n 2 ) (53.5) where the support of the summation over the high-resolution grid (m 1 ,m 2 ) at a particular observed pixel (n 1 ,n 2 ; k) depends on the motion trajectory connecting the pixel (n 1 ,n 2 ; k) to the ideal image, the size of the support of the low-resolution sensor PSF h a (x 1 ,x 2 ) with respect to the high resolution grid, and whether there is additional optical (out-of-focus, motion, etc.) blur. Because the relative positions of low- and high-resolution pixels in general vary by spatial coordinates, the discrete sensor PSFis space-varying. The support ofthe space-varyingPSF isindicated bytheshaded areainFig. 53.1, where the rectangle depicted by solid lines shows the support of a low-resolution pixel over the high- resolution sensor array. The shaded region corresponds to the area swept by the low-resolution pixel due to motion during the aperture time [8]. c  1999 by CRC Press LLC FIGURE 53.1: Illustration of the discrete system PSF. Note that the model (53.5) is invalid in case of occlusion. That is, each observed pixel (n 1 ,n 2 ; k) can be expressed as a linear combination of several desired high-resolution pixels (m 1 ,m 2 ),provided that (n 1 ,n 2 ; k) is connected to (m 1 ,m 2 ) by a motion trajectory. We assume that occlusion regions can be detected a priori using a proper motion estimation/segmentation algorithm. 53.2.4 Regularization Models Restorationisanill-posed problemwhichcanberegularizedbymodelingcertain aspectsofthedesired “ideal” image. Images can be modeled as either 2-D deterministic sequences or random fields. A priori information about the ideal image can then be used to define hard or soft constraints on the solution. In the deterministic case, images are usually assumed to be members of an appropriate Hilbert space, such as a Euclidean space with the usual inner product and norm. For example, in the context of set theoretic restoration, the solution can be restricted to be a member of a set consisting of all images satisfying a certain smoothness criterion [9]. On the other hand, constrained least squares (CLS) and Tikhonov-Miller regularization use quadratic functionals to impose smoothness constraints in an optimization framework. In the random case, models have been developed for the pdf of the ideal image in the context of maximum a posteriori (MAP) image restoration. For example, Trussell and Hunt [10]haveproposed a Gaussian distribution with space-varying mean and stationary covariance as a model for the pdf of the image. Geman and Geman [11] proposed a Gibbs distribution to model the pdf of the image. Alternatively, if the image is assumed to be a realization of a homogeneous Gauss-Markov random process, then it can be statistically modeled through an autoregressive (AR) difference equation [12] f ( n 1 ,n 2 ) =  ( m 1 ,m 2 ) ∈ S c c ( m 1 ,m 2 ) f ( n 1 − m 1 ,n 2 − m 2 ) + w ( n 1 ,n 2 ) (53.6) where {c(m 1 ,m 2 ) : (m 1 ,m 2 ) ∈ S c } denote the model coefficients, S c is the model support (which may be causal, semi-causal, or non-causal), and w(n 1 ,n 2 ) represents the modeling error which is Gaussian distributed. The model coefficients can be determined such that the modeling error has minimum variance [12]. Extensions of (53.6) to inhomogeneous Gauss-Markov fields was proposed by Jeng and Woods [13]. 53.3 Model Parameter Estimation In this section, we discuss methods for estimating the parameters that are involved in the observation and regularization models for subsequent use in the restoration algorithms. c  1999 by CRC Press LLC 53.3.1 Blur Identification Blur identification refers to estimation of both the support and parameters of the PSF {d(n 1 ,n 2 ) : (n 1 ,n 2 ) ∈ S d }. It is a crucial element of image restoration because the quality of restored images is highly sensitive to errors in the PSF [14]. An early approach to blur identification has been based on the assumption that the original scene contains an ideal point source, and that its spread (hence the PSF) can be determined from the observed image. Rosenfeld and Kak [15] show that the PSF can also be determined from an ideal line source. These approaches are of limited use in practice because a scene, in general, does not contain an ideal point or line source and the observation noise may not allow the measurement of a useful spread. Models for certain types of PSF can be derived using principles of optics, if the source of the blur is known [7]. For example, out-of-focus and motion blur PSF can be parameterized with a few parameters. Further, they arecompletelycharacterizedbytheir zerosinthefrequency-domain. Power spectrum andcepstrum (Fouriertransform ofthe logarithm ofthe powerspectrum)analysis methods have been successfully applied in many cases to identify the location of these zero-crossings [16, 17]. Alternatively, Chang et al. [18] proposed a bispectrum analysis method, which is motivated by the fact that bispectrum is not affected, in principle, by the observation noise. However, the bispectral method requires much more data than the method based on the power spectrum. Note that PSFs, which do not have zero crossings in the frequency domain (e.g., Gaussian PSF modeling atmospheric turbulence), cannot be identified by these techniques. Yet another approach for blur identification is the maximum likelihood (ML) estimation approach. The ML approach aims to find those parameter values (including, in principle, the observation noise variance) that have most likely resulted in the observed image(s). Different implementations of the ML image and blur identification are discussed under a unifying framework [19]. Pavlovi ´ c and Tekalp [20] propose a practical method to find the ML estimates of the parameters of a PSF based on a continuous domain image formation model. In multi-frame image restoration, blur identification using more than one frame at a time becomes possible. For example, the PSF of a possibly space-varying motion blur can be computed at each pixel from an estimate of the frame-to-frame motion vector at that pixel, provided that the shutter speed of the camera is known [21]. 53.3.2 Estimation of Regularization Parameters Regularization model parameters aim to strike a balance between the fidelity of the restored image to the observed data and its smoothness. Various methods exist to identify regularization parameters, such as parametric pdf models, parametric smoothness constraints, and AR image models. Some restoration methods require the knowledge of the power spectrum of the ideal image, which can be estimated, forexample, froman AR modelof the image. TheAR parameters can, in turn, be estimated from the observed image by a least squares [22] or an ML technique [63]. On the other hand, non-parametric spectral estimation is also possible through the application of periodogram-based methods toa prototype image [69, 23]. In the context of maximum a posteriori(MAP) methods, the a priori pdf is often modeled by a parametric pdf, such as a Gaussian [10] or a Gibbsian [11]. Standard methods for estimating these parameters do not exist. Methods for estimating the regularization parameter in the CLS, Tikhonov-Miller, and related formulations are discussed in [24]. 53.3.3 Estimation of the Noise Variance Almost all restoration algorithms assume that the observation noise is a zero-mean, white random process that is uncorrelated with the image. Then, the noise field is completely characterized by its variance, which is commonly estimated by the sample variance computed over a low-contrast local c  1999 by CRC Press LLC region of the observed image. As we will see in the following section, the noise variance plays an important role in defining constraints used in some of the restoration algorithms. 53.4 Intra-Frame Restoration Westart byfirstlookingatsomebasicregularizedrestorationstrategies, in the case of an LSI blur model with no pointwise nonlinearity. The effectofthenonlinear mapping s(.)isdiscussedin Section 53.4.2. Methods that allow PSFs with a random components are summarized in Section 53.4.3. Adaptive restoration for ringing suppression and blind restoration are covered in Sections 53.4.4 and 53.4.5, respectively. Restoration of multispectral images and space-varying blurred images are addressed in Sections 53.4.6 and 53.4.7, respectively. 53.4.1 Basic Regularized Restoration Methods When the mapping s(.) is ignored, it is evident from Eq. (53.1) that image restoration reduces to solving a set of simultaneous linear equations. If the matrix D is nonsingular (i.e., D −1 exists) and the vector g lies in the column space of D (i.e., there is no observation noise), then there exists a unique solution which can be found by direct inversion (also known as inverse filtering). In practice, however, we almost always have an underdetermined (due toboundary truncation problem [14]) and inconsistent (due to observation noise) set of equations. In this case, we resort to a minimum-norm least-squares solution. A least squares (LS) solution (not unique when the columns of D are linearly dependent) minimizes the norm-square of the residual J LS (f ) . =||g − Df || 2 (53.7) LS solution(s) with the minimum norm (energy) is (are) generally known as pseudo-inverse solu- tion(s) (PIS). Restoration by pseudo-inversion is often ill-posed owing to the presence of observation noise [14]. This follows because the pseudo-inverse operator usually has some very large eigenvalues. For ex- ample, a typical blur transfer function has zeros; and thus, its pseudo-inverse attains very large magnitudes near these singularities as well as at high frequencies. This results in excessive amplifi- cation at these frequencies in the sensor noise. Regularized inversion techniques attempt to roll-off the transfer function of the pseudo-inverse filter at these frequencies to limit noise amplification. It follows that the regularized inverse deviates from the pseudo-inverse at these frequencies which leads to other types of artifacts, generally known as regularization artifacts [14]. Various strategies for regularized inversion (and how to achieve the right amount of regularization) are discussed in the following. Singular-Value Decomposition Method The pseudo-inverse D + can be computed using the singular value decomposition (SVD) [1] D + = R  i=0 λ −1/2 i z i u T i (53.8) where λ i denote the singular values, z i and u i are the eigenvectors of D T D and DD T , respectively, and R is the rank of D. Clearly, reciprocation of zero singular-values is avoided since the summation runs to R, the rank of D. Under the assumption that D is block-circulant (corresponding to a circular convolution), the PIS computed through Eq. (53.8) is equivalent to the frequency domain c  1999 by CRC Press LLC pseudo-inverse filtering D + (u, v) =  1/D(u, v) if D(u, v) = 0 0 if D(u, v) = 0 (53.9) where D(u, v) denotes the frequency response of the blur. This is because a block-circulant matrix can be diagonalized by a 2-D discrete Fourier transformation (DFT) [2]. Regularization of the PIS can then be achieved by truncating the singular value expansion (53.8) to eliminate all terms corresponding to small λ i (which are responsible for the noise amplification) at the expense of reduced resolution. Truncation strategies are generally ad-hoc in the absence of additional information. Iterative Methods (Landweber Iterations) Several image restoration algorithms are based on variations of the so-called Landweber itera- tions [25, 26, 27, 28, 31, 32] f k+1 = f k + RD T  g − Df k  (53.10) where R is a matrix that controls the rate of convergence of the iterations. There is no general way to select the best C matrix. If the system (53.1) is nonsingular and consistent (hardly ever the case), the iterations (53.10) will converge to the solution. If, on the other hand, (53.1) is underdetermined and/or inconsistent, then (53.10) converges to a minimum-norm least squares solution (PIS). The theory of this and other closely related algorithms are discussed by Sanz and Huang [26] and Tom et al. [27]. Kawata and Ichioka [28] are among the first to apply the Landweber-type iterations to image restoration, which they refer to as “reblurring” method. Landweber-type iterative restoration methods can be regularized by appropriately terminating the iterations before convergence, since the closer we are to the pseudo-inverse, the more noise amplification we have. A termination rule can be defined on the basis of the norm of the residual image signal [29]. Alternatively, soft and/or hard constraints can be incorporated into iterations to achieve regularization. The constrained iterations can be written as [30, 31] f k+1 = C  f k + RD T  g − Df k   (53.11) where C is a nonexpansive constraint operator, i.e., ||C(f 1 ) − C(f 2 )||≤||f 1 − f 2 ||, to guarantee the convergence of the iterations. Application of Eq. (53.11) to image restoration has been extensively studied (see [31, 32] and the references therein). Constrained Least Squares Method Regularizedimage restoration can be formulated as a constrained optimization problem, where a functional ||Q(f )|| 2 of the image is minimized subject to the constraint ||g − Df || 2 = σ 2 .Here σ 2 is a constant, which is usually set equal to the variance of the observation noise. The constrained least squares (CLS) estimate minimizes the Lagrangian [34] J CLS (f ) =||Q(f )|| 2 + α  ||g − Df || 2 − σ 2  (53.12) whereαistheLagrange multiplier. Theoperator QischosensuchthattheminimizationofEq.(53.12) enforces some desired property of the ideal image. For instance, if Q is selected as the Laplacian operator, smoothness of the restored image is enforced. The CLS estimate can be expressed, by taking the derivative of Eq. (53.12) and setting it equal to zero, as [1] ˆ f =  D H D + γ Q H Q  −1 D H g (53.13) c  1999 by CRC Press LLC where H stands for Hermitian (i.e., complex-conjugate and transpose). The parameter γ = 1 α (the regularization parameter) must be such that the constraint ||g − Df || 2 = σ 2 is satisfied. It is often computed iteratively [2]. A sufficient condition for the uniqueness of the CLS solution is that Q −1 exists. For space-invariant blurs, the CLS solution can be expressed in the frequency domain as [34] ˆ F (u, v) = D ∗ (u, v) |D(u, v)| 2 + γ |L(u, v)| 2 G(u, v) (53.14) where ∗ denotescomplexconjugation. A closely relatedregularization methodis the Tikhonov-Miller (T-M) regularization [33, 35]. T-M regularization has been applied to image restoration [31, 32, 36]. Recently, neural network structures implementing the CLS or T-M image restoration have also been proposed [37, 38]. Linear Minimum Mean Square Error Method The linear minimum mean square error (LMMSE) method finds the linear estimate which minimizes the mean square error between the estimate and ideal image, using up to second order statistics of the ideal image. Assuming that the ideal image can be modeled by a zero-mean homoge- neous random field and the blur is space-invariant, the LMMSE (Wiener) estimate, in the frequency domain, is given by [8] ˆ F (u, v) = D ∗ (u, v) |D(u, v)| 2 + σ 2 v /|P (u, v)| 2 G(u, v) (53.15) where σ 2 v is the variance of the observation noise (assumed white) and |P (u, v)| 2 stands for the power spectrum of the ideal image. The power spectrum of the ideal image is usually estimated from a prototype. It can be easily seen that the CLS estimate (53.14) reduces to the Wiener estimate by setting |L(u, v)| 2 = σ 2 v /|P (u, v)| 2 and γ = 1. A Kalman filter determines the causal (up to a fixed lag) LMMSE estimate recursively. It is based on a state-space representation of the image and observation models. In the first step of Kalman filtering, a prediction of the present state is formed using an autoregressive (AR) image model and the previous state of the system. In the second step, the predictions are updated on the basis of the observed image data to form the estimate of the present state. Woods and Ingle [39] applied 2-D reduced-update Kalman filter (RUKF) to image restoration, where the update is limited to only those state variables in a neighborhood of the present pixel. The main assumption here is that a pixel is insignificantly correlated with pixels outside a certain neighborhood about itself. More recently, a reduced-ordermodel Kalmanfiltering (ROMKF),wherethestatevector is truncated to asizethat ison the order of the image model support has been proposed [40]. Other Kalman filtering formulations, including higher-dimensional state-space models to reduce the effective size of the state vector, have been reviewed in [7]. The complexity of higher-dimensional state-space model based formulations, however, limits their practical use. Maximum A posteriori Probability Method Themaximum aposterioriprobability(MAP) restorationmaximizesthe aposteriori probability density function (pdf) p(f |g), i.e., the likelihood of a realization of f being the ideal image given the observed data g. Through the application of the Bayes rule, we have p(f |g) ∝ p(g|f )p(f ) (53.16) where p(g|f ) is the conditional pdf of g given f (related to the pdf of the noise process) and p(f )is the a priori pdf of the ideal image. We usually assume that the observation noise is Gaussian, leading c  1999 by CRC Press LLC to p(g|f ) = 1 ( 2π ) N/2 |R v | 1/2 exp  −1/2 ( g − Df ) T R −1 v ( g − Df )  (53.17) where R v denotes the covariance matrix of the noise process. Unlike the LMMSE method, the MAP method uses complete pdf information. However, if both the image and noise are assumed to be homogeneous Gaussian random fields, the MAP estimate reduces to the LMMSE estimate, under a linear observation model. Trussell and Hunt [10] used non-stationary a priori pdf models, and proposed a modified form of the Picard iteration to solve the nonlinear maximization problem. They suggested using the variance of the residual signal as a criterion for convergence. Geman and Geman [11] proposed using a Gibbs random field model for the a priori pdf of the ideal image. They used simulated annealing procedures to maximize Eq. (53.16). It should be noted that the MAP procedures usually require significantly more computation compared to, for example, the CLS or Wiener solutions. Maximum Entropy Method A number of maximum entropy (ME) approaches have been discussed in the literature, which vary in the way that the ME principle is implemented. A common feature of all these approaches, however, is their computational complexity. Maximizing the entropy enforces smoothness of the restored image. (In the absence of constraints, the entropy is highest for a constant-valued image). One important aspect of the ME approach is that the nonnegativity constraint is implicitly imposed on the solution because the entropy is defined in terms of the logarithm of the intensity. Frieden was the first to apply the ME principle to image restoration [41]. In his formulation, the sum of the entropy of the image and noise, given by J ME1 (f ) =−  i f(i)ln f(i)−  i n(i)ln n(i) (53.18) is maximized subject to the constraints n = g − Df (53.19)  i f(i) = K . =  i g(i) (53.20) which enforce fidelity to the data and a constant sum of pixel intensities. This approach requires the solution of a system of nonlinear equations. The number of equations and unknowns are on the order of the number of pixels in the image. The formulation proposed by Gull and Daniell [42] can be viewed as another form of Tikhonov regularization (or constrained least squares formulation), where the entropy of the image J ME2 (f ) =−  i f(i)ln f(i) (53.21) is the regularization functional. It is maximized subject to the following usual constraints ||g − Df || 2 = σ 2 v (53.22)  i f(i) = K . =  i g(i) (53.23) on the restored image. The optimization problem is solved using an ascent algorithm. Trussell [43] showed that in the case of a prior distribution defined in terms of the image entropy, the MAP solution is identical to the solution obtained by this ME formulation. Other ME formulations were also proposed [44, 45]. Note that all ME methods are nonlinear in nature. c  1999 by CRC Press LLC [...]... nonlinearities also plays a significant role in blur identification References [1] Andrews, H.C and Hunt, B.R., Digital Image Restoration, Prentice-Hall, Englewood Cliffs, NJ, 1977 [2] Gonzales, R.C and Woods, R.E., Digital Image Processing, Addison-Wesley, MA, 1992 [3] Katsaggelos, A.K., Ed., Digital Image Restoration, Springer-Verlag, Berlin, 1991 [4] Meinel, E.S., Origins of linear and nonlinear recursive... image processing, Proc IEEE 69(5), 502–528, 1981 [13] Jeng, F.C and Woods, J.W., Compound Gauss-Markov random fields for image restoration, IEEE Trans Sign Proc., SP-39(3), 683–697, 1991 [14] Tekalp, A.M and Sezan, M.I., Quantitative analysis of artifacts in linear space-invariant image restoration, Multidim Syst and Signal Proc., 1(1), 143–177, 1990 [15] Rosenfeld, A and Kak, A.C., Digital Picture Processing, ... Proc., 37, 2024-2036, 1989 [6] Sezan, M.I and Tekalp, A.M., Survey of recent developments in digital image restoration, Optical Eng., 29, 393–404, 1990 [7] Kaufman, H and Tekalp, A.M., Survey of estimation techniques in image restoration, IEEE Control Systems Magazine, 11, 16–24, 1991 [8] Tekalp, A.M., Digital Video Processing, Prentice-Hall, Englewood Cliffs, NJ, 1995 [9] Sezan, M.I and Tekalp, A.M., Adaptive... scheme with multiple different aperture cameras, Signal Proc.: Image Comm., 5, 511–526, 1993 [80] Ur, H and Gross, D., Improved resolution from subpixel shifted pictures, CVGIP: Graphical Models and Image Processing, 54(3), 181–186, 1992 [81] Tsai, R.Y and Huang, T.S., Multiframe image restoration and registration, in Advances in Computer Vision and Image Processing, vol 1, Huang, T.S Ed., Jai Press,... classifiers, and their application to image restoration, in Digital Image Restoration, Katsaggelos, A Ed., Springer Verlag, Berlin, 1991 [39] Woods, J.W and Ingle, V.K., Kalman filtering in two-dimensions-further results, IEEE Trans Acoust Speech Sig Proc., ASSP-29, 188–197, 1981 [40] Angwin, D.L and Kaufman, H., Image restoration using reduced order models, Sig Processing, 16, 21–28, 1988 [41] Frieden, B.R., Restoring... H.J and Civanlar, M.R., Feasible solution in signal restoration, IEEE Trans Acoust Speech Sig Proc., ASSP-32(4), 201-212, 1984 [50] Youla, D.C., Generalized image restoration by the method of alternating orthogonal projections, IEEE Trans Circuits Syst., CAS-25(9), 694–702, 1978 [51] Youla, D.C and Velasco, V., Extensions of a result on the synthesis of signals in the presence of inconsistent constraints,... Florida, 1987 [53] Civanlar, M.R and Trussell, H.J., Digital image restoration using fuzzy sets, IEEE Trans Acoust Speech Sign Proc., ASSP-34(8), 919-936, 1986 [54] Tekalp, A.M and Pavlovi´ , G., Image restoration with multiplicative noise: Incorporating the c sensor nonlinearity, IEEE Trans Sign Proc., SP-39, 2132–2136, 1991 [55] Tekalp, A.M and Pavlovi´ , G., Digital restoration of images scanned from photographic... and Ward, R.K., Restoration of randomly blurred images by the Wiener filter, IEEE Trans Acoust Speech Sig Proc., ASSP-37(4), 589–592, 1989 [59] Combettes, P.L and Trussell, H.J., Methods for digital restoration of signals degraded by a stochastic impulse response, IEEE Trans Acoust Speech Sig Proc., ASSP-37(3), 393–401, 1989 [60] Mesarovic, V.Z Galatsanos, N.P., and Katsaggelos, A.K Regularized constrained... Acoust Speech Sig Proc., ASSP-32(6), 592–599, 1984 [66] Galatsanos, N.P and Chin, R.T., Digital restoration of multi-channel images, IEEE Trans Acoust Speech Sig Proc., ASSP-37(3), 415–421, 1989 [67] Tekalp, A.M and Pavlovi´ , G., Multichannel image modeling and Kalman filtering for multic spectral image restoration, Signal Process., 19, 221-232, 1990 c 1999 by CRC Press LLC [68] Ozkan, M.K., Erdem, A.T.,... ASSP-33(6), 1632–1634, 1985 [26] Sanz, J.L.C and Huang, T.S., Unified Hilbert space approach to iterative least-squares linear signal restoration, J Opt Soc Am., 73(11), 1455–1465, 1983 [27] Tom, V.T., Quatieri, T.F., Hayes, M.H and McClellan, J.H., Convergence of iterative nonexpansive signal reconstruction algorithms, IEEE Trans Acoust Speech Sig Proc., ASSP-29(5), 1052–1058, 1981 [28] Kawata, S and Ichioka, . filtering methods can be found in the book Digital Video Processing by Tekalp [8]. This chapter summarizes key results in digital image and video restoration and Hunt, B.R., Digital Image Restoration, Prentice-Hall, Englewood Cliffs, NJ, 1977. [2] Gonzales, R.C. and Woods, R.E., Digital Image Processing, Addison-Wesley,