阳明大学放射医学科学研究所3课件

An Introduction to Independent Component Analysis(ICA)吳育德陽明大學放射醫學科學研究所台北榮總整合性腦功能實驗室阳明大学放射医学科学研究所3The Principle of ICA:a cocktail-party problem x1(t)=a11 s1(t)+a12 s2(t)+a13 s3(t)x2(t)=a21 s1(t)+a22 s2(t)+a12 s3(t)x3(t)=a31 s1(t)+a32 s2(t)+a33 s3(t)阳明大学放射医学科学研究所3Independent Component AnalysisReference:A.Hyvrinen,J.Karhunen,E.Oja(2001)John Wiley&Sons.Independent Component Analysis阳明大学放射医学科学研究所3Central limit theoremThe distribution of a sum of independent random variables tends toward a Gaussian distributionObserved signal=IC1IC2ICnm1+m2.+mntoward GaussianNon-GaussianNon-GaussianNon-Gaussian阳明大学放射医学科学研究所3Central Limit TheoremPartial sum of a sequence zi of independent and identically distributed random variables ziSince mean and variance of xk can grow without bound as k,consider instead of xk the standardized variablesThe distribution of yk a Gaussian distribution with zero mean and unit variance when k.Partial sum of a sequence zi of independent and identically distributed random variables zi阳明大学放射医学科学研究所3How to estimate ICA modelPrinciple for estimating the model of ICAMaximization of NonGaussianity阳明大学放射医学科学研究所3Measures for NonGaussianityKurtosis Super-Gaussian kurtosis 0Gaussian kurtosis=0Sub-Gaussian kurtosis 0Kurtosis:E(x-)4-3*E(x-)2 2kurt(x1+x2)=kurt(x1)+kurt(x2)kurt(x1)=4kurt(x1)阳明大学放射医学科学研究所3 Assume measurementWhitening processis zero mean and Then is a whitening matrixxEDVxzT21=Iss=TEsxA=TEDV21=TTTEEVxxVzz=2121=EDEDEEDTTI=TTEEDExx=Let D and E be the eigenvalues and eigenvector matrix of covariance matrix of x,i.e.阳明大学放射医学科学研究所3Importance of whiteningFor the whitened data z,find a vector w such that the linear combination y=wTz has maximum nongaussianity under the constrain Maximize|kurt(wTz)|under the simpler constraint that|w|=1Then阳明大学放射医学科学研究所3 Constrained Optimization max F(w),|w|2=1 At the stable point,the gradient of F(w)must point in the direction of w,i.e.equal to w multiplied by a scalar.0),(=wwlL02)(=+wwwlF1),1()(),(22=+=wwwwwwTFLllwww=l2)(F阳明大学放射医学科学研究所3Gradient of kurtosis)(3)()()(224zwzwzwwTTTEEkurtF=wwwzwwzwzwww=21424)(3)(1)(3)()(TTtTTTtTEEF)(2*3)()(413wwwwzwz+=TTtTttT3)(423wwzwzzw=TTEkurtsign=TttTE1)(1yyQ阳明大学放射医学科学研究所3Fixed-point algorithm using kurtosiswk+1=wk+Note that adding the gradient to wk does not change its direction,since Convergence:|=1 since wk and wk+1 are unit vectors)(wkFwk =(+-1l2)(wkFwk233wwzwzTherefore,wTEwww/wk+1=wk-(wk)l2 =(1-2 l)wk阳明大学放射医学科学研究所3Fixed-point algorithm using kurtosis 1.Centering2.Whitening3.Choose m,No.of ICs to estimate.Set counter p 14.Choose an initial guess of unit norm for wp,eg.randomly.5.Let 6.Do deflation decorrelation7.Let wp wp/|wp|8.If wp has not converged(|1),go to step 5.9.Set p p+1.If p m,go back to step 4.One-by-one EstimationFixed-point iteration阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyThe kurtosis is very sensitive to outliers,which may be erroneous or irrelevant observations Need to find a more robust measure for nongaussianityApproximation of negentropyex.r.v.with sample size=1000,mean=0,variance=1,contains one value=10 kurtosis at least equal to 104/1000-3=7 kurtosis:Ex4-3阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyEntropyEntropyNegentropyApproximation of negentropy阳明大学放射医学科学研究所3w Ezg(wTz)Eg(wTz)ww w/|w|Fixed-point algorithm using negentropyConvergence:|=1Max J(y)阳明大学放射医学科学研究所3Fixed-point algorithm using negentropy1.Centering2.Whitening3.Choose m,No.of ICs to estimate.Set counter p 14.Choose an initial guess of unit norm for wp,eg.randomly.5.Let 6.Do deflation decorrelation7.Let wp wp/|wp|8.If wp has not converged,go back to step 5.9.Set p p+1.If p m,go back to step 4.One-by-one EstimationFixed-point iteration阳明大学放射医学科学研究所3ImplantationsCreate two uniform sources阳明大学放射医学科学研究所3ImplantationsCreate two uniform sources阳明大学放射医学科学研究所3ImplantationsTwo mixed observed signals阳明大学放射医学科学研究所3ImplantationsTwo mixed observed signals阳明大学放射医学科学研究所3ImplantationsCentering阳明大学放射医学科学研究所3ImplantationsCentering阳明大学放射医学科学研究所3ImplantationsWhitening阳明大学放射医学科学研究所3ImplantationsWhitening阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using kurtosis阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using kurtosis阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using kurtosis阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using negentropy阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using negentropy阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using negentropy阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using negentropy阳明大学放射医学科学研究所3ImplantationsFixed-point iteration using negentropy阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyEntropyA Gaussian variable has the largest entropy among all random variables of equal varianceEntropyNegentropyGaussian 0nonGaussian 0A Gaussian variable has the largest entropy among all random variables of equal varianceNegentropyGaussian 0nonGaussian 0Its the optimal estimator but computationally difficult since it requires an estimate of the pdfApproximation of negentropy阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyHigh-order cumulant approximation Its quite common that most r.v.have approximately symmetric dist.Its quite common that most r.v.have approximately symmetric dist.Replace the polynomial function by any nonpolynomial fun Giex.G1 is odd and G2 is even阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyAccording to Lagrange multiplier the gradient must point in the direction of w阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyThe iteration doesnt have the good convergence properties because the nonpolynomial moments dont have the same nice algebraic properties.Finding by approximative Newton method Real Newton method is fast.small steps for convergenceIt requires a matrix inversion at every step.large computational loadSpecial properties of the ICA problem approximative Newton methodNo need a matrix inversion but converge roughly with the same steps as real Newton method 阳明大学放射医学科学研究所3Fixed-point algorithm using negentropyAccording to Lagrange multiplier the gradient must point in the direction of wSolve this equation by Newton method JF(w)*w=-F(w)w=JF(w)-1-F(w)EzzTEg(wTz)=Eg(wTz)I diagonalizew w Ezg(wTz)w/Eg(wTz)w Ezg(wTz)Eg(wTz)ww w/|w|Multiply by-Eg(wTz)阳明大学放射医学科学研究所3w Ezg(wTz)Eg(wTz)ww w/|w|Fixed-point algorithm using negentropyConvergence:|=1Because ICs can be defined only up to a multiplication sign阳明大学放射医学科学研究所3MbJ7G4D1z-w&t!qYmVjSgOdL9I6F3B0y)v%s#pXlUiQfNcK8H5E2A+x*u$rZnWkThPeMaJ7G4C1z-w&t!pYmVjRgOdL9I6E3B0y(v%s#oXlTiQfNbK8H5D2A-x*u$qZnWkShPeMaJ7F4C1z)w&t!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaI7F4C0z)w&s!pXmUjRfOcL9H6E2B+y(u%r#oWlTiQeNbJ8G5D1A-x*t$qYnVkSgPdMaI7F3C0z)v&s!pXmUiRfOcK9H6E2B+x(u%rZoWlThQeMbJ8G4D1A-w*t!qYnVjSgPdLaI7F3C0y)v&s#pXmUiRfNcK9H5E2B+x(u$rZoWkThQeMbJ7G4D1z-w*t!qYmVjSgOdLaI6F3B0y)v%s#pXlUiQfNcK8H5E2A+x(u$rZnWkThPeMbJ7G4C1z-w&t!qYmVjRgOdL9I6F3B0y(v%s#oXlUiQfNbK8H5D2A+x*u$qZnWkShPeMaJ7F4C1z)w&t!pYmUjRgOcL9I6E3B0y(v%r#oXlTiQfNbK8G5D2A-x*u$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbG4C1z)w&t!pYmVjRgOcL9I6E3B0y(v%r#oXlTiQfNbK8G5D2A-x*u$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbK8G5D1A-x*t$qZnVkSgPdMaI7F4C0z)v&s!pXmUjRfOcK9H6E2B+y(u%rZoWlThQeNbJ8G4D1A-w*t$qYnVjSgPdLaI7F3C0z)v&s#pXmUiRfOcK9H5E2B+x(u%rZoWkThQeMbJ8G4D1z-w*t!qYnVjSgOdLaI6F3C0y)v%s#pXlUiRfNcK8H5E2A+x(u$rZoWkThPeMbJ7G4D1z-w&t!qYmVjSgOdL9I6F3B0y)v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkShPeMaJ7G4C1z)w&t!pYmVjRgOcL9I6E3B0y(v%s#oXlTiQfNbK8H5D2A-x*u$qZnWkShPdMaJ7F4C1z)w&s!pYmUjRgOcL9H6E3B+y(v%r#oWlTiQeNbK8G5D1A-x*t$qZnVkShPdMaI7F4C0z)w&s!pXmUjRfOcL9H6E2B+y(u%r#oWlThQeNbJ8G5D1A-w*t$qYnVkSgPdLaI7F3C0z)v&s#pXmUiRfOcK9H5E2B+x(u%rZoWlThQeMbJ8G4D1A-w*t!qYnVjSgPdLaI6F3C0y)v&s#pXlUiRfNcK9H5E2A+x(u$rZoWkThPeMbJ7G4D1z-w&t!qYmVjSgOdLaI6F3B0y)v%s#pXlUiQfNcK8H5E2A+x*u$rZnWkThPeMaJ7G4C1z-w&t!pYmVjRgOdL9I6E3B0y(v%s#oXlTiQfNbK8H5D2A+x*u$qZnWkShPeMaJ7F4C1z)w&t!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaI7F4C0z)w&s!pXmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbJ8G5D1&t!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbJ8G5D1A-x*t$qYnVkSgPdMaI7F3C0z)v&s!pXmUiRfOcK9H6E2B+x(u%rZoWlThQeNbJ8G4D1A-w*t$qYnVjSgPdLaI7F3C0y)v&s#pXmUiRfNcK9H5E2B+x(u$rZoWkThQeMbJ7G4D1z-w*t!qYmVjSgOdLaI6F3C0y)v%s#pXlUiRfNcK8H5E2A+x(u$rZnWkThPeMbJ7G4C1z-w&t!qYmVjRgOdL9I6F3B0y(v%s#oXlUiQfNbK8H5D2A+x*u$rZnWkShPeMaJ7G4C1z)w&t!pYmVjRgOcL9I6E3B0y(v%r#oXlTiQfNbK8G5D2A-x*u$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(v%r#oWlTiQeNbK8G5D1A-x*t$qZnVkSgPdMaI7F4C0z)v&s!pXmUjRfOcK9H6E2B+y(u%rZoWlThQeNbJ8G4D1A-w*t$qYnVkSgPdLaI7F3C0z)v&s#pXmUiRfOcK9H5E2B+x(u%rZoWkThQeMbJ8G4D1z-w*t!qYnVjSgOdLaI6F3C0y)v%s#pXlUiRfNcK8H5E2A+x(u$rZoWkThPeMbJ7G4D1z-w&t!qYmVjSgOdL9I6F3B0y)v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkShPeMaJ7G4C1z)w&t!pYmVjRgOdL9I6E3B0y(v%s#oXlTiQfNbK8H5D-w&t!qYmVjSgOdL9I6F3B0y)v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkThPeMaJ7G4C1z-w&t!pYmVjRgOdL9I6E3B0y(v%s#oXlTiQfNbK8H5D2A-x*u$qZnWkShPdMaJ7F4C1z)w&s!pYmUjRgOcL9H6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaI7F4C0z)w&s!pXmUjRfOcL9H6E2B+y(u%r#oWlThQeNbJ8G5D1A-w*t$qYnVkSgPdMaI7F3C0z)v&s!pXmUiRfOcK9H6E2B+x(u%rZoWlThQeMbJ8G4D1A-w*t!qYnVjSgPdLaI6F3C0y)v&s#pXlUiRfNcK9H5E2A+x(u$rZoWkThQeMbJ7G4D1z-w*t!qYmVjSgOdLaI6F3B0y)v%s#pXlUiQf8G4D1A-w*t!qYnVjSgPdLaI6F3C0y)v&s#pXmUiRfNcK9H5E2B+x(u$rZoWkThQeMbJ7G4D1z-w*t!qYmVjSgOdLaI6F3B0y)v%s#pXlUiQfNcK8H5E2A+x*u$rZnWkThPeMaJ7G4C1z-w&t!qYmVjRgOdL9I6F3B0y(v%s#oXlUiQfNbK8H5D2A+x*u$qZnWkShPeMaJ7F4C1z)w&t!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQfNbK8G5D2A-x*u$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbJ8G5D1A-x*t$qYnVkSgPdI6E3B0y(v%r#oXlTiQfNbK8G5D2A-x*u$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbJ8G5D1A-x*t$qZnVkSgPdMaI7F4C0z)v&s!pXmUjRfOcK9H6E2B+y(u%VkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(v%r#oWlTiQeNbK8G5D1A-x*t$qZnVkSgPdMaI7F4C0z)v&s!pXmUjRfOcK9H6E2B+y(u%rZoWlThQeNbJ8G4D1A-w*t$qYnVjSgPdLaI7F3C0z)v&s#pXmUiRfOcK9H5E2B+x(u%rZoWkThQeMbJ8G4D1z-w*t!qYnVjSgOdLaI6F3C0y)v%s#pXlUiRfNcK8H5E2A+x(u$rZoWkThPeMbJ7G4D1z-w&t!qYmVjSgOdL9I6F3B0y)v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkShPeMaJ7G4C1z)w&t!pYmVjRgOdL9I6E3B0u$rZoWkThPeMbJ7G4D1z-w&t!qYmVjSgOdL9I6F3B0y)v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkShPeMaJ7G4C1z-w&t!pYmVjRgOdL9I6E3B0y(v%s#oXlTiQfNbK8H5D2A-x*u$qZnWkShPdMaJ7F4C1z)w&s!pYmUjRgOcL9H6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVgOdL9I6E3B0y(v%s#oXlTiQfNbK8H5D2A-x*u$qZnWkShPdMaJ7F4C1z)w&s!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaI7F4C0z)w&s!pXmUjRfOcL9H6-x*u$qZnWkShPeMaJ7F4C1z)w&t!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaI7F4C0z)w&s!pXmUjRfOcL9H6E2B+y(u%nWkShPeMaJ7F4C1z)w&t!UiQfNcK8H5E2A+x*u$rZnWkThPeMaJ7G4C1z-w&t!pYmVjRgOdL9I6F3B0y(v%s#oXlUiQfNbK8H5D2A+x*u$qZnWkShPeMaJ7F4C1z)w&t!pYmUjRgOcL9I6E3B+y(v%r#oXlTiQeNbK8G5D2A-x*t$qZnVkShPdMaJ7F4C0z)w&s!pYmUjRfOcL9H6E3B+y(u%r#oWlTiQeNbJ8G5D1A-x*t$qYnVkSgPdMaI7F3C0z)v&s!pXmUiRfOcK9H6E2B+y(u%rZoWlThQeNbJ8G4D1A-w*t$qYnVjSgPdLaI7F3C0y)v&s#pXmUiRfNcK5D1A-x*t$qYnVkSgPdMaI7F3C0z)v&s!pXmUjRfOcK9H6E2B+y(u%rZoWlThQeNbJ8G4D1A-w*t$qYnVjSgPdLaI7F3C0y)v&s#pXmUiRfNcK9H5E2B+x(u$rZoWkThQeMbJ8G4D1z-w*t!qYnVjSgOdLaI6F3C0y)v%s#pXlUiRfNcK8H5E2A+x(u$rZnWkThPeMbJ7G4C1z-w&t!qYmVjRgOdL9I6F3B0y(v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkShPeMaJ7G4C1z)w&t!pYmVjRgOcL9I6E3B0y(v%r#oXlTiQfNbK8G5D2A-x*u$qZnVkShPdMaJ7F4C1z)w&s!pYmUjRgOcL9H6E3B+y(v%r#oWPeMaJ7G4C1z)w&t!pYmVjRgOcL9I6E3B0y(v%r#oXlTiQfNbK8H5D2A-x*u$qZnWkShPdMaJ7F4C1z)w&s!pYmUjRgOcL9H6E3B+y(v%r#oWlTaI6F3C0y)v%s#pXlUiRfNcK9H5E2A+x(u$rZoWkThPeMbJ7G4D1z-w&t!qYmVjSgOdL9I6F3B0y)v%s#oXlUiQfNcK8H5D2A+x*u$rZnWkShPeMaJ7阳明大学放射医学科学研究所3