Some Factor AnalysisEstimation for the Simplest Factor%Analytic ModelThe common factor model:for person i, in which f is a latent variable and the matrix of !factor loadings" #regression parameters for the ys on the fs$ and the #independent$ residual variances are to be estimated.Λ =λ λ . . . λλ λ . . . λλ λ . . . λ.........λ λ . . . λ∆ =σ2!i0 . . . 00 σ2!i. . . 00 0 . . . 0.........0 0 . . . σ2!iyi= µ + Λfi+ !iThis implies that the observed variables as distributed in multivariate normal form with mean and covariance matrix:Among the readings, Bock & Bargmann #1966$, Jennrich & Robinson #1969$, J&reskog#1969, 1971$, and Rubin & Thayer #1982$ variously develop what is now commonly called the !Wishart" likelihood that may be maximized to estimate the parameters.µΣ = ΛΦΛ!+ ∆The multivariate normal likelihood is:The maximum likelihood estimate of is #either$ obviously #or see Anderson, 1958, p. 47$ the mean vector If we let the sum of products of the N observations corrected to the sample mean beand the loglikelihood to be maximized to estimate the parameters that yield is:N · S =N!i=1yy!− Ny·y·!,L =N!i=1|Σ|−1/2(2π)p/2exp[−12(yi− µ)"Σ−1(yi− µ)]y·logL = −Np2log2π −N2log|Σ| −N2trΣ−1S.ΣµFor maximization, we compute only the part that involves the data:! = −N2log|Σ| −N2trΣ−1SSometimes the criterion function is modi(ed to include all parts of the #asymptotically chi% square distributed$ likelihood%ratio goodness of (t statistic:F = (N − 1)(log|Σ| + trΣ−1S − log|S| − p)Some examples• Naive estimation, one factor equal loadings• Jennrich & Robinson's suggestion• J&reskog's (rst !congeneric tests" example #for this we do three models, and then the unrestricted model using the the EM algorithm of Rubin & Thayer$• And (nally back to Bock & Bargmannm0.10.1M0.1M0.10.20.2M0.2M0.20.30.3M0.3M0.30.40.4M0.4M0.40.50.5M0.5M0.50.60.6M0.6M0.60.70.7M0.7M0.70.80.8M0.8M0.80.90.9M0.9M0.9m900900M900M90010001000M1000M100011001100M1100M110012001200M1200M120013001300M1300M130014001400M1400M140015001500M1500M1500loadingMloadingMloading-loglikelihoodM-loglikelihoodM-loglikelihoodThe loglikelihood for the #single, equal$ factor loading; all correlations equal 0.5m0.650.65M0.65M0.650.660.670.67M0.67M0.670.680.690.69M0.69M0.690.700.710.71M0.71M0.710.720.730.73M0.73M0.730.740.750.75M0.75M0.75m832832M832M832834834M834M834836836M836M836838838M838M838840840M840M840842842M842M842loadingMloadingMloading-loglikelihoodM-loglikelihoodM-loglikelihoodThe loglikelihood for the #single, equal$ factor loading; all correlations equal 0.5m0.10.1M0.1M0.10.20.2M0.2M0.20.30.3M0.3M0.30.40.4M0.4M0.40.50.5M0.5M0.50.60.6M0.6M0.60.70.7M0.7M0.70.80.8M0.8M0.80.90.9M0.9M0.9m10001000M1000M100015001500M1500M150020002000M2000M200025002500M2500M250030003000M3000M3000unique varianceMunique varianceMunique variance-loglikelihoodM-loglikelihoodM-loglikelihoodThe loglikelihood for the #single, equal$ unique variance; all correlations equal 0.5m0.400.40M0.40M0.400.420.440.44M0.44M0.440.460.480.48M0.48M0.480.500.520.52M0.52M0.520.540.560.56M0.56M0.560.580.600.60M0.60M0.60m835835M835M835840840M840M840845845M845M845850850M850M850855855M855M855860860M860M860unique varianceMunique varianceMunique variance-loglikelihoodM-loglikelihoodM-loglikelihoodThe loglikelihood for the #single, equal$ unique variance; all correlations equal 0.5-likelihood Contours, Equal Loading Factor AnalysisM-likelihood Contours, Equal Loading Factor AnalysisM-likelihood Contours, Equal Loading Factor AnalysisLoadingMLoadingMLoadingUnique VarianceMUnique VarianceMUnique Variancem0.500.50M0.50M0.500.550.55M0.55M0.550.600.60M0.60M0.600.650.65M0.65M0.650.700.70M0.70M0.700.750.75M0.75M0.750.800.80M0.80M0.800.850.85M0.85M0.850.900.90M0.90M0.90m0.300.30M0.30M0.300.350.35M0.35M0.350.400.40M0.40M0.400.450.45M0.45M0.450.500.50M0.50M0.500.550.55M0.55M0.550.600.60M0.60M0.600.650.65M0.65M0.650.700.70M0.70M0.70The loglikelihood for both the loading and the unique variance; all correlations equal 0.5-likelihood Contours, Equal Loading Factor AnalysisM-likelihood Contours, Equal Loading Factor AnalysisM-likelihood Contours, Equal Loading Factor AnalysisLoadingMLoadingMLoadingUnique VarianceMUnique VarianceMUnique Variancem-0.8-0.8M-0.8M-0.8-0.6-0.6M-0.6M-0.6-0.4-0.4M-0.4M-0.4-0.2-0.2M-0.2M-0.20.00.0M0.0M0.00.20.2M0.2M0.20.40.4M0.4M0.40.60.6M0.6M0.60.80.8M0.8M0.8m0.300.30M0.30M0.300.350.35M0.35M0.350.400.40M0.40M0.400.450.45M0.45M0.450.500.50M0.50M0.500.550.55M0.55M0.550.600.60M0.60M0.600.650.65M0.65M0.650.700.70M0.70M0.70The loglikelihood for both the loading and the unique variance; all correlations equal 0.5Some Factor AnalysisEM Estimation for the Factor%Analytic ModelA purely statistical #not psychological$ description of factor analysis in Rubin & Thayer's notation:Y is the #centered$ n x p observed data matrix, and Z an n x q unobserved matrix of factor scores. Each column of Z is N#0,1$ with correlation matrix R among columns. The conditional distribution #given Z$ of the ith row of Y is normal with mean and residual covarianceα + Ziβτ2= diag(τ21, . . . , τ2p)We will consider only their !case 1" with R = I #orthogonal factors$ and unrestrictedIn #our$ traditional language, the s are factor loadings #regression coe)cients of the Ys on the Zs$ and the s are the unique variances.τ2βFor the EM algorithm, consider the Zs !missing data" and then estimate the regression parameters anyway. For that we'll need covariance matrices C:Cyy=n!1Y!iYinCyz=n!1Y!iZinCz z=n!1Z!iZinE%Step:E(Cyy|Y, τ2, β, R) = CyyE(Cyz|Y, τ2, β, R) = CyyδE(Cz z|Y, τ2, β, R) = δ!Cyyδ + ∆For the simplest special case, R = I, δ = (τ2+ β!β)−1β!∆ = I − β(τ2+ β!β)−1β!Why? The s are the regression coe)cients of the Zs on the Ys.δδ = (ΣY Y)−1ΣY Z∆ = I − ΣZY(ΣY Y)−1ΣY ZZi∼ N (δYi, ∆)M%Step:β∗ = [δ!Cyyδ + ∆]−1(Cyyδ)!τ ∗2= diag(Cyy− Cyyδ[δ!Cyyδ + ∆]−1(Cyyδ)!Why?β∗ = [E(Cz z|Y, τ2, β, R)]−1[E(Cyz|Y, τ2, β, R)]"τ ∗2= diag(Cyy− [E(Cyz|Y, τ2, β, R)][E(Czz|Y, τ2, β, R)]−1[E(Cyz|Y, τ2, β, R)]")Regression.EM computations are quick per%cycle, but it takes many