Department of Mathematical Analysis and Applications of Mathematics Faculty of Science, Palacký University Olomouc Czech Republic

Transkript

1 ROBUST 13. září 2016 regression regresních modelů Categorical Continuous - explanatory, Eva Fišerová Department of Mathematical Analysis and Applications of Mathematics Faculty of Science, Palacký University Olomouc Czech Republic kristyna.vankatova@upol.cz

2 regression Standard linear regression model y j = x T j β + ε j The mixture of linear regression can be written as y j = x T j β 1 + ε 1j with probability π 1 x T j β 2 + ε 2j with probability π 2 x T j β c + ε cj. with probability π c where ε ij N(0, σ 2 i ) Accounting for the mixture structure, the conditional density of y j x j is f (y j x j, Ψ) = c π i φ(y j x T j β i, σi 2 ). i=1 Ψ denotes the vector of all parameters regression Categorical Continuous - explanatory Ψ = (π 1,..., π c, (β T 1, σ2 1 ),..., (βt c, σ2 c ))T.

3 Maximizing the log likelihood function log L(Ψ x 1,..., x n, y 1,..., y n) = n ( c ) log π i φ(y j x T j β i, σi 2 ). The EM algorithm is an iterative procedure which alternates between an Expectation step and a Maximization step. (k + 1)th iteration of EM algorithm: 1. E-step: Given the observed data y and current parameter estimates Ψ (k) in the kth iteration, calculates estimated a-posteriori probabilities, assigning each observation j to each cluster i. τ (k) ij = 2. M-step: Given the estimates τ (k) ij j=1 π (k) φ(y i j x T j c h=1 π(k) h φ(y j x T j i=1 β (k) i, σ 2(k) ) i β (k) h, σ2(k) ) h for the a-posteriori probabilities τ ij (which are functions of Ψ (k) ), obtain new estimates Ψ (k+1) of the parameters by maximizing expected complete log likelihood, which is in general easier to maximize than the original likelihood.. regression Categorical Continuous - explanatory Q(Ψ, Ψ (k) ) = n c j=1 i=1 τ (k) ij log φ(y j x T j β i, σ 2 i )

4 The maximization is equivalent to solving the weighted least squares problem: β (k+1) i = (X T W (k) X) 1 X T W (k) i i y for i = 1,..., c, where y = (y 1,..., y n) T is the vector of observations and X = (x 1,..., x n) T 1/2 (k) is the design matrix, both are weighted by τ. ij In order to estimate σ 2 i and π i we use (k+1) σ 2 (k+1) (y X β i ) T W = i (y X β (k+1) i ), for i = 1,..., c, n regression Categorical Continuous - explanatory and n π (k+1) j=1 = τ (k) ij i n for i = 1,..., c.

5 with We consider mixture of of form f (y j x j, ω j ) = c π i (ω j, α i )φ(y j x T j β i, σi 2 ), i=1 Multinomial logit model for the π i is given by π i (ω j, α i ) = exp ωt j α i for i = 1,..., c, c α h=1 expωt j h regression Categorical Continuous - explanatory with α = (α 1,..., α c) and α 1 0. The vector of all parameters is given by Ψ = ((α T 1, βt 1, σ2 1 ),..., (αt c, βt c, σ2 c ))T.

6 Parameters estimated via EM algorithm. M-step: maximization of expected complete log likelihood where and Q(Ψ, Ψ (k) ) = Q 1 (β i, σ 2 i, i = 1,..., c ; Ψ (k) ) + Q 2 (α, Ψ (k) ), Q 1 (β i, σ 2 i, i = 1,..., c ; Ψ (k) ) = Q 2 (α, Ψ (k) ) = n c j=1 i=1 n c j=1 i=1 τ (k) ij τ (k) ij log ( π i (ω j, α i ) ). ( ) log φ(y j x T j β i, σi 2 ) Formulas Q 1 and Q 2 can be maximized separately. The maximization of (k+1) Q 1 gives new estimates β i, σ 2(k+1), i = 1,..., c, and the i maximization of Q 2 gives α (k+1). Q 1 is maximized using the weighted ML estimation of linear and Q 2 by means of the weighted ML estimation of multinomial logit. regression Categorical Continuous - explanatory

7 Categorical regression The data comes from two component mixture of the following structure: 150 obs. : y = 5 10 x + ε; ε N(0, 15) 150 obs. : y = 10 2 x + ε; ε N(0, 15) Categorical assigns value 1 and 2 to observations from the first component (with accuracy of 90%) and value 3 and 4 to observations from the second component (with accuracy of 90%) Categorical Continuous - explanatory

8 Comparison of true data classification (top left), classification via ordinary mixture of regressions (bottom) and classification via mixture of (top right). y x x y y x regression Categorical Continuous - explanatory

9 Continuous The data comes from three component mixture of the following structure: 50 obs. : y = x + ε; ε N(0, 10) 80 obs. : y = 40 4 x + ε; ε N(0, 40) 30 obs. : y = 40 1 x + ε; ε N(0, 10) Explanatory can be used as for better estimates of observations membership. regression Categorical Continuous - explanatory

10 Comparison of true data classification (top left), classification via ordinary mixture of regressions (bottom left & bottom right) and classification via mixture of (top right). y y x x y y x x regression Categorical Continuous - explanatory

11 The MSE and variance of the regression parameters and standard error estimates for both mixture : MSEPAR( ψ m) β 10 β 11 β 20 β 21 β 10 β 11 β 20 β 21 β 30 β 31 klasická směs směs s dopr. prom VAR( ψ m) β 10 β 11 β 20 β 21 β 10 β 11 β 20 β 21 β 30 β 31 klasická směs směs s dopr. prom For three component mixture - most of the standard mixture estimates are completely inaccurate and not included in the estimates quality comparison. regression Categorical Continuous - explanatory

12 Conclusions regression can be applied to data where observations originate from various groups and the group affiliations are not known. Mixture of provides quality estimates in the sense of lower MSE and variance of parameters. Concomitant enhances the classification of observations. regression Categorical Continuous - explanatory

13 regression Thank you for your attention! Categorical Continuous - explanatory

14 BENAGLIA, T., CHAUVEAU, D., HUNTER, D.R. a YOUNG, D.S.. Mixtools: An R Package for Analyzing Finite Mixture Models. Journal of Statistical Software. 2009, Vol. 32, No [online ] DE VEAUX, R.D. linear regressions. Comput. Statist. Data Anal. 1989, No. 8, pp DESARBO, W.S. a CRON, W.L.. A Maximum Likelihood Methodology for Clusterwise Linear Regression. Journal of Classification. 1988, No. 5. URL: deepblue.lib.umich.edu [online ] FARIA, Susana, SOROMENHO, Gilda. Fitting mixtures of linear regressions. Journal of Statistical Computation and Simulation (Impact Factor: 0.63). 03/2010; 80: DOI: / URL: [online ] GRÜN, B. a LEISCH. F.. Fitting Finite Generalized Linear Regressions in R. Computational Statistics and Data Analysis URL: [online ] GRÜN, B. a LEISCH. F.. FlexMix Version 2: Finite Mixtures with Concomitant Variables and Varying and Constant Parameters. Journal of Statistical Software. 2008, Vol. 28, No. 4. URL: cran.at.r-project.org [online ] HOSHIKAWA, Toshiya. for observational data, with application to functional regression URL: arxiv.org [online ] JONES, P.N., MCLACHLAN, G.J. Fitting finite mixture in a regression context. Aust. J. Statist. 1992, No. 34, pp LEISCH, Friedrich. FlexMix: A General Framework for Finite Mixture Models and Latent Class Regression in R. Journal of Statistical Software. 2004, Vol. 11, No. 8. regression Categorical Continuous - explanatory