## Linear Mixed-Effects Models

Linear mixed-effects models are extensions of linear regression models for data that are collected and summarized in groups. These models describe the relationship between a response variable and independent variables, with coefficients that can vary with respect to one or more grouping variables. A mixed-effects model consists of two parts, fixed effects and random effects. Fixed-effects terms are usually the conventional linear regression part, and the random effects are associated with individual experimental units drawn at random from a population. The random effects have prior distributions whereas fixed effects do not. Mixed-effects models can represent the covariance structure related to the grouping of data by associating the common random effects to observations that have the same level of a grouping variable. The standard form of a linear mixed-effects model is

`$y=\underset{fixed}{\underbrace{X\beta }}+\underset{random}{\underbrace{Zb}}+\underset{error}{\underbrace{\epsilon }},$`

where

• y is the n-by-1 response vector, and n is the number of observations.

• X is an n-by-p fixed-effects design matrix.

• β is a p-by-1 fixed-effects vector.

• Z is an n-by-q random-effects design matrix.

• b is a q-by-1 random-effects vector.

• ε is the n-by-1 observation error vector.

The assumptions for the linear mixed-effects model are:

• Random-effects vector, b, and the error vector, ε, have the following prior distributions:

`$\begin{array}{l}b~N\left(0,{\sigma }^{2}D\left(\theta \right)\right),\\ \epsilon ~N\left(0,\sigma {}^{2}I\right),\end{array}$`

where D is a symmetric and positive semidefinite matrix, parameterized by a variance component vector θ, I is an n-by-n identity matrix, and σ2 is the error variance.

• Random-effects vector, b, and the error vector, ε, are independent from each other.

Mixed-effects models are also called multilevel models or hierarchical models depending on the context. Mixed-effects models is a more general term than the latter two. Mixed-effects models might include factors that are not necessarily multilevel or hierarchical, for example crossed factors. That is why mixed-effects is the terminology preferred here. Sometimes mixed-effects models are expressed as multilevel regression models (first level and grouping level models) that are fit simultaneously. For example, a varying or random intercept model, with one continuous predictor variable x and one grouping variable with M levels, can be expressed as

`$\begin{array}{l}{y}_{im}={\beta }_{0m}+{\beta }_{1}{x}_{im}+{\epsilon }_{im},\text{ }i=1,2,..,n,\text{ }m=1,2,...,M,\text{ }{\epsilon }_{im}~N\left(0,{\sigma }^{2}\right),\\ {\beta }_{0m}={\beta }_{00}+{b}_{0m},\text{ }{b}_{0m}~N\left(0,{\sigma }_{0}^{2}\right),\end{array}$`

where yim corresponds to data for observation i and group m, n is the total number of observations, and b0m and εim are independent of each other. After substituting the group-level parameters in the first-level model, the model for the response vector becomes

`${y}_{im}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta }_{00}+{\beta }_{1}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}}}+{\epsilon }_{im}.$`

A random intercept and slope model with one continuous predictor variable x, where both the intercept and slope vary independently by a grouping variable with M levels is

`$\begin{array}{l}{y}_{im}={\beta }_{0m}+{\beta }_{1m}{x}_{im}+{\epsilon }_{im},\text{ }i=1,2,...,n,\text{ }m=1,2,...,M,\text{ }{\epsilon }_{im}~N\left(0,{\sigma }^{2}\right),\\ {\beta }_{0m}={\beta }_{00}+{b}_{0m},\text{ }{b}_{0m}~N\left(0,{\sigma }_{0}^{2}\right),\text{ }\\ {\beta }_{1m}={\beta }_{10}+{b}_{1m},\text{ }{b}_{1m}~N\left(0,{\sigma }_{1}^{2}\right),\end{array}$`

or

`${b}_{m}=\left(\begin{array}{l}{b}_{0m}\\ {b}_{1m}\end{array}\right)~N\left(0,\left(\begin{array}{cc}{\sigma }_{0}^{2}& 0\\ 0& {\sigma }_{1}^{2}\end{array}\right)\right).$`

You might also have correlated random effects. In general, for a model with a random intercept and slope, the distribution of the random effects is

`${b}_{m}=\left(\begin{array}{l}{b}_{0m}\\ {b}_{1m}\end{array}\right)~N\left(0,\sigma {}^{2}D\left(\theta \right)\right),$`

where D is a 2-by-2 symmetric and positive semidefinite matrix, parameterized by a variance component vector θ.

After substituting the group-level parameters in the first-level model, the model for the response vector is

`${y}_{im}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta }_{00}+{\beta }_{10}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}+{b}_{1m}{x}_{im}}}+{\epsilon }_{im},\text{ }i=1,2,...,n,\text{ }m=1,2,...,M.$`

If you express the group-level variable, xim, in the random-effects term by zim, this model is

`${y}_{im}=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta }_{00}+{\beta }_{10}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}+{b}_{1m}{z}_{im}}}+{\epsilon }_{im},\text{ }i=1,2,...,n,\text{ }m=1,2,...,M.$`

In this case, the same terms appear in both the fixed-effects design matrix and random-effects design matrix. Each zim and xim correspond to the level m of the grouping variable.

It is also possible to explain more of the group-level variations by adding more group-level predictor variables. A random-intercept and random-slope model with one continuous predictor variable x, where both the intercept and slope vary independently by a grouping variable with M levels, and one group-level predictor variable vm is

`$\begin{array}{l}{y}_{im}={\beta }_{0im}+{\beta }_{1im}{x}_{im}+{\epsilon }_{im},\text{ }i=1,2,...,n,\text{ }m=1,2,...,M,\text{ }{\epsilon }_{im}~N\left(0,{\sigma }^{2}\right),\\ {\beta }_{0im}={\beta }_{00}+{\beta }_{01}{v}_{im}+{b}_{0m},\text{ }{b}_{0m}~N\left(0,{\sigma }_{0}^{2}\right),\text{ }\\ {\beta }_{1im}={\beta }_{10}+{\beta }_{11}{v}_{im}+{b}_{1m},\text{ }{b}_{1m}~N\left(0,{\sigma }_{1}^{2}\right).\end{array}$`

This model results in main effects of the group-level predictor and an interaction term between the first-level and group-level predictor variables in the model for the response variable as

`$\begin{array}{l}{y}_{im}={\beta }_{00}+{\beta }_{01}{v}_{im}+{b}_{0m}+\left({\beta }_{10}+{\beta }_{11}{v}_{im}+{b}_{1m}\right){x}_{im}+{\epsilon }_{im},\text{ }i=1,2,...,n,\text{ }m=1,2,...,M,\\ \text{ }\text{ }=\underset{fixed\text{\hspace{0.17em}}effects}{\underbrace{{\beta }_{00}+{\beta }_{10}{x}_{im}+{\beta }_{01}{v}_{im}+{\beta }_{11}{v}_{im}{x}_{im}}}+\underset{random\text{\hspace{0.17em}}effects}{\underbrace{{b}_{0m}+{b}_{1m}{x}_{im}}}+{\epsilon }_{im}.\end{array}$`

The term β11vmxim is often called a cross-level interaction in many textbooks on multilevel models. The model for the response variable y can be expressed as

`$\begin{array}{l}{y}_{im}=\left[\begin{array}{cccc}1& {x}_{1}{}_{im}& {v}_{im}& {v}_{im}{x}_{1im}\end{array}\right]\left[\begin{array}{c}{\beta }_{00}\\ {\beta }_{10}\\ {\beta }_{01}\\ {\beta }_{11}\end{array}\right]+\left[\begin{array}{cc}1& {x}_{1im}\end{array}\right]\left[\begin{array}{c}{b}_{0m}\\ {b}_{1m}\end{array}\right]+{\epsilon }_{im},\text{ }i=1,2,...,n,\text{ }m=1,2,...,M,\\ \text{ }\text{ }\end{array}$`

which corresponds to the standard form given earlier,

`$\text{ }y=X\beta +Zb+\epsilon .$`

In general, if there are R grouping variables, and m(r,i) shows the level of grouping variable r, for observation i, then the model for the response variable for observation i is

`${y}_{i}={x}_{i}^{T}\beta +\sum _{r=1}^{R}{z}_{ir}{b}_{m\left(r,i\right)}^{\left(r\right)}+{\epsilon }_{i},\text{ }i=1,2,...,n,$`

where β is a p-by-1 fixed-effects vector, b(r)m(r,i) is a q(r)-by-1 random-effects vector for the rth grouping variable and level m(r,i), and εi is a 1-by-1 error term for observation i.

 Pinherio, J. C., and D. M. Bates. Mixed-Effects Models in S and S-PLUS. Statistics and Computing Series, Springer, 2004.

 Hariharan, S. and J. H. Rogers. “Estimation Procedures for Hierarchical Linear Models.” Multilevel Modeling of Educational Data (A. A. Connell and D. B. McCoach, eds.). Charlotte, NC: Information Age Publishing, Inc., 2008.

 Hox, J. Multilevel Analysis, Techniques and Applications. Lawrence Erlbaum Associates, Inc., 2002

 Snidjers, T. and R. Bosker. Multilevel Analysis. Thousand Oaks, CA: Sage Publications, 1999.

 Gelman, A. and J. Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models. New York, NY: Cambridge University Press, 2007.