The Latent Class Model

Latent Class Analysis

Latent Class Analysis assumes that people belong to different groups (i.e., classes) that are due to nonobservable characteristics. In a latent class model, we aim to estimate two kinds of probabilities:

1. The probability that a person belongs to a particular class.
2. The conditional probabilities of item responses if a person belongs to a given class.

The likelihood

Suppose that a sample of people respond to $J$ items and $\mathbf{y}$ is a vector that contains the scores to each item $j$. Let $K$ denote the number of latent classes and let $x_k$ be class $k$. Then, the likelihood of the response pattern $\mathbf{y}$, if it was observed $n$ times in the sample and every person responds independently of each other, can be written as $$\ell \;=\; P(\mathbf{y})^{n} \;=\; \Bigg(\sum_{k=1}^{K} P(x_k)\, P(\mathbf{y}\mid x_k)\Bigg)^{n}.$$

Assuming local independence (responses within a person are independent conditional on class), we have $$P(\mathbf{y}\mid x_k) \;=\; \prod_{j=1}^{J} P(y_j \mid x_k),$$ where $y_j$ denotes the score on item $j$. Hence, $$\ell \;=\; \Bigg(\sum_{k=1}^{K} P(x_k)\, \prod_{j=1}^{J} P(y_j \mid x_k)\Bigg)^{\!n},$$ and the log-likelihood becomes $$\ell\ell \;=\; n \log \Bigg(\sum_{k=1}^{K} P(x_k)\, \prod_{j=1}^{J} P(y_j \mid x_k)\Bigg).$$ The term inside the parenthesis is the probability of a single pattern, $P(\mathbf{y})$. Assuming independence between people with different response patterns, the log-likelihood of the whole sample is the sum of log-likelihoods of each response pattern.

To simplify the computation of the logarithm likelihood and related derivatives, let $l_{y_m \mid x_g} = \log P(y_m \mid x_g)$, so that

$$\ell\ell \;=\; n \log \Bigg(\sum_{k=1}^{K} P(x_k)\, \exp\bigg(\sum_{j=1}^{J} l_{y_m \mid x_g}\bigg)\Bigg).$$

First-order derivatives

For a fixed pattern $\mathbf{y}$, define $$\begin{aligned} P(\mathbf{y}) \;&=\; \sum_{k=1}^{K} P(x_k)\, \prod_{j=1}^{J} P(y_j \mid x_k) \\ &=\; \sum_{k=1}^{K} P(x_k)\, \exp\bigg(\sum_{j=1}^{J} l_{y_m \mid x_g}\bigg). \end{aligned}$$ Then $$\frac{\partial \ell\ell}{\partial P(x_g)} \;=\; n \,\frac{1}{P(\mathbf{y})}\, \prod_{j=1}^{J} P(y_j \mid x_g).$$ For a specific item $m$ and class $g$, $$\frac{\partial \ell\ell}{\partial l_{y_m \mid x_g}} \;=\; n \,\frac{1}{P(\mathbf{y})}\, P(x_g)\, \prod_{j=1}^{J} P(y_j \mid x_g).$$

Notice that this last expression is just the posterior, $P(x_g \mid y_m)$, weighted by $n$.

Second-order derivatives

For classes $g,h$, $$\begin{aligned} \frac{\partial^2 \ell\ell}{\partial P(x_g)\,\partial P(x_h)} \;&=\; -\,n \,\frac{1}{P(\mathbf{y})^{2}} \, \Bigg(\prod_{j=1}^{J} P(y_j \mid x_g)\Bigg)\! \Bigg(\prod_{j=1}^{J} P(y_j \mid x_h)\Bigg) \\ &=\; -n \frac{P(x_g \mid y) P(x_h \mid y)}{P(x_g) P(x_h)}. \end{aligned}$$

For items $m,n$ and classes $g,h$, $$\frac{\partial^2 \ell\ell}{\partial l_{y_m \mid x_g}\,\partial l_{y_n \mid x_h}} = \begin{cases} \,n \, P(x_g \mid y) \!\ \big(1-P(x_g \mid y)\big), & \text{if } \!\ g=h,\\[1.25em] -\,n \, P(x_g \mid y) \!\ P(x_h \mid y), & \text{otherwise.} \end{cases}$$

For the mixed second derivative, $$\frac{\partial^2 \ell\ell}{\partial P(x_h)\,\partial l_{y_m \mid x_g}} = \begin{cases} \frac{1}{P(x_g)} \, n \, P(x_g \mid y) \big(1-P(x_g \mid y)\big), & \text{if } g=h,\\[1.0em] -\, \frac{1}{P(x_h)} \, n \, P(x_g \mid y) P(x_h \mid y), & \text{otherwise.} \end{cases}$$

Collecting these terms gives the Hessian in block form: $$\mathrm{Hess}(\ell\ell) \;=\; \begin{bmatrix} \displaystyle \frac{\partial^2 \ell\ell}{\partial P(x_g)\,\partial P(x_h)} & \displaystyle \frac{\partial^2 \ell\ell}{\partial P(x_h)\,\partial l_{y_m \mid x_g}} \\[0.8em] \displaystyle \frac{\partial^2 \ell\ell}{\partial l_{y_m \mid x_g}\,\partial P(x_h)} & \displaystyle \frac{\partial^2 \ell\ell}{\partial l_{y_m \mid x_g}\,\partial l_{y_m \mid x_g}} \end{bmatrix}\!.$$

Models for the conditional likelihoods

The conditional probabilities need to be parameterized with a likelihood. We consider a multinomial likelihood for categorical items and a Gaussian likelihood for continuous items.

Multinomial

For categorical items, let $\pi_{m_k \mid g}$ be the probability of scoring category $k$ on item $m$ if a subject belongs to class $g$. Then $$P(y_m \mid x_g) \;=\; \pi_{m_k \mid g},$$ where $k$ is such that $y_m = k$. With this parameterization, $$\frac{\partial l_{y_m \mid x_g}}{\partial \pi_{n_k \mid h}} = \begin{cases} \frac{1}{\pi_{n_k \mid h}}, & \text{if } y_m = k,\ m=n,\ g=h,\\ 0, & \text{otherwise.} \end{cases}$$ and $$\frac{\partial^2 l_{y_j \mid x_i}}{\partial \pi_{m_k \mid g} \partial \pi_{n_l \mid h}} = \begin{cases} -\frac{1}{\pi_{m_j \mid i}^2}, & \text{if } y_j = k = l,\ y=m=n,\ i=g=h,\\ 0, & \text{otherwise.} \end{cases}$$

Consequently, the Hessian for each conditional parameter has the following block form: $$\mathrm{Hess}(l_{y_m \mid x_g}) \;=\; \mathbf{e} \; \displaystyle \frac{\partial^2 l_{y_j \mid x_i}}{\partial \pi_{m_k \mid g} \partial \pi_{n_l \mid h}} \; \mathbf{e}^\top,$$ where $\mathbf{e}$ is a vector of zeroes with a 1 in the position corresponding to the parameter $\pi_{y_j \mid i}$.

Notice that each conditional parameter $l_{y_m \mid x_g}$ has a Hessian matrix.

Gaussian

For continuous items, let $\varphi$ denote the normal density. Let $\mu_{m\mid g}$ and $\sigma_{m\mid g}$ be the mean and standard deviation for item $m$ in class $g$. Then $$P(y_m \mid x_g) \;=\; \varphi\!\big(y_m;\, \mu_{m\mid g},\, \sigma_{m\mid g}\big).$$

First derivatives: $$\frac{\partial l_{y_m \mid x_g}}{\partial \mu_{n\mid h}} = \begin{cases} \dfrac{y_m - \mu_{m\mid g}}{\sigma_{m\mid g}^{2}}, & \text{if } m=n,\ g=h,\\[0.6em] 0, & \text{otherwise,} \end{cases}$$

$$\frac{\partial l_{y_m \mid x_g}}{\partial \sigma_{n\mid h}} = \begin{cases} \dfrac{(y_m-\mu_{m\mid g})^{2}-\sigma_{m\mid g}^{2}}{\sigma_{m\mid g}^{3}}, & \text{if } m=n,\ g=h,\\[0.6em] 0, & \text{otherwise.} \end{cases}$$

Second-order derivatives: $$\frac{\partial^{2} l_{y_m \mid x_g}}{\partial \mu_{m\mid g}\, \partial \mu_{n\mid h}} = \begin{cases} -\dfrac{1}{\sigma_{m\mid g}^{2}}, & \text{if } m=n,\ g=h,\\[0.6em] 0, & \text{otherwise,} \end{cases}$$

$$\frac{\partial^{2} l_{y_m \mid x_g}}{\partial \sigma_{m\mid g}\, \partial \sigma_{n\mid h}} = \begin{cases} \dfrac{1}{\sigma_{m\mid g}^{2}} - \dfrac{3(y_m-\mu_{m\mid g})^{2}}{\sigma_{m\mid g}^{4}}, & \text{if } m=n,\ g=h,\\[1.0em] 0, & \text{otherwise,} \end{cases}$$

$$\frac{\partial^{2} l_{y_m \mid x_g}}{\partial \mu_{m\mid g}\, \partial \sigma_{n\mid h}} = \begin{cases} -\dfrac{2(y_m-\mu_{m\mid g})}{\sigma_{m\mid g}^{3}}, & \text{if } m=n,\ g=h,\\[0.6em] 0, & \text{otherwise.} \end{cases}$$

Consequently, the Hessian for each conditional parameter has the following block form: $$\mathrm{Hess}(l_{y_m \mid x_g}) \;=\; \begin{bmatrix} \displaystyle \frac{\partial^{2} l_{y_m \mid x_g}}{\partial \mu_{m \mid g}\,\partial \mu_{n \mid h}} & \displaystyle \frac{\partial^{2} l_{y_m \mid x_g}}{\partial \mu_{m \mid g}\,\partial \sigma_{n \mid h}} \\[0.6em] \displaystyle \frac{\partial^{2} l_{y_m \mid x_g}}{\partial \sigma_{n \mid h}\,\partial \mu_{m \mid g}} & \displaystyle \frac{\partial^{2} l_{y_m \mid x_g}}{\partial \sigma_{m \mid g}\,\partial \sigma_{n \mid h}} \end{bmatrix}.$$

Notice that each conditional parameter $l_{y_m \mid x_g}$ has a Hessian matrix.

Model for the latent class probabilities

Probabilities of class membership are parameterized with the softmax transformation:

$$P(x_g) = \frac{\exp(\theta_g)}{\sum_j \exp(\theta_j)},$$ where $\theta_g$ is the log-scale parameter associated with class $g$.

The jacobian of this transformation is given by

$$J = \mathrm{diag}(P) - PP^\top.$$ Finally, the Hessian for each probability is

$$\mathrm{Hess}\big(P(x_g)\big) \;=\; P(x_g)\bigg((e_g−P)(e_g−P)^\top−J\bigg),$$ where $e_g$ is a vector of zeroes with a $1$ in position $g$.

Model for the conditional probabilities of the multinomial model

Probabilities of conditional responses are parameterized with the softmax transformation:

$$\pi_{m_k \mid g} = \frac{\exp(\eta_{m_k \mid g})}{\sum_j \exp(\eta_{m_j \mid g})},$$ where $\eta_{m_k \mid g}$ is the log-scale parameter associated with response $k$ to item $m$ in class $g$.

The jacobian of this transformation is given by

$$J_{m \mid g} = \mathrm{diag}(\pi_{m \mid g}) - \pi_{m \mid g} (\pi_{m \mid g})^\top.$$ Finally, the Hessian for each probability is

$$\mathrm{Hess}\big( \pi_{m_k \mid g} \big) \;=\; \pi_{m \mid g}\bigg((e_k−\pi_{m \mid g})(e_k−\pi_{m \mid g})^\top−J\bigg),$$ where $e_k$ is a vector of zeroes with a $1$ in position $k$.

Constant priors

For latent class probabilities

For conditional likelihoods

Multinomial

For the conditional probabilities modeled with a multinomial likelihood, we add the following term to the log-likelihood for each class $g$:

$$\lambda_2 = \sum_m \sum_k \hat{\pi}_{m_k} \sum_g \frac{\alpha}{K} \log (\pi_{m_k \mid g}),$$

Where $\hat{\pi}_{m_k}$ is the proportion of times category $k$ was selected in item $m$.

The first-order derivatives are

$$\frac{\partial \lambda_2}{\partial \pi_{m_k \mid g}} = \sum_m \sum_k \hat{\pi}_{m_k} \sum_g \frac{\alpha}{K} \frac{1}{\pi_{m_k \mid g}}.$$

The second-order derivatives are

$$\frac{\partial^2 \lambda_2}{\partial \pi_{m_k \mid g} \; \partial \pi_{m_k \mid g}} = -\sum_m \sum_k \hat{\pi}_{m_k} \sum_g \frac{\alpha}{K} \frac{1}{\pi^2_{m_k \mid g}}.$$

Gaussian

For the conditional probabilities modeled with a gaussian likelihood, we add the following term to the log-likelihood for each class $g$: $$\begin{aligned} \lambda_3 &= \sum_g^K \left( -0.5\frac{\alpha}{K} \log\bigg(\prod_j \sigma^2_{j\mid g}\bigg) - 0.5\frac{\alpha}{K} \sum_j \frac{\hat{\sigma}^2_j}{\sigma^2_{j \mid g}} \right) \\ &= \sum_g^K \left( -\frac{\alpha}{K} \sum_j s_{j\mid g} - 0.5\frac{\alpha}{K} \sum_j \frac{\hat{\sigma}^2_j}{\sigma^2_{j \mid g}} \right), \end{aligned}$$

where $\hat{\sigma}^2_j$ is the variance of item $j$.

The first-order derivatives are $$\begin{aligned} \frac{\partial \lambda_3}{\partial \sigma_{m \mid g}} \;&=\; \sum_g^K -\frac{\alpha}{K \sigma_{m \mid g}} + \frac{\alpha}{K \sigma^3_{m \mid g}} \hat{\sigma}^2_{m \mid g}\\ &= \sum_g^K \frac{\alpha}{K} \Bigg(\frac{\hat{\sigma}^2_{m \mid g}}{\sigma^3_{m \mid g}} - \frac{1}{\sigma_{m \mid g}}\Bigg). \end{aligned}$$

The second-order derivatives are $$\frac{\partial^2 \lambda_3}{\partial \sigma_{m \mid g} \partial \sigma_{m \mid g}} \;=\; \sum_g^K \frac{\alpha}{K} \Bigg(\frac{1}{\sigma_{m \mid g}} - 3\frac{\hat{\sigma}^2_{m \mid g}}{\sigma^4_{m \mid g}}\Bigg).$$