Normal Linear Model

Why

We consider the probabilistic linear model in which all random variables are normal.

Definition

A normal linear model is a probabilistic linear model in which the parameter and noise vectors have normal (Gaussian) densities. The model is also called the Gaussian linear model or the linear model with Gaussian noise.

Let $(x: \Omega \to \R ^d, A \in \R ^{n \times d}, e: \Omega \to \R ^n)$ be a probabilistic linear model over the probability space $(\Omega , \mathcal{A} , \mathbfsf{P} )$ in which $x$ and $e$ have normal densities. Recall that a probabilistic linear model has observation vector $y: \Omega \to \R ^n$ defined by

\[ y = Ax + e. \]

Conditional density of $x$ on $y$

Since $x$ and $e$ are normal and independent, $y$ is normal.¹ Moreover, the random vector $(x, y)$ is normal with covariance

\[ \pmat{ \Sigma _x & \Sigma _{x}A^\top \\ A \Sigma _{x} & A\Sigma _{x}A^\top + \Sigma _{e} }. \]

So the conditional density (see Normal Conditionals) of $g_{x \mid y}(\cdot , \gamma )$ is normal with mean

\[ \Sigma _{x}A^\top (A\Sigma _{x}A^\top + \Sigma _e)^{-1}\gamma \]

and covariance

\[ \Sigma _{x} - \Sigma _{x}A^\top (A\Sigma _{x}A^\top + \Sigma _e)^{-1}A\Sigma _{x}. \]

This density is sometimes called the posterior for the parameters given the observations. So the parameter posterior of the normal linear model is normal.

We can write the conditional mean as

\[ (\Sigma _{x}^{-1} + A^\top \Sigma _{e}^{-1}A)^{-1} A^\top \Sigma _{e}^{-1} \]

and the conditional covariance as²

\[ (\Sigma _x^{-1} + A^\top \Sigma _{e}^{-1} A)^{-1}. \]

Very frequently we use these forms when $d < n$. In other words, in the case that we have fewer unknowns than measurements. In that case $\Sigma _{x}$ is smaller than $A\Sigma _{x}A^\top $.

Maximum conditional estimate of $x$

The maximum conditional estimate of $x: \Omega \to \R ^d$ given observed value $\gamma \in \R ^n$ of $y: \Omega \to \R ^n$ is the conditional mean

\[ \Sigma _{x} A^\top (A\Sigma _{x}A^\top + \Sigma _{e})^{-1}\gamma . \]

Future editions will include an account. ↩︎
A proof will appear in future editions. Use the matrix inversion lemma or facts about inverses. ↩︎