Reproducing Kernels

Definition

Let $X$ be a (nonempty) set and $k$ a field. Let $F \subset (X \to k)$ and let $\ip{\cdot , \cdot }: F \times F \to k$ be an inner product so that $(F, \ip{\cdot ,\cdot })$ is a complete inner product space.

A reproducing kernel of $(F, \ip{\cdot ,\cdot })$ is a map $R: X \times X \to k$ satisfying (1) for every $y \in X$ the function $R(\cdot , y): X \to k$ is an element of $F$ and (2) for every $f \in F$, at every $y \in X$, $f(y) = \ip{f, R(\cdot , y)}$ (the reproducing property).

$R$ is called a “reproducing” kernel because of the following implication of the reproducing property. Notice that $R(\cdot , y) \in F$. For this reason,

Properties

If a reproducing kernel exists, it is unique.

Separate sheet

Let $X$ be nonempty (index) set. For example, $X$ may be $\upto{N}$, $\Z $, $[0, 1]$, $\R ^d$, $\Set{x \in \R ^3}{\norm{x} \leq 1}$ (the unit sphere), or $\Set{x \in \R ^3}{\alpha \leq \norm{x} \leq \beta }$ (the atmosphere, or volume between two concentric spheres).

A symmetric, real-valued function $k: X \times X \to \R $ of two variables is said to be positive semidefinite if for any $n \in \N $, for any real $a_1, \dots , a_n \in \R $ and $x_1, \dots , x_n \in X$, we have

\[ \sum_{i, j = 1}^{n} a_ia_j k(x_i, x_j) \geq 0, \]

and positive definite if the above holds with \say{$>$}.¹

Positive semidefinite kernels are useful for the following constructive reason:

Let $X \neq \varnothing$ be a set. If $k: X \times X \to \R $ is positive semidefinite, then there exists a probability space $(\Omega , \C A, \mathbfsf{P} )$ and a family of zero-mean normal real-valued random variables $\set{f_x: \Omega \to \R }_{x \in X}$ with covariance function $k$, that is,

\[ \E f(a)f(b) = k(a, b), \quad \text{ for all } a, b \in X. \]

This result is known by the names Kolmogorov extension theorem, Kolmogorov existence theorem, Kolmogorov consistency theorem and Daniell-Kolmogorov theorem.

Some authors use the term “positive definite” for our term positive semidefinite and the term “strictly positive definite” for our term “positive definite.” ↩︎