Optimal Classifiers

Why

Discussion

Given two classifiers $G_1$ and $G_2$ and a dataset, we can associate to each its false positve and false negative rate on the dataset. For a finite dataset, these are two rational numbers. It is natural to prefer $G_1$ to $G_2$ if the former has a smaller false positive rate. Conversely, it is natural to prefer $G_2$ to $G_1$ if the former has a smaller false negative rate. Unfortunately, one may need to trade-off these two desiderata (see Combined Orders), since there is no total order. In other words, choosing betwen $G_1$ and $G_2$ is a multiobjective optimization problem.

Scalarization

Let $\mathcal{G} $ be a set of classifiers and let $f: \mathcal{G} \to \R ^2$ be defined so that $f_1(G)$ is the false negative rate of $G$ on some dataset and $f_2(G)$ is the false positive rate of $G$ on the same dataset. The $\kappa $-scalarized error metric (or Neyman-Pearson metric associated with $G \in \mathcal{G} $ is $\kappa f_1(G) + f_2(G)$. In the case that $\kappa > 1$, false negatives are given higher cost than false positives, and vice versa whtn $\kappa < 1$. For $\kappa = 1$, the scalarized error metric is the same as the overall error rate.

1. Future editions will include. ↩︎