Error Tolerance (PAC learning)

Error Tolerance (PAC learning)

In PAC learning, error tolerance refers to the ability of an algorithm to learn when the examples received have been corrupted in some way. In fact, this is a very common and important issue since in many applications it is not possible to access noise-free data. Noise can interfere with the learning process at different levels: the algorithm may receive data that have been occasionally mislabeled, or the inputs may have some false information, or the classification of the examples may have been maliciously adulterated.

Notation and the Valiant learning model

In the following, let $X$ be our $n$-dimensional input space. Let $\mathcal{H}$ be a class of functions that we wish to use in order to learn a $\{0,1\}$-valued target function $f$ defined over $X$. Let $\mathcal{D}$ be the distribution of the inputs over $X$. The goal of a learning algorithm $\mathcal{A}$ is to choose the best function $h \in \mathcal{H}$ such that it minimizes $error(h) = P_{x \sim \mathcal{D} }( h(x) \neq f(x))$. Let us suppose we have a function $size(f)$ that can measure the complexity of $f$. Let $Oracle(x)$ be an oracle that, whenever called, returns an example $x$ and its correct label $f(x)$.

When no noise corrupts the data, we can define learning in the Valiant setting:^[1][2]

Definition: We say that $f$ is efficiently learnable using $\mathcal{H}$ in the Valiant setting if there exists a learning algorithm $\mathcal{A}$ that has access to $Oracle(x)$ and a polynomial $p(\cdot,\cdot,\cdot,\cdot)$ such that for any $0 < \varepsilon \leq 1$ and $0 < \delta \leq 1$ it outputs, in a number of calls to the oracle bounded by $p\left(\frac{1}{\varepsilon},\frac{1}{\delta},n,size(f)\right)$ , a function $h \in \mathcal{H}$ that satisfies with probability at least $1-\delta$ the condition $error(h) \leq \varepsilon$.

In the following we will define learnability of $f$ when data have suffered some modification.^[3][4][5]

Classification Noise

In the Classification Noise Model^[6] a noise rate $0 \leq \eta < \frac{1}{2}$ is introduced. Then, instead of $Oracle(x)$ that returns always the correct label of example $x$, algorithm $\mathcal{A}$ can only call a faulty oracle $Oracle(x,\eta)$ that will flip the label of $x$ with probability $\eta$. As in the Valiant case, the goal of a learning algorithm $\mathcal{A}$ is to choose the best function $h \in \mathcal{H}$ such that it minimizes $error(h) = P_{x \sim \mathcal{D} }( h(x) \neq f(x))$. In applications it is difficult to have access to the real value of $\eta$, but we assume we have access to its upperbound $\eta_B$.^[7] Note that if we allow the noise rate to be $1/2$, then learning becomes impossible in any amount of computation time, because every label conveys no information about the target function.

Definition: We say that $f$ is efficiently learnable using $\mathcal{H}$ in the classification noise model if there exists a learning algorithm $\mathcal{A}$ that has access to $Oracle(x,\eta)$ and a polynomial $p(\cdot,\cdot,\cdot,\cdot)$ such that for any $0 \leq \eta \leq \frac{1}{2}$, $0\leq \varepsilon \leq 1$ and $0\leq \delta \leq 1$ it outputs, in a number of calls to the oracle bounded by $p\left(\frac{1}{1-2\eta_B}, \frac{1}{\varepsilon},\frac{1}{\delta},n,size(f)\right)$ , a function $h \in \mathcal{H}$ that satisfies with probability at least $1-\delta$ the condition $error(h) \leq \varepsilon$.

Statistical Query Learning

Statistical Query Learning^[8] is a kind of active learning problem in which the learning algorithm $\mathcal{A}$ can decide if to request information about the likelihood $P_{f(x)}$ that a function $f$ correctly labels example $x$, and receives an answer accurate within a tolerance $\alpha$. Formally, whenever the learning algorithm $\mathcal{A}$ calls the oracle $Oracle(x,\alpha)$, it receives as feedback probability $Q_{f(x)}$, such that $Q_{f(x)} - \alpha \leq P_{f(x)} \leq Q_{f(x)} + \alpha$.

Definition: We say that $f$ is efficiently learnable using $\mathcal{H}$ in the Statistical Query Learning Model if there exists a learning algorithm $\mathcal{A}$ that has access to $Oracle(x,\alpha)$ and polynomials $p(\cdot,\cdot,\cdot)$, $q(\cdot,\cdot,\cdot)$, and $r(\cdot,\cdot,\cdot)$ such that for any $0 < \varepsilon \leq 1$ the following hold:

1. $Oracle(x,\alpha)$ can evaluate $P_{f(x)}$ in time $q\left(\frac{1}{\varepsilon},n,size(f)\right)$;
2. $\frac{1}{\alpha}$ is bounded by $r\left(\frac{1}{\varepsilon},n,size(f)\right)$
3. $\mathcal{A}$ outputs a model $h$ such that $err(h)<\varepsilon$, in a number of calls to the oracle bounded by $p\left(\frac{1}{\varepsilon},n,size(f)\right)$.

Note that the confidence parameter $\delta$ does not appear in the definition of learning. This is because the main purpose of $\delta$ is to allow the learning algorithm a small probability of failure due to an unrepresentative sample. Since now $Oracle(x,\alpha)$ always guarantees to meet the approximation criterion $Q_{f(x)} - \alpha \leq P_{f(x)} \leq Q_{f(x)} + \alpha$, the failure probability is no longer needed.

The statistical query model is strictly weaker than the PAC model: any efficiently SQ-learnable class is efficiently PAC learnable in the presence of classification noise, but there exist efficient PAC-learnable problems such as parity that are not efficiently SQ-learnable.^[8]

Malicious Classification

In the Malicious Classification Model^[9] an adversary generates errors to foil the learning algorithm. This setting describes situations of error burst, which may occur when for a limited time transmission equipment malfunctions repeteadly. Formally, algorithm $\mathcal{A}$ calls an oracle $Oracle(x,\beta)$ that returns a correctly labeled example $x$ drawn, as usual, from distribution $\mathcal{D}$ over the input space with probability $1- \beta$, but it returns with probability $\beta$ an example drawn from a distribution that is not related to $\mathcal{D}$. Moreover, this maliciously chosen example may strategically selected by an adversary who has knowledge of $f$, $\beta$, $\mathcal{D}$, or the current progress of the learning algorithm.

Definition: Given a bound $\beta_B< \frac{1}{2}$ for $0 \leq \beta < \frac{1}{2}$, we say that $f$ is efficiently learnable using $\mathcal{H}$ in the Malicious Classification Model if there exist a learning algorithm $\mathcal{A}$ that has access to $Oracle(x,\beta)$ and a polynomial $p(\cdot,\cdot,\cdot,\cdot,\cdot)$ such that for any $0 < \varepsilon \leq 1$, $0 < \delta \leq 1$ it outputs, in a number of calls to the oracle bounded by $p\left(\frac{1}{1/2 - \beta_B},\frac{1}{\varepsilon},\frac{1}{\delta},n,size(f)\right)$ , a function $h \in \mathcal{H}$ that satisfies with probability at least $1-\delta$ the condition $error(h) \leq \varepsilon$.

Errors in the inputs: Nonuniform Random Attribute Noise

In the Nonuniform Random Attribute Noise^[10][11] model the algorithm is learning a Boolean function, a malicious oracle $Oracle(x,\nu)$ may flip each $i$-th bit of example $x=(x_1,x_2,\ldots,x_n)$ independently with probability $\nu_i \leq \nu$.

This type of error can irreparably foil the algorithm, in fact the following theorem holds:

In the Nonuniform Random Attribute Noise setting, an algorithm $\mathcal{A}$ can output a function $h \in \mathcal{H}$ such that $error(h)<\varepsilon$ only if $\nu < 2\varepsilon$.

See also

• Machine learning
• Data mining
• Probably approximately correct learning
• Adversarial machine learning

References

1. Valiant, L. G. (1985, August). Learning Disjunction of Conjunctions. In IJCAI (pp. 560-566).
2. Valiant, Leslie G. "A theory of the learnable." Communications of the ACM 27.11 (1984): 1134-1142.
3. Laird, P. D. (1988). Learning from good and bad data. Kluwer Academic Publishers.
4. Kearns, Michael. "Efficient noise-tolerant learning from statistical queries." Journal of the ACM (JACM) 45.6 (1998): 983-1006.
5. Brunk, Clifford A., and Michael J. Pazzani. "An investigation of noise-tolerant relational concept learning algorithms." Proceedings of the 8th International Workshop on Machine Learning. 1991.
6. Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory, chapter 5. MIT press.
7. Angluin, D., & Laird, P. (1988). Learning from noisy examples. Machine Learning, 2(4), 343-370.
8. Kearns, M. (1998). [www.cis.upenn.edu/~mkearns/papers/sq-journal.pdf Efficient noise-tolerant learning from statistical queries]. Journal of the ACM (JACM), 45(6), 983-1006.
9. Kearns, M., & Li, M. (1993). [www.cis.upenn.edu/~mkearns/papers/malicious.pdf Learning in the presence of malicious errors]. SIAM Journal on Computing, 22(4), 807-837.
10. Goldman, S. A., & Robert, H. (1991). Sloan. The difficulty of random attribute noise. Technical Report WUCS 91 29, Washington University, Department of Computer Science.
11. Sloan, R. H. (1989). Computational learning theory: New models and algorithms (Doctoral dissertation, Massachusetts Institute of Technology).