Ball tree

In computer science, a ball tree, balltree or metric tree,^[1] is a space partitioning data structure for organizing points in a multi-dimensional space. The ball tree gets its name from the fact that it partitions data points into a nested set of hyperspheres known as "balls". The resulting data structure has characteristics that make it useful for a number of applications, most notably nearest neighbor search.

Informal description

A ball tree is a binary tree in which every node defines a D-dimensional hypersphere, or ball, containing a subset of the points to be searched. Each internal node of the tree partitions the data points into two disjoint sets which are associated with different balls. While the balls themselves may intersect, each point is assigned to one or the other ball in the partition according to its distance from the ball's center. Each leaf node in the tree defines a ball and enumerates all data points inside that ball.

Each node in the tree defines the smallest ball that contains all data points in its subtree. This gives rise to the useful property that, for a given test point t, the distance to any point in a ball B in the tree is greater than or equal to the distance from t to the ball. Formally: ^[2]

$D^{Node}(t) = \begin{cases} \max(|t - B.pivot| - B.radius, D^{B.parent}), & \text{if }Node \neq Root \\ \max(|t - B.pivot| - B.radius, 0), & \text{if }B = Root \\ \end{cases}$

Where $D^{B}(t)$ is the minimum possible distance from any point in the ball B to some point t.

Ball-trees are related to the M-tree, but only support binary splits, whereas in the M-tree each level splits m to $2m$ fold, thus leading to a less deep tree structure. The M-tree also keeps the distances from the parent node precomputed to speed up queries.

Construction

A number of ball tree construction algorithms are available.^[3] The goal of such an algorithm is to produce a tree that will efficiently support queries of the desired type (e.g. nearest-neighbor) efficiently in the average case. The specific criteria of an ideal tree will depend on the type of question being answered and the distribution of the underlying data. However, a generally applicable measure of an efficient tree is one that minimizes the total volume of its internal nodes. Given the varied distributions of real-world data sets, this is a difficult task, but there are several heuristics that partition the data well in practice. In general, there is a tradeoff between the cost of constructing a tree and the efficiency achieved by this metric. ^[2]

This section briefly describes the simplest of these algorithms. A more in-depth discussion of five algoriths was given by Stephen Omohundro.^[3]

k-d Construction Algorithm

The simplest such procedure is termed the "k-d Construction Algorithm", by analogy with the process used to construct k-d trees. This is an off-line algorithm, that is, an algorithm that operates on the entire data set at once. The tree is built top-down by recursively splitting the data points into two sets. Splits are chosen along the single dimension with the greatest spread of points, with the sets partitioned by the median value of all points along that dimension. Finding the split for each internal node requires linear time in the number of samples contained in that node, yielding an algorithm with time complexity $O(n\, \log\, n)$, where n is the number of data points.

Pseudocode

   function construct_balltree is
       input: 
           D, an array of data points
       output: 
           B, the root of a constructed ball tree
       if a single point remains then
           create a leaf B containing the single point in D
           return B
       else
           let c be the dimension of greatest spread
           let L,R be the sets of points lying to the left and right of the median along dimension c
           create B with two children: 
               B.pivot = c
               B.child1 = construct_balltree(L),
               B.child2 = construct_balltree(R)
           return B
       end if
   end function

Nearest-neighbor search

An important application of ball trees is expediting nearest neighbor search queries, in which the objective is to find the k points in the tree that are closest to a given test point by some distance metric (e.g. Euclidean distance). A simple search algorithm, sometimes called KNS1, exploits the distance property of the ball tree. In particular, if the algorithm is searching the data structure with a test point t, and has already seen some point p that is closest to t among the points have encountered so far, then any subtree whose ball is further from t than p can be ignored for the rest of the search.

Description

The ball tree nearest-neighbor algorithm examines nodes in depth-first order, starting at the root. During the search, the algorithm maintains a max-first priority queue (often implemented with a heap), denoted Q here, of the k nearest points encountered so far. At each node B, it may perform one of three operations, before finally returning an updated version of the priority queue:

1. If the distance from the test point t to the current node B is greater than the furthest point in Q, ignore B and return Q.
2. If B is a leaf node, scan through every point enumerated in B and update the nearest-neighbor queue appropriately. Return the updated queue.
3. If B is an internal node, call the algorithm recursively on B's two children, searching the child whose center is closer to t first. Return the queue after each of these calls has updated it in turn.

Performing the recursive search in the order described in point 3 above increases likelihood that the further child will be pruned entirely during the search.

Pseudocode

   function knn_search is
       input: 
           t, the target point for the query
           k, the number of nearest neighbors of t to search for
           Q, max-first priority queue containing at most k points
           B, a node, or ball, in the tree
       output: 
           Q, containing the k nearest neighbors from within B
       if distance(t, B.pivot) ≥ distance(t, Q.first) then
           return Q unchanged
       else if B is a leaf node then
           for each point p in B do
               if distance(t, p) < distance(t, Q.first) then
                   add p to Q
                   if size(Q) > k then
                       remove the furthest neighbor from Q
                   end if
               end if
           repeat
       else
           let child1 be the child node closest to t
           let child2 be the child node furthest from t
           knn_search(t, k, Q, child1)
           knn_search(t, k, Q, child2)
       end if
   end function[2]

Performance

In comparison with several other data structures, ball trees have been shown to perform fairly well on the nearest-neighbor search problem, particularly as their number of dimensions grows.^[1][4] However, the best nearest-neighbor data structure for a given application will depend on the dimensionality, number of data points, and underlying structure of the data.

References

1. Kibriya, A. M.; Frank, E. (2007). "An Empirical Comparison of Exact Nearest Neighbour Algorithms". Knowledge Discovery in Databases: PKDD 2007 (PDF). Lecture Notes in Computer Science 4702. p. 140. doi:10.1007/978-3-540-74976-9_16. ISBN 978-3-540-74975-2.
2. Liu, T.; Moore, A. and Gray, A. (2006). "New Algorithms for Efficient High-Dimensional Nonparametric Classification" (PDF). Journal of Machine Learning Research 7: 1135–1158.
3. Omohundro, Stephen M. (1989) "Five Balltree Construction Algorithms"
4. Kumar, N.; Zhang, L.; Nayar, S. (2008). "What is a Good Nearest Neighbors Algorithm for Finding Similar Patches in Images?". Computer Vision – ECCV 2008 (PDF). Lecture Notes in Computer Science 5303. p. 364. doi:10.1007/978-3-540-88688-4_27. ISBN 978-3-540-88685-3.