Count-distinct problem

In computer science, the count-distinct problem^[1] (also known in applied mathematics as the cardinality estimation problem) is the problem of finding the number of distinct elements in a data stream with repeated elements. This is a well-known problem with numerous applications. The elements might represent IP addresses of packets passing through a router, unique visitors to a web site, elements in a large database, motifs in a DNA sequence, or elements of RFID/sensor networks.

Formal definition

Instance: A stream of elements $x_1,x_2,\ldots,x_s$ with repetitions, and an integer $m$. Let $n$ be the number of distinct elements, namely $n = |\left\{ {x_1,x_2,\ldots,x_s}\right\}|$, and let these elements be $\left\{ {e_1,e_2,\ldots,e_n}\right\}$.

Objective: Find an estimate $\widehat{n}$ of $n$ using only $m$ storage units, where $m \ll n$.

An example of an instance for the cardinality estimation problem is the stream: $a,b,a,c,d,b,d$. For this instance, $n = |\left\{ {a,b,c,d}\right\}| = 4$.

Naive solution

The naive solution to the problem is as follows:

 Initialize a counter, c, to zero, $c \leftarrow 0$.
 Initialize an efficient dictionary data structure, D, such as hash table or search tree in which insertion and membership can be performed quickly.  
 For each element $x_i$, a membership query is issued. 
 If $x_i$ is not a member of D ($x_i \notin D$)
   Add $x_i$ to D
   Increase c by one, $c \leftarrow c + 1$
 Otherwise ($x_i \in D$) do nothing.
 Output $n = c$.

As long as the number of distinct elements is not too big, D fits in main memory and an exact answer can be retrieved. However, this approach does not scale for bounded storage, or if the computation performed for each element $x_i$ should be minimized. In such a case, several streaming algorithms have been proposed which use a fixed number of storage units.

Streaming algorithms

To handle the bounded storage constraint, streaming algorithms use a randomization to produce a non-exact estimation of the distinct number of elements, $n$. State-of-the-art estimators hash every element $e_j$ into a low dimensional data sketch using a hash function, $h(e_j)$. The different techniques can be classified according to the data sketches they store.

Min/max sketches

Min/max sketches ^{[2] [3]} store only the minimum/maximum hashed values. Examples of known min/max sketch estimators: Chassaing et al. ^[4] presents max sketch which is the minimum-variance unbiased estimator for the problem. The continuous max sketches estimator ^[5] is the maximum likelihood estimator. The estimator of choice in practice is the HyperLogLog algorithm.^[6]

The intuition behind such estimators is that each sketch carries information about the desired quantity. For example, when every element $e_j$ is associated with a uniform RV, $h(e_j) \sim U(0,1)$, the expected minimum value of $h(e_1),h(e_2), \ldots, h(e_n)$ is $1/(n+1)$. The hash function guarantees that $h(e_j)$ is identical for all the appearances of $e_j$. Thus, the existence of duplicates does not affect the value of the extreme order statistics.

There are other estimation techniques other than min/max sketches. The first paper on count-distinct estimation by Flajolet et al. ^[7] describes a bit pattern sketch. In this case, the elements are hashed into a bit vector and the sketch holds the logical OR of all hashed values. The first asymptotically space- and time-optimal algorithm for this problem was given by Daniel M. Kane, Jelani Nelson, and David P. Woodruff.^[8]

Bottom-m sketches

Bottom-m sketches ^[9] are a generalization of min sketches, which maintain the $m$ minimal values, where $m \geq 1$. See Cosma et al.^[2] for a theoretical overview of count-distinct estimation algorithms, and Metwally ^[10] for a practical overview with comparative simulation results.

Weighted count-distinct problem

In its weighted version, each element is associated with a weight and the goal is to estimate the total sum of weights. Formally,

Instance: A stream of weighted elements $x_1,x_2,\ldots,x_s$ with repetitions, and an integer $m$. Let $n$ be the number of distinct elements, namely $n = |\left\{ {x_1,x_2,\ldots,x_s}\right\}|$, and let these elements be $\left\{ {e_1,e_2,\ldots,e_n}\right\}$. Finally, let $w_j$ be the weight of $e_j$.

Objective: Find an estimate $\widehat{w}$ of $w = \sum_{j=1}^{n}w_j$ using only $m$ storage units, where $m \ll n$.

An example of an instance for the weighted problem is: $a(3),b(4),a(3),c(2),d(3),b(4),d(3)$. For this instance, $e_1=a, e_2=b, e_3=c, e_4=d$, the weights are $w_1=3, w_2=4, w_3=2, w_4=3$ and $\sum{w_j}=12$.

As an application example, $x_1,x_2,\ldots,x_s$ could be IP packets received by a server. Each packet belongs to one of $n$ IP flows $e_1,e_2,\ldots,e_n$. The weight $w_j$ can be the load imposed by flow $e_j$ on the server. Thus, $\sum_{j=1}^{n}{w_j}$ represents the total load imposed on the server by all the flows to which packets $x_1,x_2,\ldots,x_s$ belong.

Solving the weighted count-distinct problem

Any extreme order statistics estimator (min/max sketches) for the unweighted problem can be generalized to an estimator for the weighted problem .^[11] For example, the weighted estimator proposed by Cohen et al.^[5] can be obtained when the continuous max sketches estimator is extended to solve the weighted problem. In particular, the HyperLogLog algorithm ^[6] can be extended to solve the weighted problem. The extended HyperLogLog algorithm offers the best performance, in terms of statistical accuracy and memory usage, among all the other known algorithms for the weighted problem.

See also

• HyperLogLog
• Count–min sketch
• Streaming algorithm
• Maximum likelihood
• Minimum-variance unbiased estimator

References

1. Ullman, Jeff; Rajaraman, Anand; Leskovec, Jure. "Mining data streams" (PDF).
2. Cosma, Ioana A.; Clifford, Peter (2011). "A statistical analysis of probabilistic counting algorithms". Scandinavian Journal of Statistics.
3. Giroire, Frederic; Fusy, Eric (2007). "Estimating the Number of Active Flows in a Data Stream over a Sliding Window" (PDF). ANALCO.
4. Chassaing, Philippe; Gerin, Lucas (2006). "Efficient estimation of the cardinality of large data sets". Proceedings of the 4th Colloquium on Mathematics and Computer Science.
5. Cohen, Edith (1997). "Size-estimation framework with applications to transitive closure and reachability". J. Comput. Syst. Sci.
6. Flajolet, Philippe; Fusy, Eric; Gandouet, Olivier; Meunier, Frederic (2007). "HyperLoglog: the analysis of a near-optimal cardinality estimation algorithm". Analysis of Algorithms.
7. Flajolet, Philippe; Martin, G. Nigel (1985). "Probabilistic counting algorithms for data base applications". J. Comput. Syst. Sci. (Academic Press, Inc.).
8. Kane, Daniel M.; Nelson, Jelani; Woodruff, David P. (2010). "An Optimal Algorithm for the Distinct Elements Problem". Proceedings of the 29th Annual ACM Symposium on Principles of Database Systems (PODS).
9. Cohen, Edith; Kaplan, Haim (2008). "Tighter estimation using bottom k sketches" (PDF). PVLDB (Academic Press, Inc.).
10. Metwally, Ahmed; Agrawal, Divyakant; Abbadi, Amr El (2008), Why go logarithmic if we can go linear?: Towards effective distinct counting of search traffic, Proceedings of the 11th international conference on Extending Database Technology: Advances in Database Technology
11. Cohen, Reuven; Katzir, Liran; Yehezkel, Aviv (2014). "A Unified Scheme for Generalizing Cardinality Estimators to Sum Aggregation". Information Processing Letters 115: 336–342. doi:10.1016/j.ipl.2014.10.009.