The VariableIncrement Counting Bloom Filter Ori Rottenstreich Joint

The Variable-Increment Counting Bloom Filter Ori Rottenstreich Joint work with Yossi Kanizo and Isaac Keslassy Technion, Israel

Problem Definition Flow yx Flow z Flow u y Yes No • Support queries of the form Flow y Set S (Special Flows) • Requirements for data structure: o Space efficient o Fast (Insertion, Query) 2

Naïve Solutions Flow x Flow y • O(n) – Searching in a list Flow z • O(log(n)) – Searching in a sorted list Set S (Special Flows) • O(1) ? o Tradeoff: We allow False Positives with low probability • Two possible errors o False Positives o False Negatives - but the answer is 3

Bloom Filters (Bloom, 1970) • Initialization: Array of 0 0 zero bits. 0 0 0 0 0 • Insertion: Each of the elements is hashed times, the corresponding bits are set. • Query: Hashing the element, checking that all bits are set. y x 1 0 0 1 x 1 1 0 0 1 11 0 • False positive rate (probability) of • No false negatives. 1 0 1 z 0 1 1 0 0 11 1 w 1 0 0 . 4

Counting Bloom Filters (CBFs) • Bloom filters do not support deletions of elements. Simply resetting bits might cause false negatives. y x 1 0 1 1 0 0 0 • The solution: Counting Bloom filters - Storing array of instead of bits. o Insertion: Incrementing counters by one. o Deletion: Decrementing counters by one. o Query: Checking that counters are positive. y x +1 +1 0 +1 +1 1 0 0 2 0 +1 1 counters +1 0 1 • The same false positive probability. • Require too much memory, e. g. 57 bits per element for 0 .

(Counting) Bloom Filters are Widely Used • • • Packet Classification Intrusion Detection Routing Accounting Beyond networking: Spell Checking, DNA Classification • Can be found in o Google's web browser Chrome o Google's database system Big. Table o Facebook's distributed storage system Cassandra o Mellanox's IB Switch System 6

Outline Ø Introduction to Bloom Filters Ø The Variable-Increment Counting Bloom Filter o Intuition for Variable Increments o The Bh-CBF Scheme o The VI-CBF Scheme Ø Experimental Results Ø Summary 7

Intuition for Variable Increments • Upon query, we should consider the exact values of the counters and not just their positiveness. 0 1 0 2 x 4 0 1 7 0 1 2 1 y • Idea: Use variable increments to encode the element identity. 8

Architecture • Each hash entry contains a pair of counters: o , fixed increments → number of elements in entry (as in CBF) o , variable increments → weighted sum of elements o weights from a pre-determined set • We use two sets of hash functions: o The first set uses hash functions with range , i. e. it points to the set of entries. o The second set uses hash functions with range , i. e. it points to the set. c 1 c 2 1 2 3 4 5 6 7 8 9 0 5 3 2 2 3 3 3 4 0 34 25 26 17 21 9 6 26 9

Insertion • Insertion: At each entry , the two counters are updated as follows. o from the set o • Example 1: c 1 c 2 1 2 3 4 5 6 7 8 9 001 5 334 2 324 3 43 5 3 4 00 8 34 25 2529 17 301743 21 30934 13 26 +8 x +4 +13 z +4 10

Query • Query c 1 c 2 Flow y ( with ) 1 2 3 4 5 6 7 8 9 0 5 3 2 3 3 4 0 34 25 17 30 21 30 13 26 4? 8? y? • We ask whether o 17 can be a sum of 2 elements from the set o 30 can be a sum of 3 elements from the set • No: • How should we pick the set of variable increments? We should use including 4 including 8 Sequences! 11

Bh Sequences • Definition 1: Let Then, with is a be a sequence of positive integers. sequence iff all the sums are distinct. • Example 2: All the sums of • elements of are distinct: Therefore, is a sequences are widely used in error-correcting codes. 12

The Bh-CBF Scheme Query • Example 3: c 1 c 2 is a sequence 1 2 3 4 5 6 7 8 9 0 5 3 2 3 3 4 34 25 17 30 21 30 13 26 0 1? 4? X? o Since , then the Bh-CBF can determine that 13

The Bh-CBF Scheme Operations Query • Example 3: c 1 c 2 is a sequence 1 2 3 4 5 6 7 8 9 0 5 3 2 3 3 4 0 34 25 17 30 21 30 13 26 1? X? o Here, Since 4? 8? y? and then necessarily , the Bh-CBF can determine that 13

The Bh-CBF Scheme Operations Query • Example 3: c 1 c 2 is a sequence 1 2 3 4 5 6 7 8 9 0 5 3 2 3 3 4 0 34 25 17 30 21 30 13 26 1? X? o Since 4? y? 8? 4? 13? z? , the Bh-CBF cannot exclude that 13

Outline Ø Introduction to Bloom Filters Ø The Variable-Increment Counting Bloom Filter o Intuition for Variable Increments o The Bh-CBF Scheme o The VI-CBF Scheme Ø Experimental Results Ø Summary 14

The VI-CBF Scheme Principles • Two counters in each hash entry use more space. • Can we only keep the variable increment counter? c 1 c 2 1 2 3 4 5 6 7 8 9 0 5 3 2 2 3 3 3 4 0 34 25 26 17 21 9 6 26 • In the VI-CBF (Variable-Increment Counting Bloom Filter), each hash entry only contains the variable-increment counter. • The counter is updated like the variable-increment counter in the Bh-CBF. 15

The VI-CBF Scheme Principles • Problem: We do not know the number of elements in each hash entry. • Example 4: (with the c 1 c 2 sequence ) 1 2 3 4 5 6 7 8 9 0 5 3 2 3 3 4 0 34 25 17 30 21 30 13 26 4? 8? y? o including 8 o However, from the set cannot be a sum of 3 elements from the set can be a sum of 5 elements including 8 16

The VI-CBF Scheme Principles • In the VI-CBF , the set of variable increments not necessarily a sequence is • Example 5: x +7 0 0 5 0 5 o Based on +5 +4 +4 +5 or 0 0 9 7 z 0 y 0 4 +5 0 0 5 0 6 , the VI-CBF can deduce that 17

A Simple Option for D: DL = [L, 2 L-1] • For , we define the set of size as • Intuition: sum of zero elements not possible sum of one element sum of two or more elements not possible • Lemma 1: Let be an element whose -th hash function entry of the value If hashes into an then 18

VI-CBF Outperforms CBF • Theorem 1: While keeping the same bit-per-element ratio the following properties when compared to CBF: , VI-CBF satisfies (i) VI-CBF obtains a lower false positive rate than CBF. (ii) (iii) VI-CBF obtains a lower counter overflow probability bound than the classical bound of CBF. • Cost: Limited implementation overhead. 19

Outline Ø Introduction to Bloom Filters Ø The Variable-Increment Counting Bloom Filter o Intuition for Variable Increments o The Bh-CBF Scheme o The VI-CBF Scheme Ø Experimental Results Ø Summary 20

Experimental Results • Internet trace (equinix-chicago) with real hash functions. For the Bh-CBF, (with ). For the VI-CBF, and. . 21

Concluding Remarks • Encoding the element identity using Variable Increments o Considering the exact values of the counters upon query • Can extend many variants of the counting Bloom filter • First time sequences are presented in networking applications 22

Thank You