Suffix Arrays A new method for online string

Suffix Arrays: A new method for on-line string searches l l Presented by: Oren Weimann 1 Udi Manber Gene Myers May 1989

Introduction - Problem definition “Is W a substring of A? ” Ø Ø Ø 2 |A|=N and |W|=P A = a 0 a 1…a. N-1 Ai = suffix beginning at index i = aiai+1…a. N-1 W= badgfbb A= abccbbadgfbbcahgjf

Introduction – what is a suffix array? Example: A = assassin 0 1 2 3 4 5 6 7 Pos[2] = 6 (A 6 = in) 3 Pos

Introduction – what is a suffix array? A lexicographically sorted array- Pos[N], of all the suffixes of A: Pos[k] = i Ai is the kth smallest suffix in the set {A 0, A 1, A 2…… AN-1} 4

Introduction – what is a suffix tree? Example: l A trie that contains all suffixes of A: A = assassin 0 1 2 3 4 5 6 7 5 n 0 i s s a i n 3 s i n 6 s a s s i n 1 i i n a n s s 5 4 i n 2

The Article Overview 1. 2. 3. 4. 6 A search algorithm In O(P+log. N) (assuming we already computed Pos[ ] and the longest common prefix (lcp) information). How to construct Pos[ ] in O(Nlog. N) time and O(N) space. (assuming lcp info is known) An Algorithm for computing the lcp information in O(Nlog. N). Algorithms for Expected-time improvement.

The Search algorithm p l For any string u, u = u u Definitions 1 l l l 7 2 3……. p (or u if |u| p) Let “ “ denote a Lexicographical order, We say u v up vp Note that for any choice of p: Note that W is a substring of A there is an i such that W

The Search algorithm – how does the array help us know if W is a substring of A? l l 8 We define a search interval: LW = min {k | W APos[k] or k = N} RW = max {k | W APos[k] or k = -1} W matches ai ai+1. . . ai+P-1 i=Pos[k] for some k [LW, RW]

Example: 0 1 2 3 4 5 6 7 Pos A = assassin Option 1 Option 2 Option 3 9

Why finding LW, RW == Finding the matches: l l If LW > RW => W is not a substring of A. Else: there are (RW-LW+1) matches APos[LW], …, APos[RW] Pos W>APos[k] 10 LW RW W<APos[k]

The Search algorithm – The easy way - O(Plog. N) W=“abcx” Pos abcde. . . abcdf. . . abd. . . L M R Log(N) iterations, each iteration sets new L, R bonds (initially L=0, R=N-1) according to a comparison of W with APos[M] , where M=(L+R)/2. 11 In the end LW R

The Search algorithm using lcp values in O(P+log. N) – Definitions: Speedup using precomputed lcp Values, for now We assume lcp is known. Each iteration We define: – l = lcp(APos[L], W) – r=lcp(W, APos[R]) – – 12 Llcp[M] = lcp(APos[L] APos[M]) Rlcp[M] = lcp(APos[M], APos[R])

The Search algorithm using lcp values in O(P+log. N) Example: A=“abcx” l=3 Pos r=2 abcde. . . abcdf. . . abd. . . L M R Llcp[M]=4 Rlcp[M]=2 Note that Llcp[M] is well defined because every midpoint M has one LM and one RM 13

So how do we use l, r, Llcp[M] ? Example: W=abcx l=3 abcde. . . L 14 Llcp[M]=4 abc. . . r=2 abc. . . abcdf… M Case 1: Llcp[M] > l (Llcp[M]=4 and l=3 ) W>APos[L] ]W>APos[M] ]Go right ]l is unchanged = 3 abd… R

Example: W=abcx (cont. ) l=3 abcde. . . L Llcp[M]=2 r=2 abdf… M Case 2: Llcp[M] < l (Llcp[M]=2 and l=3 ) APos[L] <APos[M] ð W<APos[M] ð Go left ð r = Llcp[M] = 2 15 abd… R

Example: W=abcx (cont. ) l=3 abcde. . . L 16 Llcp[M]=3 abc. . . r=2 abcp… abd… M R Case 3: Llcp[M] = l (Llcp[M]=3 and l=3 ) Compare Wl and APos[M]l until Wl+j APos[M]l+j ] Go right or left according to Wl+j, APos[M] l+j ] new l or r = (l+j) ] Number of comparisons = j+1

The Search algorithm using lcp valuescomplexity In each iteration there are maximum j+1 comparisons, when in total ] Total comparisons (P + #Iterations) ] O(P+log. N) running time l 17 Requires only 3 N-sized arrays

The Article Overview 1. 2. 3. 4. 18 A search algorithm In O(P+log. N) (assuming we already computed Pos[ ] and the longest common prefix (lcp) information). How to construct Pos[ ] in O(Nlog. N) time and O(N) space. (assuming lcp info is known) An Algorithm for computing the lcp information in O(Nlog. N). Algorithms for Expected-time improvement.

Construction of suffix array in O(Nlog. N) Sorting the suffixes in a unique Radix sort – We Will have O(log. N) stages (numbered 1, 2, 4, 8, 16…) In stage H the suffixes are sorted in buckets called H Buckets, according to the first H characters. (next stage is 2 H– thus, in stage H the suffixes are sorted by ) 19

Construction of suffix array – The general idea If Ai, Aj H-bucket we Sort them by the Next H symbols, but: first bucket second bucket third bucket fourth bucket Their next H symbols = first H symbols of abcd… ab… bb. . . bb… cd… ef… Aabef… and A which are already sorted in phase H. i+H j+H 20 Ai Aj H=2: Aj+H Ai+H

Construction of suffix array – The general idea (cont. ) Let Ai be in first H-bucket after stage H l Ai starts with smallest H-symbol string labef… Ai-H abcd… should ab… be first bb. . . in its bb…H-bucket cdef… cdab… l 21 Ai H=2: Ai-H ef…

Construction of suffix array – The algorithm l l 22 Go over the suffix array: For each Ai: Move Ai-H to next available place in its H-bucket The suffixes are now sorted according to -order Go over the array again, and decide which suffix opens a new 2 H-bucket, use lcs knowledge (described later)

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin A 3 assassin in n ssin sassin A 2 H=1 Ai sets Ai-1 23 sin ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin assassin A 0 in n H=1 Ai sets Ai-1 24 sassin sin ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin assassin in A 6 n H=1 Ai sets Ai-1 25 sassin sin A 5 ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin assassin in n A 6 A 7 H=1 Ai sets Ai-1 26 sassin ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin assassin in n sassin A 2 H=1 Ai sets Ai-1 27 sin ssassin A 1

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin assassin in n H=1 Ai sets Ai-1 28 sassin A 5 ssassin A 4

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assin assassin in n A 0 H=1 Ai sets Ai-1 29 sassin ssassin A 1 ssin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in n A 3 H=1 Ai sets Ai-1 30 sassin ssassin A 4

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 Go over array to get new 2 -buckets assassin in n sassin ssassin lcs(sassin, sin)= 1+ lcs(assin, in)= 1+0=1 so “sin” opens a new 2 -bucket H=1 31 back Ai sets Ai-1

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in n A 0 H=2 Ai sets Ai-2 32 sassin ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in n sin ssassin A 1 A 3 H=2 Ai sets Ai-2 33 sassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in A 6 n sin ssassin A 4 H=2 Ai sets Ai-2 34 sassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in n A 7 H=2 Ai sets Ai-2 35 sassin A 5 ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin A 0 assin in n H=2 Ai sets Ai-2 36 sassin A 2 sin ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin A 3 in n H=2 Ai sets Ai-2 37 sassin A 5 ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in n H=2 Ai sets Ai-2 38 sassin ssassin A 1 ssin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 assassin in n H=2 Ai sets Ai-2 39 sassin A 2 sin ssassin A 4

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 Go over array to get new 4 -buckets assassin in n H=2 Ai sets Ai-2 40 sassin ssassin

Construction of suffix array – The algorithm Example: A = assassin 0 1 2 3 4 5 6 7 That’s it, we are sorted! assassin in n H=4 41 sassin ssassin

Construction of suffix array – Complexity Summary l l l Sorting by first char – O(N) O(log. N) stages of O(N) operations = O(Nlog. N) Total - time: O(Nlog. N) - space: 2 integer arrays of size N back 42

The Article Overview 1. 2. 3. 4. 43 A search algorithm In O(P+log. N) (assuming we already computed Pos[ ] and the longest common prefix (lcp) information). How to construct Pos[ ] in O(Nlog. N) time and O(N) space. An Algorithm for computing the lcp information in O(Nlog. N). Algorithms for Expected-time improvement.

How to find Longest Common Prefixes – the general idea l l 44 We don’t care what is the lcp between suffixes in the same H-bucket. For Ap, Aq in the same H-bucket but different 2 H-buckets: – H lcp(Ap, Aq) < 2 H – lcp(Ap, Aq) = H + lcp(Ap+H, Aq+H) – lcp(Ap+H, Aq+H) < H that is why Ap+H, Aq+H Are in different H-buckets, but which ones?

How to find Longest Common Prefixes – the general idea l l 45 If Ap+H and Aq+H were in adjacent H-buckets then lcp is known. how? If not, Then: lcp(APos[i], APos[j]) = {lcp(APos[k], APos[k+1])}

How to find Longest Common Prefixes – the general idea lcp(Ap+H, Aq+H) = min{1, 1, 2} = 1 H=2 assassin in 1 n sassin Ap+h 46 1 sin ssassin 2 ssin Aq+h Notice that if 2 neighbors are in the same H-bucket we can consider there lcp to be H, since lcp(Ap+H, Aq+H) < H

How to find lcp – algorithm and data structures – Hgt[] During the construction stage, we build an array Called Hgt[N]: Hgt(i)=lcp(APos[i-1], APos[i, ([ initialized so that Hgt[i]=N+1 for every i. In stage H=1: Hgt(i)=0 for APos[i] that are first in their buckets. In stage 2 H: we update every Hgt(i) that APos[i] is the first in a newly created 2 H bucket 47

How to find lcp – Hgt[] example: H=1 assin assassin 9 in n sin sassin ssassin 0 0 0 9 in n sassin 0 0 0 1 9 9 H=2 assin assassin 9 48 ssin ssassin 1 9 lcp(ssin, sin)=1+lcp(sin, in)=1+min{lcp(in, n), lcp(sin, n)}=1

How to find lcp – Hgt[] example (cont. ) H=4 assassin 3 in n sassin 0 0 0 1 ssassin 1 2 lcp(assassin, assin)=2+lcp(sin, sassin)=2+1=3 lcp(ssin, ssassin)=2+lcp(in, assin)=2+0=2 49

How to find lcp – data structures We need a data structure that will contain lcp(APos[j , [APos[i]) between any i and j (not just i and i+1 which Hgt[] supplies) Hgt[] will become the leaves of a binary balanced tree called the Interval tree. 50

How to find lcp – example of Interval tree 0 91 0 0 )0, 1( 51 93 0 91 )1, 2( )2, 3( )3, 4( 0 0 0 )4, 5( )5, 6( )6, 7( 91 91 92

How to find lcp – Complexity l l l Each time a leaf opens a new bucket we change Hgt[i] for that leaf. That change requires O(log. N) changes in the interval tree There are O(N) leaves opening new bucket In total we get O(Nlog. N) to get all lcp values 52

The Article Overview 1. 2. 3. 4. 53 A search algorithm In O(P+log. N) (assuming we already computed Pos[ ] and the longest common prefix (lcp) information). How to construct Pos[ ] in O(Nlog. N) time and O(N) space. An Algorithm for computing the lcp information in O(Nlog. N). Algorithms for Expected-time improvement.

Time Expected-case Improvement of the construction of pos[] l l Assumptions: - All N-symbol strings are equally likely. – Under this assumption: Expected length of longest repeated substring = O(log| |N) This immediately implies that construction of pos[] is reduced to O(NLog. N). why? Next is a way to reduce it to O(N). 54

Time Expected-case Improvement of the construction of pos[] l l l 55 Let T = We encode each possible T length string to an integer with the isomorphism Int. T(u) Map each AP to Int. T(AP) T-1 [0, | |T-1] :

Example of the mapping 0*4^0 + 0 0 | |= 4 , a=0, i=1, n=2, s=3 3*4^0 + 0 3 N=8 3*4^0 T= +0 0*4^0 + 0 3*4^0 + 0 Int. T(AP) = ap| |T-13*4^0 + +0 56 1*4^0 + 0 2*4^0 + 0 3=1 0 3 3 1 2

Time Expected-case Improvement of the construction of pos[] By the definition of Int. T(AP) it takes O(N) to compute all Int. T(AP) values of all suffixes. So now instead of starting with H=1 we start with H= But since the longest repeated substring length is O(log| |N) we will have O(1) stages of the radix sort. Thus, the total time for constructing pos[] = O(N) 57

So is a suffix array better then a suffix tree? Suffix array Construction O(Nlog. N) - for small | | time O(N) – needs additional space 58 Time Complexity Space Complexity dependent on | | ? Suffix tree O(N) O(P+log. N) – good for large alphabets O(Plog| |) requires 2 N integers – this is the main advantage. O(N) No Yes

59