Fulltext index suffix tree Weiner 73 suffix array

Full-text indexのデータ構造 • suffix tree [Weiner 73] • suffix array [Manber, Myers 93] • String B-tree[Ferragina, Grossi 95] 7

I/O complexity • 検索のI/O complexity • 更新のI/O complexity • 構成のI/O complexity 11

検索のI/O complexity ・Suffix tree – Nに依存しない ○ String B-tree ・Suffix array p : キーワード長 occ : 答えの数 N : 文字列長 12

構成のI/O complexity ◎suffix tree (optimal) ○ suffix array ・String B-tree N : 文字列長 M : メモリサイズ B : ディスクページサイズ 14

Suffix treeの構成 • メモリ上 (線形時間) * Weiner 73 * Mc. Creight 76 * Ukkonen 95 * Farach 97 • divide and conquer, batch処理 • ディスク上 * Farach, Ferragina, Muthukrishnan 98 16

Disk上でのsuffix tree構成 • アルゴリズムをsortingとscanで表現 • 数のsortingと同じI/O complexity (optimal) I/O 17

Sorting I/O complexity • 次の問題はsortingと同じI/O complexityを持つ – treeのノードのlcaをK個 (K個のrange minima) – tree T のEuler Tour ET(T)とノードの深さ – 文字列中の任意の位置のK文字 – treeの各ノードの子孫でmarkされているもの – uncompacted trieのmerge – suffix treeの全てのsuffix link – suffix treeの構成 18

Block vs. Random I/O • 2 -way merge • M/B-way merge 補題: random I/Oが アルゴリズムは block I/O random I/O 回のsorting 回のI/Oを必要とする。 19

Algorithm outline $ a b • Odd treeを作る a $ • Even treeを作る b $ • mergeする Odd tree b $ a b $ Even tree $a b b a $ b $ 21

Building the even tree • 偶数番目のsuffixを辞書順にradix sortする – (先頭の文字, 奇数番目のsuffixの辞書順) • 隣り合うsuffix間のlcpを求める • compacted trieを作る abab$ 2 4 $ a b 1 $ a b $ 2 3 2: (b, ab$) = (b, 2) 4: (b, $) = (b, 1) b $ a b $ Even tree 23

Merging the odd and even trees • • • anchor pairを見つける side tree pairに分割する pull nodeを見つける merge nodeを見つける TeとToをmergeする 24

Suffix arrayのメモリ上での構成 • quick sort × 文字列の比較なので非常に遅い • ternary partitioning[Bentley, Sedgewick 97] ○ 無駄な文字列比較が少ない × 極端に遅くなることがある • doubling algorithm – Manber, Myers 93 – Sadakane, Imai 98 • 多くの場合最速 25

Suffix arrayのディスク上での構成 • Gonnet, Baeza-Yates, Snider 92 – diskはsequential accessのみ I/O • Crauser, Ferragina 98 – doubling algorithm + discarding I/O 32

Word indexes vs. Full-text indexes

参考文献(1/3) [1] L. Arge, P. Ferragina, R. Grossi, and J. S. Vitter. On sorting strings in external memory. In ACM Symposium on Theory of Computing, pp. 540 --548, 1997. URL [2] J. L. Bentley and R. Sedgewick. Fast algorithms for sorting and searching strings. In Proceedings of the 8 th Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 360 URL --369, 1997. [3] M. Burrows and D. J. Wheeler. A Block-sorting Lossless Data Compression Algorithms. Technical Report 124, Digital SRC Research Report, 1994. URL [4] A. Crauser and P. Ferragina. External memory construction of full-text indexes. In DIMACS Workshop on External Memory Algorithms and/or Visualization, 1998. URL [5] M. Farach. Optimal Suffix Tree Construction with Large Alphabets. In 38 th Symp. on Foundations of Computer Science, pp. 137 --143, 1997. URL 40

参考文献(2/3) [6] P. Ferragina and R. Grossi. The String B-Tree: a new data structure for string search in external memory and its applications. Journal of the ACM, 1998. URL [7] G. H. Gonnet, R. Baeza-Yates, and T. Snider. New Indices for Text: PAT trees and PAT arrays. In W. Frakes and R. Baeza-Yates, editors, Information Retrieval: Algorithms and Data Structures, chapter 5, pp. 66 --82. Prentice-Hall, 1992. URL [8] R. M. Karp, R. E. Miller, and A. L. Rosenberg. Rapid identification of repeated patterns in strings, arrays and trees. In 4 th ACM Symposium on Theory of Computing, pp. 125 -136, 1972. [9] U. Manber and G. Myers. Suffix arrays: A New Method for On-Line String Searches. SIAM Journal on Computing, Vol. 22, No. 5, pp. 935 --948, October 1993. 41

参考文献(3/3) [10] E. M. Mc. Creight. A space-economical suffix tree construction algorithm. Journal of the ACM, Vol. 23, No. 12, pp. 262 --272, 1976. [11] K. Sadakane and H. Imai. A Cooperative Distributed Text Database Management Method Unifying Search and Compression Based on the Burrows-Wheeler Transformation. In URL Proceedings of New. DB’ 98, 1998. [12] K. Sadakane and H. Imai. Constructing Suffix Arrays of Large Texts. In Proceedings of DEWS'98, 1998. URL [13] E. Ukkonen. On-line construction of suffix trees. Algorithmica, Vol. 14, No. 3, pp. 249 -260, September 1995. [14] P. Weiner. Linear Pattern Matching Algorihms. In Proceedings of the 14 th IEEE Symposium on Switching and Automata Theory, pp. 1 --11, 1973. 42