DNA Sequence Compression using the BurrowsWheeler Transform Don

DNA Sequence Compression using the Burrows-Wheeler Transform Don Adjeroh, Yong Zhang Computer Science and Electrical Engineering West Virginia University, Morgantown, USA Amar Mukherjee Computer Science University of Central Florida, Orlando USA Matt Powell and Tim Bell Computer Science University of Canterbury, Christchurch New Zealand August 16, 2002 1

Outline z Introduction and The Problem z Background z Overview of Approach z BWT and Repeat Analysis z Parsing Strategies z Results z Conclusion 2

Introduction z DNA as an information storage medium z Draft sequence of the human genome now available z Complete genomes available for many other organisms z Implications and Possibilities y Genome-wide analysis of entire genomes y Cross-genome analysis with complete genomes y Drug discovery and medical science y Potential cure for gene-related diseases, such as sickle-cell anemia, Huntington’s disease, Fragile-X mental retardation syndrome, cancer … 3

The problem Size ! 4

The problem of size. . . z Some important genomes are in the order of billions of base pairs z Exponential growth in the number of complete genomes z Exponential growth in the size of available DNA sequence data Source: Genbank website z We need. . . y Efficient and effective algorithms for sequence analysis and interpretation y Efficient techniques for management, organization, and distribution of huge-volume sequence data 5

Nature of DNA Sequences z Four types of nucleotide bases y A - adenine, C - cytosine, G - guanine, T - thymine z Various forms of repetitions y Direct and tandem repeats y Reverse repeats, complimented repeats, palindromes y Combinations y Approximate repeats z Introns and Exons y Coding areas - generally less repetitive y Non-coding areas - generally more redundant y Non-coding areas make up > 50% of genomes z DNA - only part of the whole story 6

DNA Sequence Compression z General Data Compression y Symbol-wise substitution (Huffman codes) y Dictionary-based (LZ family) y Context-based (BWT and PPM) z DNA Sequence Compression y 4 symbols (A, C, G, T) at most 2 bpc on average y Generally dictionary-based y Exploit the different forms of repeat z Key Issues y Speedy identification of the repeats y Parsing with the repeats y Representation of the parsed sequence y Encoding the results Example: S X S Y S Z Y On-line dictionary S X . Y . Z . Off-line dictionary. X . . . S Y Z Dictionary . 7

Overview of Approach S z Off-line dictionary based . X S X . z Repeat analysis via BWT z Variable-length parsing z Integer encoding using a simple Huffman code Y S . . S Y Z Dictionary z Compression using the BWT compression pipeline input Repeat Analysis BWT Repeat Analysis MTF VLC output 8 .

The Burrows-Wheeler Transform z Forward Transform Let T be the input sequence, T=t 1, t 2, …, tu. y Form u permutations of T by cyclic rotations of the characters in T. These form a u u matrix M’. Each row represents one permutation of T; y Sort the rows of M' lexicographically to form another matrix M. y Record L, the last column of M, and id, the row number for the row in M that corresponds to T. The BWT output is the pair (L, 2). 9

The BWT … z Inverse Transform Given only the pair (L, id) y Sort L to produce F, the array of first characters y Compute V, provides 1 -1 mapping between the L and F. F[V[j]]=L[j]. y Generate original sequence T, the symbol L[j] cyclically precedes the symbol F[j] in T, that is, L[V [j]] cyclically precedes the symbol L[j] in T. z BWT Compression Pipeline input BWT MTF VLC output 10

BWT - Auxiliary Arrays z From inverse BWT, F[j] = L[V [j]]. We generate F T mapping: z Let Hr=reverse(H). we have z Define Hrs as the index vector to Hr: z Hr &Hrs can be obtained in linear time from L, V and F. z Example T=ACTAGA 11

BWT, Suffix Trees and Repeats z The BWT provides a lexicographic sorting of the contexts z The BWT is closely related to the suffix tree & suffix array z The suffix array corresponds exactly to our auxiliary array Hrs ! z Repetition analysis as is done on suffix trees can now be done on the BWT output ! 12

Parsing Strategy I -- VPS 1 z Off-line dictionary with pointers in the dictionary z Repetition codes: y 1 - direct repeat; 2 - reverse; 3 palindrome; 4 - compliment; 5 - complimented palindrome; 6 - reverse compliment z General Dictionary structure Dictionary for example z Example: 13

Analysis of VPS 1 z Costs (in bits) y Original sequence, S: y Parsed sequence: y Vocabulary: y Positions , = y Dictionary: z Compression gain: G(S) = z With + , we underestimate the gain: 14

Parsing Strategy II -- VPS 2 z Off-line dictionary with pointers in the sequence z Fixed-length versus variable-length parsing ; Parsing schema 1: <reference index> <r. I> ; Parsing schema 2: <reference index, repeat type > <r. I, r. T> ; Parsing schema 3: <index, repeat type, range> <r. I, r. T, s. P, n. P> z Example Dictionary z With fixed length parsing: y Parse(S): x 1<1, 1>x 2 <3, 1>x 3<2, 1>x 4<4, 1>x 5<2, 2>x 6<1, 5>x 7<3, 1> z With variable-length parsing y Parse(S): x 1<1>x 2<3>x 3<2>x 4<4>x 5<1, 1, 1, 5>x 6<1, 5>x 7<3> 15

Analysis of VPS 2 z Parsing ; Parse. Part 1: x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 ; Parse. Part 1 lengths: sk 1, sk 2, sk 3 sk 4, sk 5, sk 6 sk 7, sk 8, sk 9 ; ski = length of xi ; Parse. Part 2: <1>0<3>0<2>0<4>0<1, 1, 1: 5>0<1, 5>0<3>0<4>0 z Costs (per occurrence costs for each repeat) y Parse. Part 1 Parse. Part 2: y Vocabulary + y Total cost z Per-replacement cost z Compression gain = z Constraints on l(r), n(r), and r. I(r) 16

Encoding the Integers z 1 st order Fibonacci codes (FK 1) z 2 nd order Fibonacci codes (AF 1) z Simple Huffman codes (H 1) F - 00 1 - 010 2 - 011 0 - 1000 3 - 1001 4 - 1010 5 - 1011 6 - 1100 7 - 1101 8 - 1110 9 - 1111 17

Results z Compressed file size (bytes) 18

Results z Compressed file size (bits per character) 19

Results… Size (bytes) Method index 1 - gzip 2 - arith 3 - BWT 4 - vps 5 - vps. H 1 a Size (bpc) 6 - vps. H 1 b 7 - vps. FK 1 a 8 - vps. FK 1 b 9 - vps. AF 1 a 10 - vps. AF 1 b 20

Results … z Computation Time (H 1 versus AF 1 and KF 1) (Time in seconds) 21

Conclusion z BWT presents an effective mechanism for repeat analysis z Not all repeats can lead to compression z Off-line dictionaries are effective for DNA sequence compression z Performance depends on the representation of the parsed sequence, type, length, and number of repeats. z What next ? y Variable-length coding y More rigorous theoretical analysis y Further testing and comparative study (approximate repeats, palindromes, speeding up the process, etc. ) y DNA sequence entropy estimation y Towards compressed domain DNA sequence analysis 22

The End Thank You 23