Fast HASH A New Algorithm for Fast and

Fast. HASH: A New Algorithm for Fast and Comprehensive Next-generation Sequence Mapping Hongyi Xin 1, Donghyuk Lee 1, Farhad Hormozdiari 2, Can Alkan 3, Onur Mutlu 1 Departments of Computer Science and Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 2 Department of Computer Science, University of California Los Angeles, CA 3 Department of Genome Sciences, University of Washington, Seattle, WA 1 Next-generation DNA Sequencing and the State-of-the-art Sequence Mapping Tools mr. FAST Background: DNA Sequencing Challenge of Next-generation DNA Sequencing Existing Mapping Tools • Goal: Acquire individual’s entire DNA sequence • Mechanism: Read DNA fragments and reconstruct it Break DNA into pieces and store them as strings Compare the strings to a known reference DNA string -- Search for matching coordinates in reference DNA Stitch fragments together in corresponding order • Difficulties: Individuals have mutations including Mismatch, insertions and deletions; must tolerate • Next-generation DNA Sequencing: Instead of reading fewer long fragments, read many short fragments in parallel This pushes the challenge to computation • Challenge: Shorter but many reads: billions of them Mapping a fragment to entire reference genome is costly: cost does not reduce vs. a long fragment, and may increase for a shorter fragment More potential mapping locations: harder to search for all possible matches in the reference DNA -- Even harder when mutations are allowed • Requirement: Algorithm that is fast and efficient which can process enormous amount of data • Suffix tree or prefix tree based alignment tools: Newer tools use Burrows-Wheeler transformation -- Bowtie, BWA, SOAPv 2 Advantage -- Fast in finding the exact match without mutations Disadvantages -- Very slow when mutations are allowed -- Not comprehensive: does not search for all possible locations • Hash table based alignment tools: Use hash table for filtering non-matching coordinates -- mr. FAST, mrs. FAST Advantage -- Comprehensive, and fast when comprehensive Disadvantage -- Slower in searching for just the exact match Our Goal and Fast. HASH Our First Observation qq base pair (bp) Mismatch Reference DNA Fragment mr. FAST: Two Key Components • Hash table (HT): Stores coordinates of segments in reference DNA Coordinate list Segments 11 12 229 304 AAAAAAC AAAAAAG AAAAAAT ---- 991 TTTTTTT 1105 303 400012 798 qq 4991 …. 4001451 900321 Compare input fragment against reference DNA Check for mutations: mismatches, insertions and deletions (allow e mutations) Need to compare every base pair very slow Effect of Adjacency Filtering 3 E+10 1 E+09 5 E+07 1 E+08 1 E+07 1 E+06 Original mr. FAST string comparisons String comparisons after AF Number of string matches qq Our Second Observation String comparison Other Adjacency Filtering Time with AF (s) 0 5000 10000 15000 • Problem with mr. FAST: Slow: 5 hours to process 1 M fragments (108 bp) • Our goal: Reduce the execution time while maintaining comprehensiveness q • Fast. HASH Overview: Two key components: Adjacency Filtering: Reject obviously non-matching coordinates at early stage to avoid unnecessary expensive string comparisons Cheap segment selection: Reduce the absolute number of coordinates that are subject to examination • Current Result: 38 x speedup for 1 M fragments compared to mr. FAST 20000 • Adjacency Filtering becomes the bottleneck • We can speed this up by avoiding the probing of long coordinate lists • Observation: Hash table is imbalanced Cheap segments: Segments that have few coordinates in hash table Expensive segments: Segments that have many coordinates lead to slow execution during AF • Idea: Select cheapest segments within a fragment Selecting the cheapest e+1 segments guarantees comprehensiveness (at least one has no error) • Example: If e = 1, select the cheapest 2 segments q AAAAAAACGTAACCTTAAAACCCATTTACC Cheapest • Effect of CSS: The number of coordinates examined First segments Cheapest segments 100% 6. 4% 0 50000 100000 150000 200000 250000 300000 1 1. Divide fragment into segments TTTTTT segments CCCCCC AAAAAA 2. Check HT to get coordinates segments’ 2 303 coordinates in 1105 7712 11 991 Hash Table 444991 reference DNA 900321 (HT) Stores 3. Retrieve reference coordinates DNA strings at the (coord. ) of Reference DNA coordinates 3 segments in Database 4. Compare fragment to reference DNA strings q AAAAAACCCCTCCCTTTTTT AAAAAACCCCCCTTTTCGAT AAAAAATAACAACCCCCCTTTTTT String Compare 10000 15000 20000 Other q • Most string comparisons are useless: result in no match 3 E+10 Number of string comparisons conducted 1 E+10 1 E+09 1 E+08 Number of string matches 1 E+07 m m+12 m AAAAAA n 303 505 ? m+12 CCCCCC n+12 557 1033 ? m+24 TTTTTT n+25 … q 10000 5000 4935 478 Do > e coordinate lists contain consecutive coordinates? coordinate list Preliminary GPU Execution Time of Fast. HASH Intel i 7 2600 / 16 GB DRAM 15000 Reference string m+24 ? m • Input fragment set: Run time (s) Fragment length: 108 base-pairs 600 Fragment size: 1 million 500 Number of errors: 3 mismatches, insertions or deletions 400 Run time (s) Input string q coordinate CPU Execution Time 18369 4 • Observation: If perfect match, consecutive segments should be at consecutive coordinates! • Idea: For a coordinate, check if consecutive coordinates are in the coordinate lists of consecutive segments If yes Do string comparison If no No need for string comparison 1 E+06 20000 String compare: Compare every base pair very slow …AAAAAACCCCCCTTTTTT… String comparison 1 E+11 Reference strings AAAAAACCCCCCTTTTTT Execution time (s) 0 Expensive AAAAAACCCCCCTTTTTT • Goal: Reduce the number of string comparisons • String comparisons take too long 95% of execution time 5000 fragment Adjacency Filtering (AF) Preliminary Results Cheap Segment Selection (CSS) • String comparisons are drastically reduced: 3. 7 x speedup Original time (s) Fast. HASH q 0 Each segment looked up in HT to get coordinate list For each coordinate in the list, look up reference string expensive • String Compare: 1 E+11 q mr. FAST Flow Chart mr. FAST kernel + Adjacency Filtering + Cheap Segment Selection (Fast. HASH) • Conclusions Adjacency Filtering provides 3. 7 x speedup Adjacency Filtering + Cheap Segment Selection provides 38 x speedup 478 331 300 200 100 0 Nvidia Tesla C 2070 q GPU CPU • Conclusion GPU provides 1. 44 x speedup (early result) • Ongoing work Schedule work better on GPU for higher speedup