Adaptive Parallelism for Web Search Myeongjae Jeon Rice

Adaptive Parallelism for Web Search Myeongjae Jeon Rice University In collaboration with Yuxiong He (MSR), Sameh Elnikety (MSR), Alan L. Cox (Rice), and Scott Rixner (Rice) 1

Performance of Web Search 1) Query response time – Answer quickly to users – Reduce both mean and high-percentile latency 2) Response quality (relevance) – Provide highly relevant web pages – Improve with resources and time consumed 2

How Microsoft Bing Works Aggregator Query - All web pages are partitioned across index servers - Distributed query processing (embarrassingly parallel) - Aggregate top K relevant pages Top K pages Top K pages Index Server Index Server Partition Partition All web pages Partition 3

Our Work Query 1 Query 2 Query 3 • Multicore index server – Multiple queries executed concurrently – Each query executed sequentially • Optimization opportunity – Low CPU utilization – Many idle cores • Contributions Index Server – Parallelize a single query – Reduce response times 4

Outline 1. Query parallelism – Run a query with multiple threads 2. Adaptive parallelism – Select degree of parallelism per query 3. Evaluation 5

Query Processing and Early Termination Highest Docs sorted by static rank Doc 1 Doc 2 Doc 3 ……. Lowest Doc N-2 Doc N-1 Doc N Web documents ……. Inverted index “Euro. Sys” Processing Not evaluated • Processing “Not evaluated” part is useless – Unlikely to contribute to top K relevant results 6

Assigning Data to Threads • Purpose: data processing similar to sequential execution Sorted web documents Highest rank T 1 Lowest rank T 2 T 1 T 2 1 2 3 4 5 6 7 8 • Key approach: dynamic alternating on small chunks 7

Share Thread Info Using Global Heap Global heap (global top K) Sync Thread 1 (local top k) Thread 2 (local top k) • Share information to reduce wasted execution • Use a global heap and update asynchronously • Small sync overhead by batched updates 8

Outline 1. Query parallelism – Run a query with multiple threads 2. Adaptive parallelism – Select degree of parallelism per query 3. Evaluation 9

Key Ideas • System load – Parallelize a query at light load – Execute a query sequentially at heavy load • Speedup – Parallelize a query more aggressively if parallelism shows a better speedup 10

No Load Heavy Load Query 1 Query 4 Query 2 Query 5 Query 3 Query 6 11

1 2 3 4 5 6 # Threads Query 1 2 3 4 5 6 # Threads 60 T 1 50 40 30 20 10 T 06 Speedup 6 5 4 3 2 1 Linear Speedup Exec. time 60 T 1 50 40 30 20 10 0 Speedup Exec. time No Speedup 1 2 3 4 5 6 # Threads 6 5 4 3 2 1 1 2 3 4 5 6 # Threads Query 12

Speedup in Reality 90 80 70 60 50 40 30 20 10 0 Speedup Exec. Time (ms) • Mostly neither no speedup nor linear speedup 1 2 3 4 5 # Threads 6 6 5 4 3 2 1 Linear speedup Bing No speedup 1 2 3 4 5 # Threads 6 13

Adaptive Algorithm • Decide parallelism degree at runtime – Pick a degree (p) that minimizes response time min(Tp + K Tp p / N) My execution time Latency impact on waiting queries K: system load (queue length) Tp: Execution time with parallelism degree p 14

Experimental Setup • Machine setup – Two 6 -core Xeon processors (2. 27 GHz) – 32 GB memory • 22 GB dedicated to caching – 90 GB web index in SSD • Experimental system – Index server – Client • Replay obtained queries • Poisson distribution • Varying arrival rate (query per second) • Workload – 100 K Bing user queries 15

Mean Response Time (ms) Mean Response Time 100 90 80 70 60 50 40 30 - Fixed Parallelism - Degree = 1 Degree = 2 Degree = 3 Degree = 6 10 20 30 40 50 60 70 System Load (QPS) 80 90 100 • No fixed degree of parallelism performs well for all loads 16

Mean Response Time (ms) Mean Response Time 100 90 80 70 60 50 40 30 - Adaptive Parallelism - Degree = 1 Degree = 2 Degree = 3 Degree = 6 Adaptive 10 20 30 40 50 60 70 System Load (QPS) 80 90 100 • Lower than all other fixed degrees 17

100 90 80 70 60 50 40 30 - Adaptive Parallelism - Degree = 1 Degree = 2 Degree = 3 Degree = 6 Adaptive 10 20 30 60 Fraction (%) Mean Response Time (ms) Mean Response Time 40 50 40, 88 40 30 20 0 70 80 90 100 System Loa 51, 04 50 10 60 0, 53 1 7, 09 2 0, 33 0, 14 3 4 5 Parallelism Degree 6 - Select any degree among all possible options - Parallelism degrees are utilized unevenly to produce the best performance 18

Mean Response Time (ms) Mean Response Time 100 90 80 70 60 50 40 30 - Adaptive Parallelism 47% Degree = 1 Adaptive 10 20 30 40 50 60 70 System Load (QPS) Interesting range 80 90 100 • Much lower than sequential execution 19

95% Response Time (ms) 95 th-Percentile Response Time 400 350 300 52% 250 Degree = 1 Degree = 2 Degree = 3 Degree = 6 200 150 10 20 30 40 50 60 70 System Load (QPS) 80 90 100 • Similar improvements in 99% response time 20

More Experimental Results • Quality of responses – Equivalent to sequential execution • Importance of speedup – Compare to system load only – Using both system load and speedup provides up to 17% better for the average 21

Conclusion 1. Effectively parallelize search server – Parallelize a single query with multiple threads – Adaptively select parallelism degree per query 2. We improve response time by: – 47% on average, 52% for the 95 th-percentile – Same quality of responses 3. In Bing next week! 22