Scalable Internet Services Cluster Lessons and Architecture Design

Scalable Internet Services Cluster Lessons and Architecture Design for Scalable Services BR 01, TACC, Porcupine, SEDA and Capriccio Ben Y. Zhao ravenben@cs. ucsb. edu

Outline • Overview of cluster services • lessons from giant-scale services (BR 01) • SEDA (staged event-driven architecture) • Capriccio Ben Y. Zhao ravenben@cs. ucsb. edu

Scalable Servers • Clustered services • natural platform for large web services • search engines, DB servers, transactional servers • Key benefit • low cost of computing, COTS vs. SMP • incremental scalability • load balance traffic/requests across servers • Extension from single server model • reliable/fast communication, but partitioned data Ben Y. Zhao ravenben@cs. ucsb. edu

Goals • Failure transparency • hot-swapping components w/o loss of avail • homogeneous functionality and/or replication • Load balancing • partition data / requests for max service rate • need to colocate requests w/ associated data • Scalability • aggregate performance should scale w/# of servers Ben Y. Zhao ravenben@cs. ucsb. edu

Two Different Models • Read-mostly data • web servers, DB servers, search engines (query) • replicate across servers + (RR DNS / redirector) client client IP Network (WAN) Round Robin DNS Ben Y. Zhao ravenben@cs. ucsb. edu

Two Different Models … • Read-write model • mail servers, e-commerce sites, hosted services • small(er) replication factor for stronger consistency client client IP Network (WAN) Load Redirector Ben Y. Zhao ravenben@cs. ucsb. edu

Key Architecture Challenges • Providing high availability • availability across component failures • Handling flash crowds / peak load • need support for massive concurrency • Other challenges • upgradability: maintaining availability and minimal cost during upgrades in S/W, H/W, functionality • error diagnosis: fast isolation of failures / performance degradation Ben Y. Zhao ravenben@cs. ucsb. edu

Nuggets • Definition • uptime = (MTBF – MTTR)/MTBF • yield = queries completed / queries offered • harvest = data available / complete data • MTTR • at least as important at MTBF • much easier to tune and quantify • DQ principle • data/query x queries/second constant • physical bottlenecks limit overall throughput Ben Y. Zhao ravenben@cs. ucsb. edu

Staged Event-driven Architecture • SEDA (SOSP’ 05) Ben Y. Zhao ravenben@cs. ucsb. edu

Break… • Come back in 5 mins • more on threads vs. events… Ben Y. Zhao ravenben@cs. ucsb. edu

Tapestry Software Architecture applications application programming interface Dynamic Tap. core router distance map network Patchwork SEDA event-driven framework Java Virtual Machine Ben Y. Zhao ravenben@cs. ucsb. edu

Impact of Correlated Events + ? ? ? A B C + = event handler Network ? • • • ? ? ? web / application servers independent requests maximize individual throughput Ben Y. Zhao • • correlated requests: A+B+C D e. g. online continuous queries, sensor aggregation, p 2 p control layer, streaming data mining ravenben@cs. ucsb. edu

Capriccio • User-level light-weight threads (SOSP 03) • Argument • threads are the natural programming model • current problems result of implementation • not fundamental flaw • Approach • aim for massive scalability • compiler assistance • linked stacks, block graph scheduling Ben Y. Zhao ravenben@cs. ucsb. edu

The Price of Concurrency • Why is concurrency hard? • Race conditions • Code complexity • Scalability (no O(n) • Scheduling & resource sensitivity • Inevitable overload • Performance vs. Ease of Programming operations) Ideal Threads Events Threads Performance Programmability • No good solution Ben Y. Zhao ravenben@cs. ucsb. edu

The Answer: Better Threads • Goals • Simple programming model • Good tools & infrastructure • Languages, compilers, debuggers, etc. • Good performance • Claims • Threads are preferable to events • User-Level threads are key Ben Y. Zhao ravenben@cs. ucsb. edu

“But Events Are Better!” • Recent arguments for events • Lower runtime overhead • Better live state management • Inexpensive synchronization • More flexible control flow • Better scheduling and locality • All true but… • Lauer & Needham duality argument • Criticisms of specific threads packages • No inherent problem with threads! Ben Y. Zhao ravenben@cs. ucsb. edu

Criticism: Runtime Overhead • Criticism: Threads don’t perform well for high concurrency • Response • Avoid O(n) operations • Minimize context switch overhead • Simple scalability test • Slightly modified GNU Pth • Thread-per-task vs. single thread • Same performance! Ben Y. Zhao ravenben@cs. ucsb. edu

Criticism: Synchronization • Criticism: Thread synchronization is heavyweight • Response • Cooperative multitasking works for threads, too! • Also presents same problems • • • Starvation & fairness Multiprocessors Unexpected blocking (page faults, etc. ) • Both regimes need help • • Ben Y. Zhao Compiler / language support for concurrency Better OS primitives ravenben@cs. ucsb. edu

Criticism: Scheduling • Criticism: Thread schedulers are too generic Task • Can’t use application-specific information • 2 D scheduling: task & program location • Threads schedule based on task only • Events schedule by location (e. g. SEDA) • Allows batching • Allows prediction for SRCT • Threads can use 2 D, too! • Runtime system tracks current location • Call graph allows prediction Ben Y. Zhao Program Location • Response Events Threads ravenben@cs. ucsb. edu

The Proof’s in the Pudding • User-level threads package • Subset of pthreads • Intercept blocking system calls 900 • No O(n) operations 800 Knot. A (Favor Accept) 700 • Support > 100 K threads • Simple web server: Knot • 700 lines of C code • Similar performance • Linear increase, then steady • Drop-off due to poll() overhead Ben Y. Zhao Haboob 600 Mbits / second • 5000 lines of C code Knot. C (Favor Connections) 500 400 300 200 100 0 1 4 16 64 256 1024 4096 16384 Concurrent Clients ravenben@cs. ucsb. edu

Arguments For Threads • More natural programming model • Control flow is more apparent • Exception handling is easier • State management is automatic • Better fit with current tools & hardware • Better existing infrastructure Ben Y. Zhao ravenben@cs. ucsb. edu

Why Threads: control Flow • Events obscure control flow • For programmers and tools Threads thread_main(int sock) { Events Accept. Handler(event e) { struct session s; struct session *s = new_session(e); accept_conn(sock, &s); Request. Handler. enqueue(s); read_request(&s); } pin_cache(&s); Request. Handler(struct session *s) { write_response(&s); unpin(&s); } } pin(s); pin_cache(struct session *s) { } Read Request Pin Cache if( !in_cache(s) ) Read. File. Handler. enqueue(s); pin(&s); read_file(&s); Accept Conn. …; Cache. Handler. enqueue(s); Cache. Handler(struct session *s) { if( !in_cache(&s) ) Web Server else Response. Handler. enqueue(s); } Read File Write Response . . . Exit. Handler(struct session *s) { …; unpin(&s); free_session(s); } Ben Y. Zhao Exit ravenben@cs. ucsb. edu

Why Threads: control Flow • Events obscure control flow • For programmers and tools Threads thread_main(int sock) { Events Cache. Handler(struct session *s) { struct session s; pin(s); accept_conn(sock, &s); if( !in_cache(s) ) Read. File. Handler. enqueue(s); read_request(&s); else Response. Handler. enqueue(s); pin_cache(&s); } write_response(&s); Request. Handler(struct session *s) { unpin(&s); } …; Cache. Handler. enqueue(s); }. . . pin_cache(struct session *s) { read_file(&s); } …; unpin(&s); free_session(s); } Read Request Pin Cache Read File Write Response Accept. Handler(event e) { struct session *s = new_session(e); Request. Handler. enqueue(s); } Ben Y. Zhao Accept Conn. Exit. Handler(struct session *s) { pin(&s); if( !in_cache(&s) ) Web Server Exit ravenben@cs. ucsb. edu

Why Threads: Exceptions • Exceptions complicate control flow • Harder to understand program flow • Cause bugs in cleanup code Threads Events thread_main(int sock) { Cache. Handler(struct session *s) { struct session s; pin(s); accept_conn(sock, &s); if( !in_cache(s) ) Read. File. Handler. enqueue(s); if( !read_request(&s) ) else return; pin_cache(&s); } } Accept Conn. Read Request. Handler(struct session *s) { write_response(&s); unpin(&s); Response. Handler. enqueue(s); Web Server …; if( error ) return; Cache. Handler. enqueue(s); } Pin Cache . . . Exit. Handler(struct session *s) { pin_cache(struct session *s) { …; unpin(&s); free_session(s); pin(&s); } if( !in_cache(&s) ) Accept. Handler(event e) { read_file(&s); } struct session *s = new_session(e); Request. Handler. enqueue(s); } Ben Y. Zhao Read File Write Response Exit ravenben@cs. ucsb. edu

Why Threads: State Management • • Events require manual state management Hard to know when to free • Use GC or risk bugs Threads thread_main(int sock) { Web Server Events Cache. Handler(struct session *s) { struct session s; pin(s); accept_conn(sock, &s); if( !in_cache(s) ) Read. File. Handler. enqueue(s); if( !read_request(&s) ) else return; pin_cache(&s); } Read Request } Request. Handler(struct session *s) { write_response(&s); unpin(&s); Response. Handler. enqueue(s); Accept Conn. …; if( error ) return; Cache. Handler. enqueue(s); } Pin Cache . . . Exit. Handler(struct session *s) { pin_cache(struct session *s) { …; unpin(&s); free_session(s); pin(&s); } if( !in_cache(&s) ) Accept. Handler(event e) { read_file(&s); } struct session *s = new_session(e); Read File Write Response Exit Request. Handler. enqueue(s); } Ben Y. Zhao ravenben@cs. ucsb. edu

Why Threads: Existing Infrastructure • Lots of infrastructure for threads • Debuggers • Languages & compilers • Consequences • More amenable to analysis • Less effort to get working systems Ben Y. Zhao ravenben@cs. ucsb. edu

Building Better Threads • Goals • Simplify the programming model • Thread per concurrent activity • Scalability (100 K+ threads) • Support existing APIs and tools • Automate application-specific customization • Mechanisms • User-level threads • Plumbing: avoid O(n) operations • Compile-time analysis • Run-time analysis Ben Y. Zhao ravenben@cs. ucsb. edu

Case for User-Level Threads • Decouple programming model and OS • Kernel threads • Abstract hardware • Expose device concurrency App User • User-level threads • Provide clean programming model • Expose logical concurrency Threads OS • Benefits of user-level threads • Control over concurrency model! • Independent innovation • Enables static analysis • Enables application-specific tuning Ben Y. Zhao ravenben@cs. ucsb. edu

Case for User-Level Threads • Decouple programming model and OS • Kernel threads • Abstract hardware • Expose device concurrency App • User-level threads • Provide clean programming model • Expose logical concurrency • Benefits of user-level threads • Control over concurrency model! • Independent innovation • Enables static analysis • Enables application-specific tuning Ben Y. Zhao Threads User OS Similar argument to the design of overlay networks ravenben@cs. ucsb. edu

Capriccio Internals • Cooperative user-level threads • Fast context switches • Lightweight synchronization • Kernel Mechanisms • Asynchronous I/O (Linux) • Efficiency • Avoid O(n) operations • Fast, flexible scheduling Ben Y. Zhao ravenben@cs. ucsb. edu

Safety: Linked Stacks • The problem: fixed stacks • Overflow vs. wasted space • Linux. Threads: 2 MB/stack • Limits thread numbers • The solution: linked stacks • Allocate space as needed • Compiler analysis • • Ben Y. Zhao Fixed Stacks overflow waste Linked Stack Add runtime checkpoints Guarantee enough space until next check ravenben@cs. ucsb. edu

Linked Stacks: Algorithm • Parameters • Max. Path • Min. Chunk • Steps • Break cycles • Trace back • 3 2 5 2 4 chkpts limit Max. Path length • Special Cases • Function pointers • External calls • Use large stack Ben Y. Zhao 3 3 6 Max. Path = 8 ravenben@cs. ucsb. edu

Special Cases • Function pointers • categorize f* by # and type of arguments • “guess” which func will/can be called • External functions • users annotate trusted stack bounds on libs • or (re)use a small # of large stack chunks • Result • use/reuse stack chunks much like VM • can efficiently share stack chunks • memory-touch benchmark, factor of 3 reduction in paging cost Ben Y. Zhao ravenben@cs. ucsb. edu

Scheduling: Blocking Graph • Lessons from event systems • Break app into stages • Schedule based on stage priorities • Allows SRCT scheduling, finding Web Server Accept Read Open bottlenecks, etc. Read • Capriccio does this for threads • Deduce stage with stack traces at blocking points • Prioritize based on runtime information Ben Y. Zhao Close Write Close ravenben@cs. ucsb. edu

Resource-Aware Scheduling • Track resources used along BG edges • Memory, file descriptors, CPU • Predict future from the past • Algorithm • Increase use when underutilized • Decrease use near saturation • Advantages • Operate near the knee w/o thrashing • Automatic admission control Ben Y. Zhao Web Server Accept Read Open Read Close Write Close ravenben@cs. ucsb. edu

Pitfalls • What is the max amt of resource? • depends on workload • e. g. : disk thrashing depends on sequential or random seeks • use early signs of thrashing to indicate max capacity • Detecting thrashing • only estimate using “productivity/overhead” • productivity from guessing (threads created, files opened/closed) Ben Y. Zhao ravenben@cs. ucsb. edu

Thread Performance Capriccio-notrace Linux. Threads NPTL Thread Creation 21. 5 37. 5 17. 7 Context Switch 0. 56 0. 24 0. 71 0. 65 Uncontested mutex lock 0. 04 0. 14 0. 15 Time of thread operations (microseconds) • Slightly slower thread creation • Faster context switches • Even with stack traces! • Much faster mutexes Ben Y. Zhao ravenben@cs. ucsb. edu

Runtime Overhead • Tested Apache 2. 0. 44 • Stack linking • 78% slowdown for null call • 3 -4% overall • Resource statistics • 2% (on all the time) • 0. 1% (with sampling) • Stack traces • 8% overhead Ben Y. Zhao ravenben@cs. ucsb. edu

Microbenchmark: Producer / Consumer Ben Y. Zhao ravenben@cs. ucsb. edu

Web Server Performance Ben Y. Zhao ravenben@cs. ucsb. edu

Example of “Great Systems Paper” • observe higher level issue • threads vs. event programming abstraction • use previous work (duality) to identify problem • why are threads not as efficient as events? • good systems design • call graph analysis for linked stacks • resource aware scheduling • good execution • full solid implementation • analysis leading to full understanding of detailed issues • cross-area approach (help from PL research) Ben Y. Zhao ravenben@cs. ucsb. edu

Acknowledgements • Many slides “borrowed” from the respective talks / papers: • Capriccio (Rob von Behren) • SEDA (Matt Welsh) • Brewer 01: “Lessons…” Ben Y. Zhao ravenben@cs. ucsb. edu