Apache Performance Tuning Part 2 Scaling Out Sander

Apache Performance Tuning Part 2: Scaling Out Sander Temme <sander@temme. net>

Who am I? l Member, Apache Software Foundation l l l Infrastructure Team Member Contractor l l Lurker on dev@httpd Walt Disney Imagineering ADP University of California Currently unemployed…

Agenda l l l Introduction Redundancy in Hardware Building Out: Separate Tiers Building Out: Load Balancing Putting it All Together Conclusion

Introduction l Why do This? l l Scalability (Oh my gosh, I’m so popular!) Single Server won’t cut it Additional capacity without redesign Why NOT do This? l l l It costs money It costs time Application design less straightforward

Scaling vs. High Availability l Scalability l l High Availability l l Increase serving capacity Increase uptime Not the same thing

Redundancy in Hardware l Moving Parts Break l l l Hard Disks Power Supplies Hard Disks Fans Hard Disks Buy High Quality Disks l l l Refurbished, OEM, Brand Name Which has longer warranty? Which is more reliable?

Server Configuration l Mirror those Disks l l Install the RAID utility Have it warn you RAID is no good if you don’t learn of failures! Redundant Power Supplies l On different circuits

Scaling Vertically

Scaling Vertically l l Move Services to Other Hosts Pros: l l Less resource contention Specialized hardware Scale out tiers individually Cons: l l Development/Deployment harder More hosts to manage

Moving Services l l l Most communications over TCP/IP anyway Connect to other box instead of 127. 0. 0. 1 Separate network traffic across interfaces l l Most boxes come with two Reduce network resources

Specialized Hardware l Web Servers l l l Application Server(s) l l l Fairly lightweight machines Several of ’em Sized to fit Application Platform Probably lots of RAM, CPU Database Server l l Lots of RAM Fast disks in RAID-5

Scaling Horizontally

Load Balancing Schemes l l DNS Tricks Peer Server Pools l l Network Load Balancing (Win 2 k 3) Load Balancing Appliance l l Box from F 5, Juniper, Cisco, Foundry, … Linux Virtual Server

DNS Round-Robin l l l Easy! Multiple A Records in DNS Zone File Not Smart: l l l DNS Lookups are cached Load on Server Outage

Example Zone File scalingout. org. 86400 2006051401 86400 7200 86400 ) scalingout. org. gw bagheera IN SOA ns. scalingout. org. sctemme. scalingout. org. ( ; Serial ; refresh (1 day) ; retry (2 hours) ; expire (10 days) ; minimum (1 day) IN NS bagheera. scalingout. org. IN IN A A 10. 11. 0. 1 10. 11. 0. 2 mail ns IN IN CNAME bagheera www IN IN IN A A A 10. 113 10. 114 10. 115 ; . . .

Peer-based: NLB l l Windows 2000 Server Enterprise Ed. , Windows Server 2003 Up to 32 hosts in cluster All hosts assume cluster IP, MAC NLB makes LB decision l l Only one host gets to answer TCP handshake Should be application independent

Load Balancing Device Client Internet

Load Balancing l l l One Load Balancer Many Web Servers Choice of Balancing Schemes l l Reliability l l Round-robin, Least Used, … Heartbeats, unavailable servers don’t receive requests Feature War

Load Balancer Types l Layer 2 l l Rewrite IP packet headers Limited or no content inspection No application level affinity Layer 4 l l l Proxy at HTTP level Application level affinity IP, Port, HTTP rewriting

Example: Linux Virtual Server l l l Free, Open Source, etc. IP Virtual Server module in kernel Lots of auxiliary modules l l l Like a box of Legos May come with Your Distribution Do It Yourself http: //www. linuxvirtualserver. org/

Example: mod_proxy_balancer l l l New in Apache HTTP Server 2. 2 Part of mod_proxy Two Load Balancing Methods l l l By number of requests By number of bytes Detects failed backends

Apache Configuration Listen 80 Log. Level debug Transfer. Log logs/access_log Load. Module proxy_modules/mod_proxy. so Load. Module proxy_http_modules/mod_proxy_http. so Load. Module proxy_balancer_modules/mod_proxy_balancer. so Proxy. Pass / balancer: //mycluster/ Proxy. Pass. Reverse / http: //1. 2. 3. 4: 80 Proxy. Pass. Reverse / http: //1. 2. 3. 5: 80 <Proxy balancer: //mycluster> Balancer. Member http: //1. 2. 3. 4: 80 Balancer. Member http: //1. 2. 3. 5: 80 </Proxy>

Example: Tomcat and mod_jk l l l Apache + mod_jk Multiple Tomcat servers Balancer Worker

Apache Configuration Load. Module jk_module /Volumes/Files/asf/httpd-r 415210 w/modules/mod_jk. so JKMount /servlets-examples/* loadbalancer JKMount /*. jsp loadbalancer Jk. Mount /jkmanager/* jkstatus JKLog. File logs/jk_log JKLog. Level debug JKWorker. Property worker. list=loadbalancer, jkstatus JKWorker. Property worker. tc 1. port=15109 JKWorker. Property worker. tc 1. host=localhost JKWorker. Property worker. tc 1. type=ajp 13 JKWorker. Property worker. tc 1. lbfactor=1 JKWorker. Property worker. tc 2. port=15209 JKWorker. Property worker. tc 2. host=localhost JKWorker. Property worker. tc 2. type=ajp 13 JKWorker. Property worker. tc 2. lbfactor=1 JKWorker. Property worker. tc 3. port=15309 JKWorker. Property worker. tc 3. host=localhost JKWorker. Property worker. tc 3. type=ajp 13 JKWorker. Property worker. tc 3. lbfactor=1 JKWorker. Property worker. loadbalancer. type=lb JKWorker. Property worker. loadbalancer. balance_workers=tc 1, tc 2, tc 3 JKWorker. Property worker. jkstatus. type=status

Tomcat Configuration l l Put same content on all Tomcats Edit conf/server. xml: <Engine name="Catalina" default. Host="localhost" jvm. Route="tc 1"> l jvm. Route must match jk worker name!

Tomcat Session Replication l l Share Http. Session objects across instances One instance dies, session lives on Apache will route requests to other instance Uses IP Multicast

Session Replication Configuration l l Uncomment <Cluster> element in server. xml Put empty <distributable /> element in <webapp> element in web. xml

Load Balanced PHP l l l Shared-nothing Architecture All shared data in database Unless you use Sessions l Stored in file on disk

Tuning the Database Tier l l Not my area (sorry) Give Money to Oracle Consultants l l (or My. SQL) (or …) Tip: Separate Read and Write Operations l l Replicate from Write db to Read db data slightly stale l Does it matter?

Putting it All Together Read-only Internet Client Read-write

Monitoring the Farm l l l Your stuff will break The sooner you know… Install monitoring software l l Check status of servers and services E-mail or pages when non-responsive

Nagios l l l Host and Service Monitoring Free In use at ASF http: //www. nagios. org/

Performance Monitoring l l l Monitor resource utilization Check for bottlenecks Extrapolate for Capacity Planning Multi Router Traffic Grapher Ganglia http: //www. mrtg. org/ http: //ganglia. sourceforge. net/

MRTG Graph

Q&A

Conference Roadmap l l l l T 01 Scalable Internet Architectures T 12 Open. Solaris Zones: Web server and application consolidation TU 8 PHP and My. SQL Best Practices WE 26 The Ins and Outs of Clustered Logging TH 5 The Open Management Consortium and You TH 27 Configuring Apache Derby for Performance and Durability FR 22 Clustering Apache Geronimo

Current Version http: //people. apache. org/~sctemme/Apcon. US 2006/WE 23/

Thank You