Usenet Training May 4 7 2004 Orlando FL

  • Slides: 19
Download presentation
Usenet Training May 4 -7, 2004 Orlando, FL At Disney’s Coronado Spring Resort

Usenet Training May 4 -7, 2004 Orlando, FL At Disney’s Coronado Spring Resort

Day Two Building an Enterprise Usenet Environment

Day Two Building an Enterprise Usenet Environment

Questions to ask • How many concurrent NNTP reading sessions? – – – •

Questions to ask • How many concurrent NNTP reading sessions? – – – • How much retention? – – • 97 -99% size from binaries Full feed is 1. 1 TB/day - doubles every 9 months Redundancy desired – – – • • • What content? Full feed, internal discussion, partial feed Binaries/Text – – • I don't know What is the size of the user base? 5 -10% or more may access news Full data replication Disaster recovery Failover Feeding architecture to provide/receive Authentication and classes of service Performance – Honda, Audi, Porsche

Sample Architecture Criteria • • • 8, 000 Concurrent Connections 14 days binary, 180

Sample Architecture Criteria • • • 8, 000 Concurrent Connections 14 days binary, 180 days text Two data centers Full redundancy, outage of one DC can support 4000 connections. Sell 10 full feeds, peer with 50 providers Authentication by IP address

Calculations • Note about calculations – Change over time – Not hard and fast

Calculations • Note about calculations – Change over time – Not hard and fast – Change with experience and new architectures

Storage • • 14 days Binary and 180 days text 17 TB x 2

Storage • • 14 days Binary and 180 days text 17 TB x 2 with full redundancy Adaptive spooling

Machine Class Choices • Problems we're solving – Totally random I/O – Significant network

Machine Class Choices • Problems we're solving – Totally random I/O – Significant network volume – We're not NASA – No Complex Number Theory modelling here – WOPR is misplaced – Disk mechanics – Slowest evolving technology – Physical limitations • Cost – Administration – Deployment

Machine Classes • • • 2000 connections per host - 4 hosts 1000 connections

Machine Classes • • • 2000 connections per host - 4 hosts 1000 connections per host - 8 hosts 400 connections per host - 20 hosts • Machine Class dictates OS or vice versa • Summary – We're looking for an efficient I/O mover – Many paths, independent busses – Carl Lewis, not Andre the Giant.

Architecture Diagramming • Start with Tornado Front Ends – At least 100 GB storage

Architecture Diagramming • Start with Tornado Front Ends – At least 100 GB storage • At least 1 "fast" spindle per 400 connections • At least 2 "slow" spindles per 300 connections • • Sun 280 R's, 2 CPU, 4 GB RAM, Sun 3510 attached array (shared) Module to support 2000 connections

Tornado Back Ends • • • 4000 connections accessing articles 1 BE with one

Tornado Back Ends • • • 4000 connections accessing articles 1 BE with one array can be enough BE's can be smaller than an FE – – Direct spool sharing only Not quite a Commodore 64… but close

Tornado Back Ends • Storage Array – – • FCAL Switching – – •

Tornado Back Ends • Storage Array – – • FCAL Switching – – • Very rough estimate 1 drive per 75 connections. Optimize for Random I/O Benchmark Precautions • Usually sequential I/O, push for random access data. • Plan on performance 10 -20% of spec benchmark or even less. 48 x 72 GB 10, 000 RPM drives (3456 GB) Writes – approx 120 mbps ~ 13 MB/sec Reads • 4000 connections – Average? 1 Mbps – Many “constant downloaders” –. 11 MB/sec X 4000 = 450 MB/sec – 450 MB/sec x ~6 (truly random adjustment) – 2. 6 GB/sec Switching Spec » Start with one 2 GB/sec switch with plans to add another. We only have 3 TB and we need 17? !

Enter Storm Cellar • • • 17 TB total, 3 TB deployed - need

Enter Storm Cellar • • • 17 TB total, 3 TB deployed - need 14 TB 3 Storm cellars with 6 TB usable Split cascade feed from Tornado Back Ends 4 U NAS Appliance 24 x 300 GB drives (7. 03 TB / ATA Drives) Linux 2. 6. 2+

Feeding • • 10 Full feeds @ 120 mbps ~ 3 -5 feeds per

Feeding • • 10 Full feeds @ 120 mbps ~ 3 -5 feeds per box 2 Peering Cyclones per site 1 Master Cyclone 1 Hurricane

Data Center Architecture

Data Center Architecture

Full Architecture

Full Architecture

What could we do with Linux? • Features we can use – FE adaptive

What could we do with Linux? • Features we can use – FE adaptive caching – Cyclone Split Feeding – Newest Linux kernels • FCAL sharing probably not feasible – IP spool sharing with adaptive caching – Hardware choices • 1 Day’s worth of disk on each FE for cache • BE's have 4+ TB each

Linux Architecture • • • Local spool cache on Tornado FE’s serves recent articles

Linux Architecture • • • Local spool cache on Tornado FE’s serves recent articles that are actually read (8 copies – 16 for 2 Data Centers) Spools are aggregated by all FE’s Back Ends contain only partial data – spreading I/O over many BE’s.

Quiz 1. 2. Trace the path of locally posted articles. Trace the path of

Quiz 1. 2. Trace the path of locally posted articles. Trace the path of articles received from peers 3. Where and how are articles numbered? 4. If a Front End is down, what happens? 5. If a Back End is down what happens? 6. If the storage array is out, what happens? 7. What do we do if the Master Cyclone goes down? 8. What do we do if the Hurricane goes down? 9. How do we add a feed from another data center to only fill in articles we’re missing? 10. How do we add a feed to only turn on if our Master Cyclone goes down? 11. Who should we peer with?

Questions

Questions