Copysets Reducing the Frequency of Data Loss in
- Slides: 37
Copysets: Reducing the Frequency of Data Loss in Cloud Storage Asaf Cidon, Stephen M. Rumble, Ryan Stutsman, Sachin Katti, John Ousterhout and Mendel Rosenblum Stanford University 1
Goal: Tolerate Node Failures Choose random Random replication used by: • HDFS • GFS • Windows Azure • RAMCloud • …
Not All Failures are Independent • Power outages – 1 -2 times a year [Google, Linked. In, Yahoo] • Large scale network failures – 5 -10 times a year [Google, Linked. In] • And more: – Rolling software/hardware upgrades – Power down
Random Replication Fails Under Simultaneous Failures Confirmed by: Facebook, Yahoo, Linked. In
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Copysets: {1, 5, 6}, {2, 6, 8} Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Random Replication Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Copysets: {1, 2, 3}, {1, 2, 4}, {1, 2, 5}, {1, 2, 6}, {1, 2, 7}, {1, 2, 8}, … Node 9
Random Replication Causes Frequent Data Loss • Random replication eventually creates maximum number of copysets – Any combination of 3 nodes – = 84 copysets • If 3 nodes fail, 100% probability of data loss –
Min. Copysets Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
Min. Copysets Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Copysets: {1, 5, 7}, {2, 4, 9}, {3, 6, 8} Node 7 Node 8 Node 9
Min. Copysets Minimizes Data Loss Frequency • Min. Copysets creates minimum number of copysets – Only {1, 5, 7}, {2, 4, 9}, {3, 6, 8} • If 3 nodes fail, 3. 5% of data loss –
Min. Copysets Reduces Probability of Data Loss
The Trade-off Min. Copysets Random Replication Mean Time to Failure 625 years 1 year Amount of Data Lost 1 TB 5. 5 GB • 5000 -node cluster • Power outage occurs every year Confirmed by: Facebook, Linked. In, Net. App, Google
Problem: Min. Copysets Increases Single Node Recovery Time 800 Recovery Time (seconds) 700 600 500 400 300 200 100 0 Random Replication Min. Copysets Time to Recovery a 100 GB Node in 39 -node HDFS cluster
Facebook Extension to HDFS Choose random Buddy Group
A Compromise 800 Recovery Time (seconds) 700 600 500 400 300 200 100 0 HDFS Random Facebook Extension to Min. Copysets Replication HDFS Time to Recovery a 100 GB Node in 39 -node HDFS cluster
Can We Do Better?
Definition: Scatter Width Min. Copysets Scatter Width = 2 Facebook Extension to HDFS Scatter Width = 10
Facebook Extension to HDFS 1 2 3 4 5 6 7 8 9 Buddy group • Node 1’s copysets: – {1, 2, 3}, {1, 2, 4}, {1, 2, 5}, {1, 3, 4}, {1, 3, 5}, {1, 4, 5} • Overall: 54 copysets • If 3 nodes fail simultaneously: •
Copyset Replication: Intuition • Same scatter width (4), different scheme: {1, 2, 3}, {4, 5, 6}, {7, 8, 9} {1, 4, 7}, {2, 5, 8}, {3, 6, 9} 1 Ingredients of ideal scheme 1. Maximize scatter width 2. Minimize overlaps • 2 3 4 7
Copyset Replication: Initialization 1 2 3 4 5 6 7 8 9 1 8 4 Random Permutation 7 3 5 6 2 9 Split into copysets (Scatter width = 2) 7 3 Copyset 5 6 2 Copyset 9 1 8 Copyset 4
Copyset Replication: Initialization 1 2 3 4 5 6 7 8 9 Permutation 1: Scatter width = 2 7 3 5 6 2 9 1 8 4 2 3 Permutation 2: Scatter width = 4 9 7 1 5 6 8 4 … Permutation 5: Scatter width = 10
Copyset Replication: Replication 1 2 3 4 5 6 7 8 9 Randomly choose copyset 7 3 5 6 2 9 1 8 4 9 7 1 5 6 8 4 2 3 …
Insignificant Overhead 800 Recovery Time (seconds) 700 600 500 400 300 200 100 0 HDFS Facebook Copyset Min. Copysets Extension to HDFS Replication Time to Recovery a 100 GB Node in 39 -node HDFS cluster
Copyset Replication
Inherent Trade-off
Related Work • BIBD (Balanced Incomplete Block Designs) – Originally proposed for designing agricultural experiments in the 1930’s! [Fisher, ’ 40] • Other applications – Power downs [Harnik et al ’ 09, Leverich et al ’ 10, Thereska ’ 11] – Multi-fabric interconnects [Mehra, ’ 99]
Summary 1. Many storage systems randomly spray their data across a large number of nodes 2. Serious problem with correlated failures 3. Copyset Replication is a better way of spraying data that decreases the probability of correlated failures
Thank You! Stanford University 34
More Failures (Facebook)
RAMCloud
HDFS
- Bials test
- Reducing and non reducing end of glycogen
- Carbohydrates
- Reducing vs non reducing sugar
- Normal loss treatment in process costing
- Joint frequency
- How to calculate relative frequency
- Constant phase difference
- Vmax = aw
- Relative frequency bar chart
- Joint frequency vs marginal frequency
- Marginal relative frequency
- Data loss prevetion
- Data loss
- Data loss
- Hát kết hợp bộ gõ cơ thể
- Ng-html
- Bổ thể
- Tỉ lệ cơ thể trẻ em
- Gấu đi như thế nào
- Glasgow thang điểm
- Hát lên người ơi alleluia
- Các môn thể thao bắt đầu bằng từ đua
- Thế nào là hệ số cao nhất
- Các châu lục và đại dương trên thế giới
- Công thức tính độ biến thiên đông lượng
- Trời xanh đây là của chúng ta thể thơ
- Mật thư anh em như thể tay chân
- Phép trừ bù
- Phản ứng thế ankan
- Các châu lục và đại dương trên thế giới
- Thể thơ truyền thống
- Quá trình desamine hóa có thể tạo ra
- Một số thể thơ truyền thống
- Cái miệng nó xinh thế
- Vẽ hình chiếu vuông góc của vật thể sau
- Biện pháp chống mỏi cơ
- đặc điểm cơ thể của người tối cổ