HDFSGFS Outline Requirements for a Distributed File System

HDFS/GFS

Outline • Requirements for a Distributed File System • HDFS – Architecture – Read/Write • Research Directions – Popularity – Failures – Network

Properties of a Data Center • Servers are built from commodity devices – Failure is extremely common – Servers only have a limited amount of HDD space • Network is over-subscribed – The bandwidth between servers is different • Demanding applications – High throughput, low latency • Resources are grouped into failure-zone – Independent units of failure

Data-Center Architecture 10 GB 25 GB 100 GB

Properties of a Data Center • Servers are built from commodity devices – Failure is extremely common – Servers only have a limited amount of HDD space • Network is over-subscribed – The bandwidth between servers is different • Demanding applications – High throughput, low latency • Resources are grouped into failure-zone – Independent units of failure

Data-Center Architecture Failure Domain 1 Failure Domain 2

Goals for a Data Center File System • Reliable – Over come server failures • High performing – Provide good performance to application • Aware of network disparities – Make data local to the applications

Common Design Principles • For performance: Partitioning the data – Split data into chunks and distribute • provides high throughput – Many people can read the chunks in parallel • Better than everyone reading the same file How data is partitioned across nodes • For reliability: Replication: – overcome failure by making copies • At least one copy should be online How data is duplicated across nodes • For Network-disparity: rack-aware allocation – Read from the closest block – Write to the closest location

Outline • Requirements for a Distributed File System • HDFS – Architecture – Read/Write • Research Directions – Popularity – Failures – Network

HDFS Architecture • Name Node – Master (only 1 in a data center) – All reads/write go through the master – Manages the data nodes • • • Detects failures – triggers replication • Tracks performance – Tracks location of blocks • Data Node – – One per server – Stores the blocks • • Tracks status of blocks Ensures integrity of block • Name Node Data. B Node B` • Tracks block to node mapping Tracks status of data nodes Rebalances the data center Orchestrates read/writes Data B Node B` Data Node

What is a Distributed FS Write? • HDFS – For high-performance • Make N copies of the data to be written • Default N= 3 B Write B HDFS Master B B

What is a Distributed FS Write? • HDFS – For Fault tolerance • Place in two different fault domains • 2 copies in the same rack • 1 in a different rack B Zone 1 B B Zone 2

What is a Distributed FS Write? • HDFS – For Network awareness • Currently does nothing Picks two random racks

What is a Distributed FS Read? • HDFS – For Network awareness/performance • Pick closest copy to read from. – Nothing specific for Reliability Read B Name Node B Zone 1 B B Zone 2

Implications of Read/Write Semantics • One application write == 3 HDFS writes – Writes are costly!! – HDFS is optimized for write-once/read-many times workloads • What is an update/edit? Rewrite blocks? Modify B Name Node B Zone 1 B B Zone 2

Implications of Read/Write Semantics • One application write == 3 HDFS writes – Writes are costly!! – HDFS is optimized for write-once/read-many times workloads • An update/Edit: – delete old data + write new data Modify B Name Node B B` B` B

Interesting Challenges • How happens with more popular blocks? – Or less popular blocks? • What happens during server failures? – Can you loose data? • What happens if you have a better network? – No oversubscription

Outline • Requirements for a Distributed File System • HDFS – Architecture – Read/Write • Research Directions – Popularity – Failures – Network

Popularity in HDFS • Not all files are equivalent – E. g. More people search for bball than hockey • More popular blocks will have more contention – Leads to slower performance – Search for bball will be slower

Popularity in HDFS • # of copies of a block = function(popularity) – If 50 people search for bball, then make 50 blocks – If only 3 search for hockey, then make 3 • You want as many copies of a block as readers

Popularity in HDFS • As data becomes old less people care about it – So last year’s weather versus today’s weather • When a block becomes old (older than a week) – Reduce the number of copies. – In Facebook data centers, only one copy of old data

Failures in Data Center • Do servers fail? ? – Facebook: 1% of servers fail after-reboot – Google: at least one server fails a day Data. B B` Node Data Node B` Data BNode Data B Node B` Name Node • Failed node doesn’t send heart beat • Name node determines blocks on failed node • Starts replication.

Failures in Data Center • Do servers fail? ? – Facebook: 1% of servers fail after-reboot – Google: at least one server fails a day Name Node Data. B B` Node Data. B Node B` Data Node • Failed node doesn’t send heart beat • Name node determines blocks on failed node • Starts replication.

Problems With Locality aware DFS • Ignores contention on the servers – I/O contention greatly impacts performance

Problems With Locality aware DFS • Ignores contention on the servers – I/O contention greatly impacts performance • Ignores contention in the network – Similar performance degradation

Types of Network Topologies • Current Networks • Future Networks – Uneven B/W everywhere – Even B/W everywhere 10 GB 100 GB 25 GB 100 GB

Implications of Network Topologies • Blocks can be more spread out! – No need for two blocks within the same rack – Same BW everywhere so no need for locality aware placement

Summary • Properties for a DFS • Research Challenges – Popularity – Failure – Data Placement

Un-discussed • Cluster rebalancing – Move blocks around based on utilization. • Data integrity – Use checksum to check if data has gotten corrupted. • Staging + pipeline