An Architecture for Internet Data Transfer Niraj Tolia

An Architecture for Internet Data Transfer Niraj Tolia Michael Kaminsky*, David G. Andersen, and Swapnil Patil Carnegie Mellon University and *Intel Research Pittsburgh

Innovation in Data Transfer is Hard • Imagine: You have a novel data transfer technique • How do you deploy? 1. Update HTTP. Talk to IETF. Modify Apache, IIS, Firefox, Netscape, Opera, IE, Lynx, Wget, … 2. Update SMTP. Talk to IETF. Modify Sendmail, Postfix, Outlook… 3. Give up in frustration 2 Niraj Tolia © May 2006

Barriers to Innovation in Data Transfer • Applications bundle: • Content Negotiation: What data to send – Naming (URLs, directories, …) – Languages – Identification –… • Data Transfer: Getting the bits across • Both are tightly coupled (e. g. , HTTP, SMTP) • Hinders innovation and evolution of new services 3 Niraj Tolia © May 2006

Solution: A Data Transfer Service Sender Application Protocol and Data Xfer Service Receiver Xfer Service Data • Decouple content negotiation from data transfer • Applications perform negotiation as before • But hand data objects to the Transfer Service • The Transfer Service is shared by applications 4 Niraj Tolia © May 2006

Extensible Transfer Architecture Sender Application Protocol Xfer Service USB Keychain Receiver Xfer Service Bittorrent Plugins ü Application-independent cache ü New network features ü Non-networked transfers 5 Local Cache USB Keychain Niraj Tolia © May 2006

Transfer Service Benefits ü Apps. can reuse available transfer techniques • No reimplementation needed ü Easier deployment of new technologies • Applications need no modification ü Provides for cross-application sharing • Can interpose on all data transfers ü Handles transient disconnections 6 Niraj Tolia © May 2006

Outline • • • Motivation Data Oriented Transfer (DOT) service Evaluation Open Issues and Future Work Conclusion 7 Niraj Tolia © May 2006

10, 000 Foot View of Transfers using DOT Request File X Sender Receiver put(X) Xfer Service ? ? read() data Xfer Service • How does the transfer service name data? • How does the transfer service locate data? 8 Niraj Tolia © May 2006

DOT: Object Naming • Application defined names are not portable • Use content-naming for globally unique names • Objects represented by an OID Foo. txt OID Cryptographic Hash File • Objects are further sub-divided into “chunks” Desc 1 Desc 2 Desc 3 File • Each OID corresponds to a list of descriptors • Descriptor lists allow for partial transfers 9 Niraj Tolia © May 2006

DOT: Object Location • Data transfers in DOT are receiver driven • Receiver has better idea of available resources • Senders specify ‘hints’ - potential data locations – dot: //sender. example. com: 12000/ – dht: //opendht. org/ –… 10 Niraj Tolia © May 2006

A Transfer using DOT Request File X OID, Hints Sender put(X) OID, Hints Xfer Service Receiver get(OID, Hints) Transfer Plugins 11 read() data Xfer Service Niraj Tolia © May 2006

DOT’s Modular Architecture Application (1) Application API Transfer Network Plugin DOT (3) Storage Plugin API Storage Plugin (2) Transfer Plugin API Local Storage 12 Niraj Tolia © May 2006

Transfer Plugin API • Simple API • get_descriptor_list( OID, hints ) • get_chunks( descriptor_list, hints ) • cancel_chunks( chunk_list ) • Transfer plugin chaining is easy Transfer Plugin • e. g. , multipath plugin DOT Network Transfer Multi. Path Plugin 13 Transfer Plugin Niraj Tolia © May 2006

Implementation • In C++ using libasync event-driven library • One storage plugin: – In-memory hash tables, disk backed. • Three transfer plugins: • Default Xfer-Xfer plugin • Portable Storage plugin • Multipath plugin • Applications • gcp, an scp-like tool for file transfers • A DOT-enabled Postfix email server – Included a socket-like adapter library 14 Niraj Tolia © May 2006

Current DOT Prototype USB Xfer NET wireless Xfer NET Internet SENDER NET ( DSL ) Multipath cache RECEIVER NET Plugins Xfer ü Application-independent cache ü Multipath and Mirror support ü Non-networked transfers MIRROR 15 Niraj Tolia © May 2006

Outline • • • Motivation Data Oriented Transfer (DOT) service Evaluation Open Issues and Future Work Conclusion 16 Niraj Tolia © May 2006

Evaluation • • Standard file transfer Portable Storage Multi-Path Case Study: Postfix Email Server • Capture and analysis of email trace • Evaluation of DOT-enabled SMTP server • Integration effort 17 Niraj Tolia © May 2006

Standard File Transfer Setup Network Emulator • Two DOT-enabled machines • Network Emulator • Evaluate various b/w + delay combinations • Use gcp for the file transfers • Used 40 MB, 400 KB, 4 KB files • Presenting 40 MB here 18 Niraj Tolia © May 2006

Standard File Transfer • Overhead: hashing, extra RTT • No noticeable overheads with latency 19 Niraj Tolia © May 2006

Portable Storage Experiment 2 Mbit/s • 255 MB transfer over emulated DSL • Based on Virtual Machine transfers at Carnegie Mellon • DOT preemptively copies data onto Flash drive • Wait 5 minutes, plug flash drive into receiver • Two drive speeds • 8 MB/s - 1 GB • 20 MB/s - 2 GB 20 Niraj Tolia © May 2006

Portable Storage Results Device Inserted . . 1126 s (~ 19 min) 21 Niraj Tolia © May 2006

Multipath Plugin: Load Balancing Network Emulator Gigabit Experimental links • Varied capacity + delay of experimental links • Compare fastest link alone with multipath plugin on both links; what speedup? • Transferred 40 MB file • 128 KB socket buffer sizes 22 Niraj Tolia © May 2006

Multipath Plugin is Effective Link 1 Gigabit Link 1 Link 2 100/0 10/0 Link 2 Single Multipath Savings 3. 59 1. 90 3. 54 47% 1. 4% -40 MB @100/66 100 Mbit/s ideal: 3. 223. 20 seconds 100/66 10/66 43. 33 -Multipath 1/66 plugin nearly 46% 22. 97 47% doubles 38. 25 throughput 12% - TCP effects dominate. Pipe not full. - Multipath plugin doubles by adding second stream. Actual capacity irrelevant. 23 Niraj Tolia © May 2006

Postfix Email Trace Replay • Generated 10, 000 email messages from trace • Random data matched to chunk hash data • Preserves some similarity between messages • Replayed through Postfix to a single local server Program Seconds Bytes Sent Postfix 468 172 MB Postfix + DOT 468 117 MB (68%) • Postfix disk bound… DOT CPU overhead negligible • Savings due to duplication within emails 24 Niraj Tolia © May 2006

Postfix Integration • Integrated DOT with the Postfix mail server Program Lo. C Added Lo. C % GTC Lib -- 421 Postfix 70, 824 184 0. 3% smtpd 6, 413 107 1. 7% smtp 3, 378 71 2. 1% • 1 part-time week, 1 student new to Postfix • Includes time to write generic adapter library 25 Niraj Tolia © May 2006

Discussion on Deployment • Application Resilience • DOT is a service - it’s outside the control of the application. • Our Postfix falls back to normal SMTP if – No Transfer Service contact – Transfer keeps failing • In the short term, a simple fallback is encouraged. However, this could interfere with some functions – DOT-based virus scanner… • In the long term, DOT would be a part of a system’s core infrastructure 26 Niraj Tolia © May 2006

Future Work • Security • Application encrypts before DOT - No block-based caching, reuse, mirroring, … • No encryption - Resembles the status quo • In progress: Convergent encryption – Requires integration with DOT chunking • Application Preferences • Encryption, Qo. S, priorities, … – DOT might benefit from application input • Need an extensible way to express these 27 Niraj Tolia © May 2006

Conclusion • DOT separates app. logic from data transfer • Makes it easier to extend both • Architecture works well • Overhead low (especially in wide-area) • Major benefits • Caching • Flexibility to implement new transfer techniques 28 Niraj Tolia © May 2006

Backup Slides 29 Niraj Tolia © May 2006

Normal SMTP Server SMTP Client EHLO 250 Hello MAIL FROM: user … DATA 250 OK 30 Niraj Tolia © May 2006

DOT-Enabled SMTP Server SMTP Client EHLO 250 Hello MAIL FROM: user … X-DOT-DATA (OID+Hints) Xfer Service 250 OK 31 Niraj Tolia © May 2006

Convergent Encryption Hash 1 Hash 2 File Hash 3 • Chunki is encrypted using Hashi • All identical cleartext blocks will map to the same encrypted block • Hashi is further encrypted using a private key 32 Niraj Tolia © May 2006

Standard File Transfer 33 Niraj Tolia © May 2006

Mail Server Evaluation • Trace: 159 days at low volume academic mail server • 458, 861 messages • hash, size of: message, headers, body • Message chunks • hash and size of each chunk • Static chunking and Rabin fingerprinting (Content-based block division) 34 Niraj Tolia © May 2006

DOT chunk caching benefits email Method Total Bytes Percent Bytes SMTP Default 6800 MB - DOT body 5876 MB 86. 41% Rabin body 5056 MB 74. 35% Rabin whole 5496 MB 80. 81% 35 Niraj Tolia © May 2006

Default GTC-GTC Transfer Protocol Sender Receiver GET_DESCRIPTORS(OID) Desc list 1, 2, … GET_CHUNKS(…) Chunk 1 Chunk 2 GET_CHUNKS(…) • GTC-GTC protocol mirrors transfer plugins • Implemented as RPC calls • (Fetches are actually pipelined) 36 Niraj Tolia © May 2006

Rabin Fingerprinting Hash 1 Hash 2 File Data Rabin Fingerprints 4 7 8 2 Natural Boundary 8 Natural Boundary Given Value - 8 37 Niraj Tolia © May 2006

Rabin Fingerprinting: Examples of Edits 1. Original File 2. Addition in chunk – Changes only one hash Figure from “A Low-bandwidth Network File System” 3. Addition creating a new breakpoint 4. Deletion changing size of chunk 38 Niraj Tolia © May 2006

DOT Objects Naming • Objects represented by an OID • Divided into “chunks”, each with a descriptor • Each OID corresponds to a list of descriptors • Data is fetched using descriptor lists • Supports partial transfers 39 Niraj Tolia © May 2006

Innovation in Data Transfer is Hard • Imagine: You have a novel data transfer technique • Say… Bittorrent, a P 2 P protocol for sharing large files • How do you deploy? 1. Update HTTP. Talk to IETF. Modify Apache, IIS, Firefox, Netscape, Opera, IE, Lynx, Wget, … 2. Update SMTP. Talk to IETF. Modify Sendmail, Postfix, Exchange, Mail. app, Eudora, … 3. Give up in frustration 40 Niraj Tolia © May 2006

DOT’s Modular Architecture Application (1) Application API Transfer Plugin DOT (3) Storage Plugin API Storage Plugin Network Transfer Plugin (2) Transfer Plugin API Transfer Plugin Local Storage 41 Niraj Tolia © May 2006

DOT Plugins • Multipath plugin • List of sub-plugins • Balances load • Portable storage plugin • Sender: Copies new data onto USB flash device • Receiver: Scans USB flash device for blocks – naïve filesystem layout, unoptimized • but effective 42 Niraj Tolia © May 2006

Portable Storage Results Device Inserted . . 1126 s (~ 19 min) 43 Niraj Tolia © May 2006

DOT email chunk caching evaluation • Step 1: Caching analysis (infinite cache) • SMTP default: What was really sent • DOT body: Whole-body only caching, headers sent separately • Rabin body: Headers sent separately, rabin fingerprint chunking of body • Rabin whole: Headers+body chunked together • Easiest to implement for application. Just send data… • Step 2: Trace replay through Postfix 44 Niraj Tolia © May 2006

Related Work • BEEP • Proxy-based data interposition approaches – RON, X-Bone, OCALA • Other Content-Addressable Systems – Bittorrent, DHTs, DTNs, EMC’s Centera • Using Content-Addressability to save on data transfers – CASPER, LBFS, Rhea et al. , Spring et al. • Portable Storage – Lookaside Caching, Blue. FS • Other transfer protocols – Grid. FTP, IBP, HTTP, etc. 45 Niraj Tolia © May 2006

Transfer plugin API • get_descriptors( OID, hints ) • get_chunks( descriptor, hints ) • cancel_chunks( chunk_list ) • Hints specify a plugin + data • gtc: //sender. example. com: 12000/ • dht: //opendht. org/ • … • Transfer plugin chaining is easy • e. g. , multipath plugin 46 Niraj Tolia © May 2006