OnTheFly Verification of Rateless Erasure Codes Max Krohn

On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)

Multicas t Authen tication: Dead/E xhauste d On-The-Fly Verification of Rateless Erasure Codes Max Krohn (MIT CSAIL) Michael Freedman and David Mazières (NYU)

The Setting n A large file F q n H(F) is available q n Linux ISO (650 MB) signed by Publisher (Red. Hat) A handful of untrusted sources/mirrors S 1, …S 8

A Handful of Senders

The Setting n A large file F q n H(F) is available q n signed by Publisher (Red. Hat) A handful of untrusted sources S 1, …S 8 q n Linux ISO (650 MB) Their aggregate BW is limited A slew of receivers R 1, . . . , R 1, 000 q Version 81. 3 just released! Want it Now!

Three Desirable Properties Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

Receivers Get Fast, Verifiable Downloads n The trusted publisher (Red. Hat) q q q n Splits up F into n blocks Hashes all blocks Signs all hashes (or hash tree) Receivers: q q Download and verify hashes Download needed file blocks in parallel

Everyone for Themselves S 1 S 2 S 3 R 9 R R 7 R 2 R R 12 3 R 8 10 R 13 R R 4 R 1 11 R 5 R 6 S 4

Everyone For Themselves Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

Verifiable Multicast (Bit. Torrent) S 3 S 2 S 1 R 7 R 13 R 5 R 9 R 8 R 11 S 4 R 12 R 10 R 6 R 3 R 4 R 1 R 2

Verifiable Multicast (Bit. Torrent) Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

Multicast With Erasure Codes n Sources erasure encode the file F F n blocks ? ? ? ? ? ? ? ? ? ? ? ? ? blocks … … …

Multicast With Erasure Codes Sources erasure encode the file F n n blocks F ? ? ? ? ? ? ? ? ? ? ? ? ? blocks Receivers collect blocks and decode n ? ? ? ? ? ? ? ? 1. 03 n blocks ? ? n blocks F … … …

Multicast With Erasure Codes S 2 S 1 R 9 R 8 S 3 R 10 R 6 R 5 R 12 R 4 R 11 R 3 R 13 R 7 R 2 S 4

Multicast With Erasure Codes n n Bullet [SOSP 2003] Split. Stream [SOSP 2003] Big Downloads [IPTPS 2003] Informed Content Delivery [SIGCOMM 2002]

Receivers Cannot Verify Content S 2 S 1 S 3 S 4 ? ? ? ? ? R 1 ? ? ?

Receivers Cannot Verify Content S 2 S 1 S 3 ? ? ? R 1 S 4

Multicast With Erasure Codes Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

Multicast With Erasure Codes Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

What is the Attack Goal? n To corrupt the file. n To waste bandwidth. S 1 S 2 R S 3

How To Attack? n Send correct blocks but with skewed distributions. q n “Distribution Attack” Send incorrect blocks q S 1 S 2 “Pollution Attack” n Karlof et al. [NDSS ’ 04] R S 3

Properties of a Solution to n. Pollution OK: Receivers can tell good from bad. n Much better: Receivers can finger bad blocks as they arrive. CONTRIBUTION S 1 S 2 R S 3

Outline n n n Introduction Review of LT Codes Strawman #1 Strawman #2 Efficiently Catching Bad Blocks as They Arrive

LT-Codes [Luby, FOCS 2002] F= b 1 b 2 b 3 n=5 input blocks b 4 b 5

LT-Codes – How To Encode E(F)= 1. Pick degree d 1 from a pre-specified distribution. (d 1=2) c 1 2. Select d 1 input blocks uniformly at random. (Pick b 1 and b 4 ) 3. Compute their sum. 4. Output F= b 1 b 2 b 3 b 4 b 5

LT-Codes – How To Encode (cont’d) E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

How To Decode E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

How To Decode E(F)= F= c 1 b 2 c 2 b 5 b 2 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

How To Decode E(F)= F= c 1 b 2 c 2 b 5 b 2 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

Outline n n n Introduction Review of LT Codes Strawman #1 q n n Simple Solution To Tell Good Blocks From Bad Strawman #2 Efficiently Catching Bad Blocks as They Arrive

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 c 3 c 4 c 5 b 1 b 2 b 3 b 4 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 b 1 X b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 b 1 b 2 c 3 X b 3 b 4 b 5 c 5 b 5 c 6 c 7

“Smart Decoder” for LT-Codes E(F)= F= c 1 c 2 b 1 b 2 b 5 c 3 b 5 c 4 b 5 c 5 b 5 c 6 c 7

“Smart Decoder: ” Problem • Data collected from 50 random Online encodings of a 10, 000 block file.

Outline n n Introduction Review of LT Codes Strawman #1 Strawman #2 q n Hashing/Signing Encoded Blocks Efficiently Catching the Bad as They Arrive

Hashing/Signing Encoded Blocks F n ? ? ? ? ? ? ? ? ? ? ? ? ? n blocks Trusted Publisher (Red. Hat) q q q Picks e, computes e·n encoded blocks Hashes all encoded blocks Signs the hashes. e·n blocks

Hashing/Signing Encoded Blocks n n n Expansion factor e should be big to avoid duplicate blocks. e should be small to make crypto overhead acceptable. Our analysis shows there’s no “sweet spot”.

Hashing/Signing Encoded Blocks n n n Expansion factor e should be big to avoid duplicate blocks. e should be small to make crypto overhead acceptable. Our analysis shows there’s no “sweet spot”. q q e. g. , best case bandwidth requirements: +5% e. g. , generating hashes is very expensive as e gets large.

Outline n n n Introduction Review of LT Codes Strawman #1 Strawman #2 Efficiently Catching the Bad as They Arrive

Best of Both Worlds n Goal: q q n Crypto overhead of one hash for every block in the input file (Strawman #1) Verify blocks as they arrive (Strawman #2) Idea: q q Distribute hashes of file blocks, and use them to verify encoded blocks. Need a better hash function.

Insight: Homomorphic Hashing Assume function h exists such that: n 1. is homomorphic: 2. is a CRHF:

Homomorphic Hashing: Intuition n R receives the block R knows: R wants proof that: c c b 2 b 5

Homomorphic Hashing: Intuition n R receives the block R knows: c R wants proof that: c b 2 b 5

Homomorphic Hashing: Intuition n R receives the block R knows: c R wants proof that: c b 2 b 5

Homomorphic Hashing: Intuition n R receives the block R knows: R wants proof that:

Homomorphic Hashing: Intuition n R receives the block R knows: Property 1 R wants proof that:

Homomorphic Hashing: Intuition n R receives the block R knows: R wants proof that: Property 1 Property 2

Homomorphic Hashing: Protocol n R receives the block q q Compute h(c) If n q Accept block; mark as valid else n Suspect sender of being bad guy, and switch.

Homomorphic Hashing: Protocol n R receives the block q q Compute h(c) If n q else n n Accept block; mark as valid Suspect sender of being bad guy, and switch. Can such an h possibly exist?

Homomorphic Hashing: Related Work n DLog-Based CRHF q q n One-Way Accumulators q q n Benaloh and de Mare [EUROCRYPT ’ 93] Barić and Pfitzmann [EUROCRYPT ’ 93] Incremental Hashing q n Pederson Commitment [CRYPTO ’ 91] Chaum et al. [CRYPTO ’ 91] Bellare et al. [CRYPTO ’ 94] Homomorphic Signatures q q Micali and Rivest [RSA ’ 02] Johnson et al. [RSA ’ 02]

Mechanics of Homomorphic Hashing n Discrete Log Hash n Pick 1024 -bit prime p and 256 -bit prime q, q divides (p-1) n Pick from n Write F as elements in 512 generators of order q: F= 256 -bit “fragment” 16 K “block”

How to Encode (example) Standard LT-Codes: Homomorphic Scheme:

How To DLog Hash n Hashes are elements in (128 bytes big) n Hash reduces 16 K block by a factor of 128

How To DLog Hash n Hashes are elements in n Hash reduces 16 K block by a factor of 128 q +1% overhead (128 bytes big)

DLog-Hash: Key Property n Note that:

DLog-Hash: Key Property n Note that: n Goal achieved!

“This Seems Really Expensive” Operation on a 16 K Block DLog Hash Arrival on 1. 5 Mbps DSL SHA 1 Hash Throughput (k. B/sec) 39 190 57, 600

Key Optimizations n Hash Generation q q n Each publisher picks her own parameters, compute with 1 exponentiation (not 512) Hash Verification q Receiver verifies hashes probabilistically and in batches. n Bellare et al. [EUROCRYPT ’ 98]

Much Better Operation on a 16 K Block Naïve DLog Hash Per-publisher Generation Throughput (MB/sec) 0. 038 11. 210 Batch Verification 7. 620 Arrival on 1. 5 Mbps DSL 0. 186 SHA 1 Hash 56. 250

Homomorphic Hashing: Key +Points Key Algebraic Feature + + + Fast + + Homomorphism: Receivers can compose hashes the way encoders sum file blocks. Can check encoded blocks as they arrive. Can be optimized to achieve good generation and verification throughputs Provably Secure + As hard as discrete log (SHA 1/MD 5 not needed)

Conclusion Clients Get Fast Downloads Sources Can Multicast Clients Can Verify Blocks On-the-Fly

Thank you. Now accepting questions.