Big Data Infrastructure CS 489698 Big Data Infrastructure
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2016) Week 4: Analyzing Text (2/2) January 28, 2016 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These slides are available at http: //lintool. github. io/bigdata-2016 w/ This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3. 0 United States See http: //creativecommons. org/licenses/by-nc-sa/3. 0/us/ for details
Count. Search! Source: http: //www. flickr. com/photos/guvnah/7861418602/
Abstract IR Architecture Query Documents online offline Representation Function Query Representation Document Representation Retrieval Comparison Function Indexing Hits
Doc 1 Doc 2 one fish, two fish 1 blue red fish, blue fish 2 3 1 egg fish 1 1 1 green 1 ham 1 hat one 1 1 red two 1 1 cat in the hat Doc 4 green eggs and ham 4 1 cat Doc 3 What goes in each cell? boolean count positions
Doc 1 Doc 2 one fish, two fish 1 blue red fish, blue fish 2 3 1 egg fish 1 1 cat in the hat Doc 4 green eggs and ham 4 1 cat Doc 3 1 blue 2 cat 3 egg 4 fish 1 green 4 ham 1 ham 4 hat 3 one 1 red 2 two 1 hat one 1 1 red two 1 1 2 sts er) i l s g n i t pos rted ord in (always so
Doc 1 Doc 2 one fish, two fish red fish, blue fish Doc 3 cat in the hat Doc 4 green eggs and ham tf 1 blue 2 3 1 cat 1 egg fish 4 1 2 2 df 1 blue 1 2 1 1 cat 1 3 1 1 egg 1 4 1 2 fish 2 1 2 green 1 1 green 1 4 1 ham 1 4 1 1 hat 1 3 1 1 one 1 1 red 1 2 1 1 two 1 1 1 hat one 1 1 red two 1 1 2 2
Doc 1 Doc 2 one fish, two fish red fish, blue fish Doc 3 cat in the hat Doc 4 green eggs and ham tf 1 blue 2 3 1 cat 1 egg fish 4 1 2 2 df 1 blue 1 2 1 [3] 1 cat 1 3 1 [1] 1 egg 1 4 1 [2] 2 fish 2 1 2 [2, 4] green 1 1 green 1 4 1 [1] ham 1 1 ham 1 4 1 [3] 1 hat 1 3 1 [2] 1 one 1 1 1 [1] 1 red 1 2 1 [1] 1 two 1 1 1 [3] hat one 1 1 red two 1 1 2 2 [2, 4]
Inverted Indexing with Map. Reduce Doc 1 Doc 2 one fish, two fish Map Doc 3 red fish, blue fish cat in the hat one 1 1 red 2 1 cat 3 1 two 1 1 blue 2 1 hat 3 1 fish 1 2 fish 2 2 Shuffle and Sort: aggregate values by keys cat Reduce fish 3 1 1 2 one 1 1 red 2 1 2 2 blue 2 1 hat 3 1 two 1 1
Inverted Indexing: Pseudo-Code
Positional Indexes Doc 1 Doc 2 one fish, two fish Map Doc 3 red fish, blue fish cat in the hat one 1 1 [1] red 2 1 [1] cat 3 1 [1] two 1 1 [3] blue 2 1 [3] hat 3 1 [2] fish 1 2 [2, 4] fish 2 2 [2, 4] Shuffle and Sort: aggregate values by keys cat Reduce fish one red 3 1 1 2 1 [1] [2, 4] [1] 2 2 blue 2 1 [3] hat 3 1 [2] two 1 1 [3] [2, 4]
Inverted Indexing: Pseudo-Code What ? m e l b o r ’s th e p
Another Try… (key) fish (values) (keys) (values) 1 2 [2, 4] fish 1 [2, 4] 34 1 [23] fish 9 [9] 21 3 [1, 8, 22] fish 21 [1, 8, 22] 35 2 [8, 41] fish 34 [23] 80 3 [2, 9, 76] fish 35 [8, 41] 9 1 [9] fish 80 [2, 9, 76] How is this different? • Let the framework do the sorting • Term frequency implicitly stored Where have we seen this before?
Inverted Indexing: Pseudo-Code
Postings Encoding Conceptually: fish 1 2 9 1 21 3 34 1 35 2 80 3 … In Practice: • Don’t encode docnos, encode gaps (or d-gaps) • But it’s not obvious that this save space… fish 1 2 8 1 12 3 13 1 1 2 45 = delta encoding, delta compression, gap compression
Overview of Integer Compression ¢ Byte-aligned technique l ¢ Bit-aligned l l l ¢ VByte Unary codes / codes Golomb codes (local Bernoulli model) Word-aligned l l Simple family Bit packing family (PFor. Delta, etc. )
VByte ¢ Simple idea: use only as many bytes as needed l l ¢ Need to reserve one bit per byte as the “continuation bit” Use remaining bits for encoding value 7 bits 0 14 bits 1 0 21 bits 1 1 0 Works okay, easy to implement… Beware of branch mispredicts!
Simple-9 ¢ How many different ways can we divide up 28 bits? 28 1 -bit numbers 14 2 -bit numbers 9 3 -bit numbers 7 4 -bit numbers “selectors” (9 total ways) l ¢ Efficient decompression with hard-coded decoders Simple Family – general idea applies to 64 -bit words, etc. Beware of branch mispredicts?
Bit Packing ¢ What’s the smallest number of bits we need to code a block (=128) of integers? 3 … 4 … 5 l ¢ … Efficient decompression with hard-coded decoders PFor. Delta – bit packing + separate storage of “overflow” bits Beware of branch mispredicts?
Golomb Codes ¢ x 1, parameter b: l l ¢ Example: l l ¢ q + 1 in unary, where q = ( x - 1 ) / b r in binary, where r = x - qb - 1, in log b or log b bits b = 3, r = 0, 1, 2 (0, 11) b = 6, r = 0, 1, 2, 3, 4, 5 (00, 01, 100, 101, 110, 111) x = 9, b = 3: q = 2, r = 2, code = 110: 11 x = 9, b = 6: q = 1, r = 2, code = 10: 100 Optimal b 0. 69 (N/df) l Different b for every term!
Chicken and Egg? (key) (value) fish 1 [2, 4] fish 9 [9] fish 21 [1, 8, 22] fish 34 [23] fish 35 [8, 41] fish 80 [2, 9, 76] But wait! How do we set the Golomb parameter b? Recall: optimal b 0. 69 (N/df) We need the df to set b… But we don’t know the df until we’ve seen all postings! … Write postings Sound familiar?
Getting the df ¢ In the mapper: l ¢ In the reducer: l ¢ Emit “special” key-value pairs to keep track of df Make sure “special” key-value pairs come first: process them to determine df Remember: proper partitioning!
Getting the df: Modified Mapper Doc 1 one fish, two fish (key) Input document… (value) fish 1 [2, 4] one 1 [1] two 1 [3] fish [1] one [1] two [1] Emit normal key-value pairs… Emit “special” key-value pairs to keep track of df…
Getting the df: Modified Reducer (key) (value) fish [63] fish 1 [2, 4] fish 9 [9] fish 21 [1, 8, 22] fish 34 [23] fish 35 [8, 41] fish 80 [2, 9, 76] … [82] [27] First, compute the df by summing contributions from all “special” key-value Computepair… b from df … Important: properly define sort order to make sure “special” key-value pairs come first! Write postings Where have we seen this before?
Inverted Indexing: IP n? o i t p m u s s a the kay? s ’ t a Wh Is it o
Merging Postings ¢ Let’s define an operation ⊕ on postings lists P: Postings(1, 15, 22, 39, 54) ⊕ Postings(2, 46) = Postings(1, 2, 15, 22, 39, 46, 54) n? o i t a r e p is o h t s i y l t ac x e t d? a e h t a W e r c we e v a h t a Wh ¢ Then we can rewrite our indexing algorithm! l l flat. Map: emit singleton postings reduce. By. Key: ⊕
What’s the issue? Postings 1 ⊕ Postings 2 = Postings. M Solution: apply compression as needed!
Inverted Indexing: LP Slightly less elegant implementation… but uses same idea
Inverted Indexing: LP
IP vs. LP? Experiments on Clue. Web 09 collection: segments 1 + 2 101. 8 m documents (472 GB compressed, 2. 97 TB uncompressed) From: Elsayed et al. , Brute-Force Approaches to Batch Retrieval: Scalable Indexing with Map. Reduce, or Why Bother?
Another Look at LP k? r a p S n i g thin y n a f o u o mind y Re RDD[(K, V)] aggregate. By. Key seq. Op: (U, V) ⇒ U, comb. Op: (U, U) ⇒ U RDD[(K, U)]
Algorithm design in a nutshell… Exploit associativity and commutativity via commutative monoids (if you can) Exploit framework-based sorting to sequence computations (if you can’t) Source: Wikipedia (Walnut)
Abstract IR Architecture Query Documents online offline Representation Function Query Representation Document Representation Retrieval Comparison Function Indexing Hits
Doc 1 Doc 2 one fish, two fish 1 blue red fish, blue fish 2 3 fish 1 1 1 green 1 ham 1 hat one 1 1 red two 1 1 green eggs and ham Indexing: building this structure Retrieval: manipulating this structure 1 egg cat in the hat Doc 4 4 1 cat Doc 3
Map. Reduce it? ¢ The indexing problem l l l ¢ Pe rfect for M Scalability is critical ap. Reduce ! Must be relatively fast, but need not be real time Fundamentally a batch operation Incremental updates may or may not be important For the web, crawling is a challenge in itself The retrieval problem l l Must have sub-second response time For the web, only need relatively few results d… o o g o s t o n Uh…
Assume everything fits in memory on a single machine… (For now)
Boolean Retrieval ¢ Users express queries as a Boolean expression l l ¢ AND, OR, NOT Can be arbitrarily nested Retrieval is based on the notion of sets l l Any given query divides the collection into two sets: retrieved, not-retrieved Pure Boolean systems do not define an ordering of the results
Boolean Retrieval ¢ To execute a Boolean query: l Build query syntax tree OR ( blue AND fish ) OR ham l l ham AND For each clause, look up postings blue 2 5 9 fish 1 2 3 5 ham 1 3 4 5 6 Traverse postings and apply Boolean operator 7 8 fish 9
Term-at-a-Time OR ham AND blue fish AND blue 2 5 9 fish 1 2 3 5 ham 1 3 4 5 5 6 7 8 9 9 fish Efficiency analysis? OR ham 1 2 3 4 5 9 AND blue fish What’s RPN?
Document-at-a-Time OR ham AND blue fish blue 2 5 9 fish 1 2 3 5 ham 1 3 4 5 1 ham 1 2 3 3 5 4 8 7 5 2 fish 6 9 6 7 5 Tradeoffs? Efficiency analysis? 8 9 9
Strengths and Weaknesses ¢ Strengths l l l ¢ Precise, if you know the right strategies Precise, if you have an idea of what you’re looking for Implementations are fast and efficient Weaknesses l l l Users must learn Boolean logic insufficient to capture the richness of language No control over size of result set: either too many hits or none When do you stop reading? All documents in the result set are considered “equally good” What about partial matches? Documents that “don’t quite match” the query may be useful also
Ranked Retrieval ¢ Order documents by how likely they are to be relevant l l l ¢ User model l l ¢ Estimate relevance(q, di) Sort documents by relevance Display sorted results Present hits one screen at a time, best results first At any point, users can decide to stop looking How do we estimate relevance? l l l Assume document is relevant if it has a lot of query terms Replace relevance(q, di) with sim(q, di) Compute similarity of vector representations
Vector Space Model t 3 d 2 d 3 d 1 θ φ t 1 d 5 t 2 d 4 Assumption: Documents that are “close together” in vector space “talk about” the same things Therefore, retrieve documents based on how close the document is to the query (i. e. , similarity ~ “closeness”)
Similarity Metric ¢ Use “angle” between the vectors: ¢ Or, more generally, inner products:
Term Weighting ¢ Term weights consist of two components l l ¢ Here’s the intuition: l l ¢ Local: how important is the term in this document? Global: how important is the term in the collection? Terms that appear often in a document should get high weights Terms that appear in many documents should get low weights How do we capture this mathematically? l l Term frequency (local) Inverse document frequency (global)
TF. IDF Term Weighting weight assigned to term i in document j number of occurrence of term i in document j number of documents in entire collection number of documents with term i
Retrieval in a Nutshell ¢ Look up postings lists corresponding to query terms ¢ Traverse postings for each query term ¢ Store partial query-document scores in accumulators ¢ Select top k results to return
Retrieval: Document-at-a-Time ¢ Evaluate documents one at a time (score all query terms) blue fish 1 2 9 2 21 1 9 1 21 3 34 1 35 2 … 80 3 … Document score in top k? Accumulators (e. g. min heap) ¢ Yes: Insert document score, extract-min if heap too large No: Do nothing Tradeoffs l l l Small memory footprint (good) Skipping possible to avoid reading all postings (good) More seeks and irregular data accesses (bad)
Retrieval: Term-At-A-Time ¢ Evaluate documents one query term at a time l Usually, starting from most rare term (often with tf-sorted postings) blue 9 2 21 1 35 1 … Accumulators Score{q=x}(doc n) = s fish ¢ 1 2 9 1 21 3 34 1 35 2 80 3 (e. g. , hash) … Tradeoffs l l Early termination heuristics (good) Large memory footprint (bad), but filtering heuristics possible
Assume everything fits in memory on a single machine… Okay, let’s relax this assumption now
Important Ideas ¢ Partitioning (for scalability) ¢ Replication (for redundancy) ¢ Caching (for speed) ¢ Routing (for load balancing) The rest is just details!
Term vs. Document Partitioning D T 1 D T 2 Term Partitioning … T 3 T Document Partitioning T … D 1 D 2 D 3
FE brokers partitions … … … cache … replicas …
Datacenter brokers partitions … … … Tier … … … … cache Tier … part replicas cache … replicas … … … partitions … … Tier … cache … … part replicas cache … replicas … … … partitions … … Tier … cache … … replicas … cache … … part … replicas Tier … … Datace … … …
Important Ideas ¢ Partitioning (for scalability) ¢ Replication (for redundancy) ¢ Caching (for speed) ¢ Routing (for load balancing)
Questions? Remember: Assignment 3 due next Tuesday at 8: 30 am Source: Wikipedia (Japanese rock garden)
- Slides: 55