Data Mining Presented By Kevin Seng 4301 CS

Data Mining Presented By: Kevin Seng 4/3/01 CS 632 - Data Mining 1

Papers Rakesh Agrawal and Ramakrishnan Srikant: n Fast algorithms for mining association rules. n Mining sequential patterns. 4/3/01 CS 632 - Data Mining 2

Outline… For each paper… n Present the problem. n Describe the algorithms. n n n 4/3/01 Intuition Design Performance. CS 632 - Data Mining 3

Market Basket Introduction n Retailers are able to collect massive amounts of sales data (basket data) n n n 4/3/01 Bar-code technology E-commerce Sales data generally includes customer id, transaction date and items bought. CS 632 - Data Mining 4

Market Basket Problem n It would be useful to find association rules between transactions. n n 4/3/01 ie. 75% of the people who buy spaghetti also by tomato sauce. Given a set of basket data, how can we efficiently find the set of association rules? CS 632 - Data Mining 5

Formal Definition (1) n n n L = {i 1, i 2, … im} set of items. Database D is a set of transactions. Transaction T is a set of items such that T L. n 4/3/01 An unique identifier, TID, is associated with each transaction. CS 632 - Data Mining 6

Formal Definition (2) n n T contains X, a set of some items in L, if X T. Association rule, X Y n n n 4/3/01 X T, Y T, X Y = Confidence – % of transactions which contain X which also contain Y. Support - % of transactions in D which contain X Y. CS 632 - Data Mining 7

Formal Definition (3) n 4/3/01 Given a set of transactions D, we want to generate all association rules that have support and confidence greater than the user-specified minimum support (minsup) and minimum confidence (minconf). CS 632 - Data Mining 8

Problem Decomposition Two sub-problems: n Find all itemsets that have transaction support above minsup. n n 4/3/01 These itemsets are called large itemsets. From all the large itemsets, generate the set of association rules that have confidence about minconf. CS 632 - Data Mining 9

Second Sub-problem Straightforward approach: n For every large itemset l, find all nonempty subsets of l. n For every such subset a, output a rule of the form a (l – a) if ratio of support(l) to support(a) is at least minconf. 4/3/01 CS 632 - Data Mining 10

Discovering Large Itemsets n n n Done with multiple passes over the data. First pass, find all individual items that are large (have minimum support). Subsequent pass, using large itemsets found in previous pass: n n n 4/3/01 Generate candidate itemsets. Count support for each candidate itemset. Eliminate itemsets that do not have min support. CS 632 - Data Mining 11

Algorithm L 1 = {large 1 -itemsets}; for( k=2; Lk-1 ; k++) do begin Ck = apriori-gen(Lk-1); // New candidates forall transactions t D do // Counting support Ct = subset(Ck, t); // Candidates in t forall candidates c Ct do c. count++; end Lk = {c Ck | c. count minsup} End Answer = k Lk; 4/3/01 CS 632 - Data Mining 12

Candidate Generation AIS and SETM algorithms: n Uses the transactions in the database to generate new candidates. n But… this generates a lot of candidates which we know beforehand are not large! 4/3/01 CS 632 - Data Mining 13

Apriori Algorithms Generate candidates using only large itemsets found in previous pass without considering the database. Intuition: n Any subset of a large itemset must be large. n 4/3/01 CS 632 - Data Mining 14

Apriori Candidate Generation n Takes in Lk-1 and returns Ck. Two steps: n Join large itemsets Lk-1 with Lk-1. n Prune out all itemsets in joined result which contain a (k-1)subset not found in Lk-1. 4/3/01 CS 632 - Data Mining 15

Candidate Generation (Join) insert into Ck select p. item 1, p. item 2, …, p. itemk-1, q. itemk-1 from Lk-1 p, Lk-1 q where p. item 1= q. item 1, …, p. itemk-2= q. itemk-2, p. itemk-1< q. itemk-1 4/3/01 CS 632 - Data Mining 16

Candidate Gen. (Example) L 3 {1 {1 {2 2 2 3 3 3 4/3/01 3} 4} 4} 5} 4} Join C 4 {1 2 3 4} Prune {1 2 3 4} {1 3 4 5} CS 632 - Data Mining 17

Counting Support n n Need to count the number of transactions which support a given itemset. For efficiency, use a hash-tree. n 4/3/01 Subset Function CS 632 - Data Mining 18

Subset Function (Hash-tree) n n n Candidate itemsets are stored in hash-tree. Leaf node – contains a list of itemsets. Interior node – contains a hash table. n n n 4/3/01 Each bucket of the hash table points to another node. Root is at depth 1. Interior nodes at depth d points to nodes at depth d+1. CS 632 - Data Mining 19

Hash-tree Example (1) depth C 3 {1 {1 {2 1 2 2 3 3} 4} 4} 5} 4} 4/3/01 2 {1 2 3} {1 2 4} 3 {1 3 4} {1 3 5} CS 632 - Data Mining 2 1 {2 3 4} 2 3 t=2 20

Using the hash-tree n n n 4/3/01 If we are at a leaf – find all itemsets contained in transaction. If we are at an interior node – hash on each remaining element in transaction. Root node – hash on all elements in transaction. CS 632 - Data Mining 21

Hash-tree Example (2) 1 D {1 2 3 4} {2 3 5} 2 {1 2 3} {1 2 4} 4/3/01 3 2 {2 3 4} {1 3 5} CS 632 - Data Mining 22

Apriori. Tid (1) n n Does not use the transactions in the database for counting itemset support. Instead stores transactions as sets of possible large itemsets, Ck. n Each member of Ck is of the form: < TID, {Xk}> , Xk is a possible large itemset 4/3/01 CS 632 - Data Mining 23

Apriori. Tid (2) Advantage of Ck n If a transaction does not contain any candidate k-itemset then it will have no entry in Ck. n Number of entries in Ck may be less than the number of transactions in D. n n 4/3/01 Especially for large k. Speeds up counting! CS 632 - Data Mining 24

Apriori. Tid (3) However… n For small k each entry in Ck may be larger than it’s corresponding transaction. n The usual space vs. time. 4/3/01 CS 632 - Data Mining 25

Apriori. Tid (4) Example 4/3/01 CS 632 - Data Mining 26

Observation n n When Ck does not fit in main memory we can see large jump in execution time. Apriori. Tid beats Apriori only when Ck can fit in main memory. 4/3/01 CS 632 - Data Mining 27

Apriori. Hybrid n n It is not necessary to use the same algorithm for all the passes. Combine the two algorithms! n n 4/3/01 Start with Apriori When Ck can fit in main memory switch to Apriori. TID CS 632 - Data Mining 28

Performance (1) n n 4/3/01 Measured performance by running algorithms on generated synthetic data. Used the following parameters: CS 632 - Data Mining 29

Performance (2) 4/3/01 CS 632 - Data Mining 30

Performance (3) 4/3/01 CS 632 - Data Mining 31

Mining Sequential Patterns (1) n n Sequential patterns are ordered list of itemsets. Market basket example: n n 4/3/01 Customers typically rent “star wars” then “empire strikes back” then “return of the Jedi” “Fitted sheets and pillow cases” then “comforter” then “drapes and ruffles” CS 632 - Data Mining 32

Mining Sequential Patterns (2) n n Looks at sequences of transactions as opposed to a single transaction. Groups transactions based on customer ID. n 4/3/01 Customer sequence. CS 632 - Data Mining 33

Formal Definition (1) n n Given a database D of customer transactions. Each transaction consists of: customer id, transaction-time, items purchased. n 4/3/01 No customer has more than one transaction with the same transaction-time. CS 632 - Data Mining 34

Formal Definition (2) n n 4/3/01 Itemset i, (i 1 i 2. . . im) where ij is an item. Sequence s, s 1 s 2…sn where sj is an itemset. Sequence a 1 a 2…an contained in b 1 b 2…bn if there exist integers i 1< i 2. . . < in such that a 1 bi 1 , a 2 bi 2 , …, an bin. A sequence s is maximal if it is not contained in any other sequence. CS 632 - Data Mining 35

Formal Definition (3) n n A customer supports a sequence s if s is contained in the customer sequence for this customer. Support of a sequence - % of customers who support the sequence. n 4/3/01 For mining association rules, support was % of transactions. CS 632 - Data Mining 36

Formal Definition (4) n n 4/3/01 Given a database D of customer transactions find the maximal sequences among all sequences that have a certain user-specified minimum support. Sequences that have support above minsup are large sequences. CS 632 - Data Mining 37

Algorithm: Sort Phase n n Customer ID – Major key Transaction-time – Minor key Converts the original transaction database into a database of customer sequences. 4/3/01 CS 632 - Data Mining 38

Algorithm: Litemset Phase (1) Litemset Phase: n Find all large itemsets. Why? n Because each itemset in a large sequence has to be a large itemset. 4/3/01 CS 632 - Data Mining 39

Algorithm: Litemset Phase (2) n n To get all large itemsets we can use the Apriori algorithms discussed earlier. Need to modify support counting. n 4/3/01 For sequential patterns, support is measured by fraction of customers. CS 632 - Data Mining 40

Algorithm: Litemset Phase (3) n n 4/3/01 Each large itemset is then mapped to a set of contiguous integers. Used to compare two large itemsets in constant time. CS 632 - Data Mining 41

Algorithm: Transformation (1) n n n 4/3/01 Need to repeatedly determine which of a given set of large sequences are contained in a customer sequence. Represent transactions as sets of large itemsets. Customer sequence now becomes a list of sets of itemsets. CS 632 - Data Mining 42

Algorithm: Transformation (2) 4/3/01 CS 632 - Data Mining 43

Algorithm: Sequence Phase (1) n n Use the set of large itemsets to find the desired sequences. Similar structure to Apriori algorithms used to find large itemsets. n n n 4/3/01 Use seed set to generate candidate sequences. Count support for each candidate. Eliminate candidate sequences which are not large. CS 632 - Data Mining 44

Algorithm: Sequence Phase (2) Two types of algorithms: n Count-all: counts all large sequences, including non-maximal sequences. n n Count-some: try to avoid counting nonmaximal sequences by counting longer sequences first. n n 4/3/01 Apriori. All Apriori. Some Dynamic. Some CS 632 - Data Mining 45

Algorithm: Maximal Phase (1) n n 4/3/01 Find the maximal sequences among the set of large sequences. Set of all large subsequences S CS 632 - Data Mining 46

Algorithm: Maximal Phase (2) n n n 4/3/01 Use hash-tree to find all subsequences of sk in S. Similar to subset function used in finding large itemsets. S is stored in hash-tree. CS 632 - Data Mining 47

Apriori. All (1) 4/3/01 CS 632 - Data Mining 48

Apriori. All (2) n n n Hash-tree is used for counting. Candidate generation similar to candidate generation in finding large itemsets. Except that order matters and therefore we don’t have the condition: p. itemk-1< q. itemk-1 4/3/01 CS 632 - Data Mining 49

Apriori. All (3) Example of candidate generation: 4/3/01 CS 632 - Data Mining 50

Apriori. All (4) n 4/3/01 Example: CS 632 - Data Mining 51

Count-some Algorithms n n Try to avoid counting non-maximal sequences by counting longer sequences first. 2 phases: n n 4/3/01 Forward Phase – find all large sequences or certain lengths. Backward Phase – find all remaining large sequences. CS 632 - Data Mining 52

Apriori. Some (1) n n Determines which lengths to count using next() function. next() takes in as a parameter the length of the sequence counted in the last pass. next(k) = k + 1 - Same as Apriori. All Balances tradeoff between: n n 4/3/01 Counting non-maximal sequences Counting extensions of small candidate sequences CS 632 - Data Mining 53

Apriori. Some (2) n n 4/3/01 hitk = Lk / Ck Intuition: As hitk increases the time wasted by counting extensions of small candidates decreases. CS 632 - Data Mining 54

Apriori. Some (3) 4/3/01 CS 632 - Data Mining 55

Apriori. Some (4) Backward Phase: n For all lengths which we skipped: n n n 4/3/01 Delete sequences in candidate set which are contained in some large sequence. Count remaining candidates and find all sequences with min. support. Also delete large sequences found in forward phase which are non-maximal. CS 632 - Data Mining 56

Apriori. Some (5) 4/3/01 CS 632 - Data Mining 57

Apriori. Some (6) n Example: Forward Phase: next(k) = 2 k minsup = 2 C 3 3 -Sequences 4/3/01 CS 632 - Data Mining 58

Apriori. Some (7) n Example Backward Phase: C 3 3 -Sequences 4/3/01 CS 632 - Data Mining 59

Performance (1) n n 4/3/01 Used generated datasets again… Parameters for data: CS 632 - Data Mining 60

Performance (2) n n Dynamic. Some generates too many candidates. Apriori. Some does a little better than Apriori. All. n 4/3/01 It avoids counting many non-maximal sequences. CS 632 - Data Mining 61

Performance (3) Advantage of Apriori. Some is reduced for 2 reasons: n Apriori. Some generates more candidates. n Candidates remain memory resident even if skipped over. n 4/3/01 Cannot always follow heuristic. CS 632 - Data Mining 62

Wrap up n Just presented two classic papers on data-mining. n n 4/3/01 Association Rules Sequential Patterns CS 632 - Data Mining 63