Advanced Topics in Data Mining Web Mining Web

Advanced Topics in Data Mining: Web Mining

Web Mining

Web Mining • Applications are ported to the Web at rapid pace • On-line services, such as America Online (AOL), and Compu. Serve (merged to AOL), are anxious to know user access patterns; not just “search” in the Web • How Amazon does it? • Understanding Web user behavior is important – – It can improve Web page organization It can increase Web server performance It can exploit Web advertising It can increase business opportunity

Amazon Web Page Association Rules

More Information Desired • Collect statistical information (page hits) only, which is insufficient since: – The hit frequency of a page depends not only on its content but also on its location – The number of users accessing a page is not available – Information on what pages accessed together is not available • Data mining in the Web (Web Mining) – Web Access Pattern Collection – Web User Pattern Mining

Web Access Pattern Collection • Server-Based Data Collection – Who are visiting a given Web site and what are they doing • Agent-Based Data Collection – What are the Web sites a particular user has visited?

Server-Based Data Collection • Examine the logs collected by HTTPd – Access Log (IP, Time, Access Data), Referred Log (A B), Error Log, … – We can combining some of them for our use if necessary • Problems – The use of proxy servers – The effect of caching

Server-Based Data Collection

Access Log IP/Domain Name Time Access Data

Referred Log 不考慮Caching的問題

Server-Based Data Collection • Have to be done in accordance with technology advances – The use of Active Server Pages (Session ID available) • The use of proxy servers • The effect of caching – HTTPd 1. 1 • Limitation – Can only capture the user behavior when they are within this site

Agent-Based Data Collection • Understanding individual Web behavior needs client-based data collection • Results are useful – Better Personalized Service – Improved Web Page Organization – Better Pricing Policies • Methods – Applets can only read/write files in their source servers • a big security constraint – Using Active Components (Active. X Control) and Plug. Ins • APCS (Access Pattern Collection Server)

APCS

APCS

APCS

APCS

APCS

Agent-Based Data Collection • Very difficult to do for non-registered users in the current Web environment – We have to be conducted with users’ consent • Very dependent upon available Web technologies

Web User Pattern Mining • Web user pattern mining is to discover user access patterns in Web servers • Pattern discovery and analysis tools – Some existing Web tools provide mechanisms for reporting user activity in the servers – Web Trends (http: //www. webtrends. com. tw/) – Open Market (http: //www. openmarket. com/) – Net. Genesis (http: //www. netgen. com/)

Path Traversal Patterns Mining • Mining path traversal patterns in a distributed information providing environment (WWW) where documents or objects are linked together (via hyperlinks) to facilitate interactive access • Solution procedure consists of three steps: – Convert the original sequence of log data into a set of maximal forward references (MF) • Filter out the effect of some backward references – Mainly made for ease of traveling and concentrate on mining meaningful user access sequences – Some objects are visited because of their locations rather than their content – Determine the frequent traversal patterns, i. e. , large reference sequences, from the maximal forward references obtained – Determine the maximal reference sequences from large reference sequences (Trivial)

Step 1: MF References • Suppose the traversal log contains the following traversal path for a user: – A, B, C, D, C, B, E, G, H, G, W, A, O, U, O, V When backward references occur, a forward reference path terminate. The set of maximal forward references is {ABCD, ABEGH, ABEGW, AOU, AOV}

Step 1: Another Example

Step 1: Arrange Database Encoding

Step 1: Database Reduction

Step 2: Find Frequent Reference Sequences • Two algorithms for finding Frequent Traversal Patterns (Frequent Reference Sequences, Frequent Consecutive Subsequences) – Full-Scan (FS) Algorithm • FS utilizes key ideas of the DHP algorithm – Selective-Scan (SS) Algorithm • SS reduces the number of database scans

Full-Scan (FS) Algorithm Generate L 1 & Hash Table Scan DB-1

Generate L 1 & Hash Table Scan DB-1 h(x, y) = [ ( order of x ) * 23 + ( order of y ) ] mod 17

Generate C 2

Generate L 2 & Reduce DB Scan DB-2

Generate L 2 & Reduce DB Scan DB-2

Generate C 3, L 3 & Reduce DB Scan DB-3

Generate C 4, L 4 & Reduce DB Scan DB-4

Selective-Scan (SS) Algorithm Scan DB-3

Step 3: Generate Frequent Traversal Patterns Maximal Reference Sequences

WAP-Mine Algorithm • The key consideration is how to facilitate the tedious support counting and candidate generating operations in the mining procedure • Given Web Access Sequence database WAS and a support threshold , mine the complete set of -patterns of WAS User ID 100 200 300 400 Web Access Sequence abdac eaebcac babfaec afbacfc WAS

WAP-Mine Algorithm (1)Scan WAS once, find all frequent-1 events (2)Scan WAS again, construct a WAP-tree (3)Recursively mine the WAP-tree using conditional search Access patterns

Find All Frequent-1 Events Item Support Frequency User ID Web Access Sequence Min_Sup=75% 100 abdac 200 eaebcac 300 babfaec 400 afbacfc User ID Web Access Sequence 100 abdac 200 eaebcac 300 babfaec 400 afbacfc a 4 b 4 c 4 d 1 e 2 f 2 Frequent Subsequence abac abcac babacc

WAP-Tree Construction • Using frequent events to register all count information for further mining User ID Frequent Subsequence 100 abac 200 abcac 300 babac 400 abacc

Mining Web Access Patterns from WAP-Tree Conditional Sequence Based on c Sequence Count aba 2 ab 1 abca 1 ab -1 baba 1 abac 1 aba -1 Generate Web Access Patterns: ac, bc Sequence Count aba 1 abca 1 baba 1 abac 1 Item Sup Frequency a 4 b 4 c 2

Mining Web Access Patterns from WAP-Tree Conditional Sequence Based on ac Sequence Count ab 3 b 1 bab 1 b -1 Generate Web Access Patterns: aac, bac Item Sup Frequency a 4 b 4

Mining Web Access Patterns from WAP-Tree Conditional Sequence Based on bac Sequence Count a 3 ba 1 Item Sup Frequent Generate Web Access Patterns: abac a 4 b 1

Mining Web Access Patterns from WAP-Tree Conditional Sequence Based on abac Sequence Count a 4 No Web Access Patterns are Generated

Mining for Web Transactions • To capture Web customer buying behavior – It is not just market basket transaction for the set of items bought by a customer in a single purchase (Association Rules) – It is not just Web user travel patterns (Path Traversal Patterns) – It is an extension from path traversal patterns • Exploring the relationship between traveling and buying

Mining for Web Transactions Web Transaction Algorithm WR (Web-transaction-Record) Web Transaction Records <Path: a Set of Purchases> Algorithm WTM, MTSPJ, MTSPC Frequent Transaction Patterns Web Transaction Association Rules

Mining for Web Transactions • Web-transaction-Record (WR) Algorithm – Extract meaningful Web transaction records from the given Web transaction • WTM (Web Transaction Mining) Algorithm – Mining Web Transaction Patterns • MTS (Maximal Transaction Segment) Algorithms are the improvement versions of WTM

Mining for Web Transactions

Mining for Web Transactions

WTM Algorithm • It joins the purchased itemsets for generating candidate transaction patterns • WTM employs a two-level hash tree, called Web transaction tree, to store candidate transaction patterns – WTM hashes not only each item but also each purchase in the path

WTM Algorithm DATABASE Web Transaction WT_ID Path ABCE 100 200 300 400 Purchase B{i 1}, C{i 2}, E{i 4} ABFGH B{i 1}, H{i 6} ASJL S{i 7}, L{i 9} ABCE B{i 1}, C{i 2}, E{i 4} ASJLQ S{i 7}, Q{i 10} ABCE B{i 1}, E{i 4} ABFG B{i 1}, G{i 5} ASJL S{i 7}, J{i 8}, L{i 9} ABD D{i 3} ABFG G{i 5} ASJLQ S{i 7}, J{i 8}, Q{i 10}

Support Count WT_ID 100 200 Path Purchase ABCE B{i 1}, C{i 2}, E{i 4} ABFGH B{i 1}, H{i 6} ASJL S{i 7}, L{i 9} ABCE B{i 1}, C{i 2}, E{i 4} ASJLQ S{i 7}, Q{i 10} Purchase Support Count AB B{i 1} 2 ABC C{i 2} 2

WTM Algorithm Support Count >= 2 C 1 T 1 Path Purchase Sup. AB B{i 1} 3 ABC C{i 2} 2 ABD D{i 3} 1 ABCE E{i 4} 3 ABFG G{i 5} 2 AS S{i 7} 4 ABFGH H{i 6} 1 ASJ J{i 8} 2 AS S{i 7} 4 ASJ J{i 8} 2 ASJL L{i 9} 2 ASJLQ Q{i 10} 2

WTM Algorithm Support Count >= 2 C 2 28 共 個 Path Purchase Sup. ABC B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 T 2 Path Purchase Sup. ABC B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 AS B{i 1} S{i 7} 0 ABCE C{i 2} E{i 4} 2 ASJ B{i 1} J{i 8} 0 ASJ S{i 7} J{i 8} 2 ASJL S{i 7} L{i 9} 2 ASJLQ S{i 7} Q{i 10} 2 ASJLQ J{i 8} Q{i 10} 1 ASJLQ L{i 9} Q{i 10} 0

WTM Algorithm Support Count >= 2 C 3 Path Purchase Sup. ABCE B{i 1} C{i 2} E{i 4} 2 T 3 Path Purchase Sup. ABCE B{i 1} C{i 2} E{i 4} 2

WTM Disadvantages • WTM may generate a lot of unqualified candidate transaction patterns without utilizing the paths of frequent transaction patterns • This will degrade the performance

MTSPJ Algorithm • Algorithm MTSPJ uses maximal transaction segment that contains frequent transaction patterns and the maximal path, to solve the unqualified candidate transaction pattern problem • MTSPJ generalizes candidate transaction patterns only when the leaf node of the Web transaction tree is reached

MTSPJ Algorithm DATABASE Web Transaction WT_ID Path ABCE 100 200 300 400 Purchase B{i 1}, C{i 2}, E{i 4} ABFGH B{i 1}, H{i 6} ASJL S{i 7}, L{i 9} ABCE B{i 1}, C{i 2}, E{i 4} ASJLQ S{i 7}, Q{i 10} ABCE B{i 1}, E{i 4} ABFG B{i 1}, G{i 5} ASJL S{i 7}, J{i 8}, L{i 9} ABD D{i 3} ABFG G{i 5} ASJLQ S{i 7}, J{i 8}, Q{i 10} A B C E S F D J G H L Q

MTSPJ Algorithm Support Count >= 2 C 1 T 1 Path Purchase Sup. AB B{i 1} 3 ABC C{i 2} 2 ABCD D{i 3} 1 ABCE E{i 4} 3 ABFG G{i 5} 2 AS S{i 7} 4 ABFGH H{i 6} 1 ASJ J{i 8} 2 ASJL L{i 9} 2 ASJLQ Q{i 10} 2 AS S{i 7} 4 ASJ J{i 8} 2 ASJL L{i 9} 2 ASJLQ Q{i 10} 2 A B C E S J F G L Q

MTSPJ Algorithm A B C S J F E L G Q B{i 1} C{i 2} E{i 4} C 2 Path Purchase B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 ABCE C{i 2} E{i 4} 2 Purchase ASJ S{i 7} J{i 8} Sup. 2 ASJL S{i 7} L{i 9} 2 ASJL J{i 8} L{i 9} 1 2 Maximal Transaction Segment ASJLQ J{i 8} Q{i 10} 1 ABFG ASJLQ L{i 9} Q{i 10} 0 B{i 1} G{i 5} Sup. ABC Path ASJLQ S{i 7} Q{i 10} Maximal Transaction Segment ABCE C 2 Maximal Transaction Segment C 2 Path ABFG Purchase B{i 1} G{i 5} Sup. 1 ASJLQ S{i 7} J{i 8} L{i 9} Q{i 10}

MTSPJ Algorithm C 2 Path Purchase ABC T 2 Sup. Path Purchase Sup. B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 ABCE C{i 2} E{i 4} 2 ABFG B{i 1} G{i 5} 1 ASJ S{i 7} J{i 8} 2 ASJL S{i 7} L{i 9} 2 ASJLQ S{i 7} Q{i 10} 2 ASJL J{i 8} L{i 9} 1 ASJLQ S{i 7} Q{i 10} 2 ASJLQ J{i 8} Q{i 10} 1 ASJLQ L{i 9} Q{i 10} 0

MTSPJ Algorithm A B S C J L E Q Maximal Transaction Segment ABCE B{i 1} C{i 2} E{i 4} C 3 Path ABCE Purchase B{i 1} C{i 2} E{i 4} Sup. 2

MTSPC Algorithm MTSPC utilizes the LC (Large Count) to Filter Candidates Support Count >= 2 C 1 T 1 Path Purchase Sup. AB B{i 1} 3 ABC C{i 2} 2 ABCD D{i 3} 1 ABCE E{i 4} 3 ABFG G{i 5} 2 AS S{i 7} 4 ABFGH H{i 6} 1 ASJ J{i 8} 2 ASJL L{i 9} 2 ASJLQ Q{i 10} 2 AS S{i 7} 4 ASJ J{i 8} 2 ASJL L{i 9} 2 ASJLQ Q{i 10} 2 A B C E S J F G L Q

MTSPC Algorithm A K=1 Maximal Transaction Segment Maximal Path ABCE Item LC B{i 1} 1 C{i 2} 1 E{i 4} 1 S B C Sup. ABC B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 ABCE C{i 2} E{i 4} 2 ASJLQ Q ABFG S{i 7} 1 J{i 8} 1 L{i 9} 1 Q{i 10} 1 Item LC Path B{i 1} 1 ASJ S{i 7} J{i 8} 2 G{i 5} 1 ASJL S{i 7} L{i 9} 2 ASJL J{i 8} L{i 9} 1 C 2 ABFG LC C 2 |I| = 2 > 1 Path Item |I| = 4 > 1 Maximal Transaction Segment C 2 Purchase Maximal Path L G Maximal Path J F E |I| = 3 > 1 (K-1) Maximal Transaction Segment Purchase B{i 1} G{i 5} Sup. 1 Purchase Sup. ASJLQ S{i 7} Q{i 10} 2 ASJLQ J{i 8} Q{i 10} 1 ASJLQ L{i 9} Q{i 10} 0

MTSPC Algorithm C 2 Path Purchase ABC T 2 Sup. Path Purchase Sup. B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 ABCE C{i 2} E{i 4} 2 ABFG B{i 1} G{i 5} 1 ASJ S{i 7} J{i 8} 2 ASJL S{i 7} L{i 9} 2 ASJLQ S{i 7} Q{i 10} 2 ASJL J{i 8} L{i 9} 1 ASJLQ S{i 7} Q{i 10} 2 ASJLQ J{i 8} Q{i 10} 1 ASJLQ L{i 9} Q{i 10} 0

MTSPC Algorithm K=2 A Maximal Transaction Segment Maximal Path ABCE Item LC B{i 1} 2 C{i 2} 2 E{i 4} 2 C 3 Path ABCE Purchase B{i 1} C{i 2} E{i 4} L T 2 Maximal Transaction Segment Maximal Path J C E |I| = 3 > 2 S B ASJLQ Q Path Purchase ABC B{i 1} C{i 2} 2 ABCE B{i 1} E{i 4} 3 ABCE C{i 2} E{i 4} 2 ASJ S{i 7} J{i 8} 2 ASJL S{i 7} L{i 9} 2 ASJLQ S{i 7} Q{i 10} 2 Item LC S{i 7} 3 J{i 8} 1 L{i 9} 1 Q{i 10} 1 Sup. |I| = 1 < 2 No Generations

Mining for Web Transactions • <ABCE : B{1}, E{4}> = 2 • <AB : B{1}> = 3 • We can derive <ABCE : B{1} => E{4}> – support_count(<ABCE : B{1} => E{4}>) = 2 – confidence(<ABCE : B{1} => E{4}>) =

Summary • Data mining in the Web is an area of growing importance – In particular, the emerging of EC – More and more applications will benefit from the knowledge from data mining • Web Mining = Web Data Collection + Traditional Data Mining? • Important Issues – Incremental Web Mining