Copyright Proprietary and Confidential All rights reserved 3

巨量資料 – 導論 Copyright © Proprietary and Confidential. All rights reserved. 3

Trend of Big Data 係指資料大量成長根據IBM的研究，全世界90%的資料是在過去 2年產生 Google、Facebook 等，就是站在Big Data上的範例巨大的數據源，將改變整個學術界，商界和政府依賴新的資訊科技來處理包括 capture，

"Data Scientist : The sexist job of the 21 st century", Harvard Business Review,

Big Data 的主要來源 Enterprise data, Social data, Machine data Source : IBM 2012全球CEO調查報告 Copyright

Big Data 的特性數量大、產生速度快、多樣性、可能存有誤差資料 Source : IBM Big Data Hub Copyright © Proprietary and

Big Data 的應用方式運用資料與演算，達成智慧決策 Source : IBM 2012全球CEO調查報告 Copyright © Proprietary and Confidential. All

Marketing and CRM Cycle Data Warehousing Data Mining E-Marketing Copyright © Proprietary and Confidential.

巨量資料 – 分析技術 Copyright © Proprietary and Confidential. All rights reserved. 13

Big data 的資料種類企業的結構性資料與非結構性資料 Copyright © Proprietary and Confidential. All rights reserved.

Twitter 200 million tweets per day Peak 10, 000 per second How to analyze

Facebook 6 billion messages per day 2 PB (compressed) online 6 PB replication 250

e. Bay Analyze & Report Discover & Explore Copyright © Proprietary and Confidential. All

Big data 的分析方式結構性資料分析 Data Mining 資料探勘非結構性資料分析 Text Mining 文字探勘轉結構性資料 Copyright ©

常見的 Data Mining 模組群集分析 Clustering 分類預測 Classification 關聯規則 Association rules 連續行為 Sequential pattern

基本原理：以相關性分析為例產品組合 {2, 5} 或 {2, 3, 5} 最常被一起購買 Copyright © Proprietary and Confidential.

分群演算法 K-means 範例 (K=2) Pick seeds Reassign clusters Compute centroids Reassign clusters x x

分類預測 : 眼科診所病例 Copyright © Proprietary and Confidential. All rights reserved.

分類預測 : 眼科診所病例 (續) 自動選擇最佳分支條件，產生決策樹 Copyright © Proprietary and Confidential. All rights reserved.

決策樹演算法範例 Weather Data: Play tennis or not ? Copyright © Proprietary and Confidential.

Which attribute to choose ? Copyright © Proprietary and Confidential. All rights reserved. 28

Which attribute to choose ? choose the attribute that produces the "purest" nodes …and

第一層選擇outlook 重複產生分支，直到結束或終止條件為止 Copyright © Proprietary and Confidential. All rights reserved. 30

練習使用SQL group by協助，產生次數統計表進行information gain計算決定欄位，之後再重複上述動作 Copyright © Proprietary and Confidential. All rights

Copyright © Proprietary and Confidential. All rights reserved. 32

非結構資料的處理 – 欄位化 Copyright © Proprietary and Confidential. All rights reserved. 33

a Copyright © Proprietary and Confidential. All rights reserved. 34

利用標記 – 方式(1) 新增標記欄位 tag 1, tag 2, … ALTER TABLE content ADD tag

利用標記 – 方式(2) 新增一張表 CREATE TABLE tag (id int, tag char(20), primary key (id,

常見有 : Apriori 演算法、 FP growth 演算法 Copyright © Proprietary and Confidential. All rights

4. 連續行為 Sequential pattern 客戶購買某產品後之某段期間內，會再購買的產品例：錄影帶 Star War → Empire Strikes Back → Return

連續行為 Sequential pattern (續) Copyright © Proprietary and Confidential. All rights reserved.

連續行為 Sequential pattern (續) 最熱門連續行為 Jurassic Park → Toy Story , Jurassic Park 2

在各產業的應用 – 以金融保險業為例 Copyright © Proprietary and Confidential. All rights reserved. 53

1. 尋找保戶購買保單的決策模型 Copyright © Proprietary and Confidential. All rights reserved. 56

自動選擇最佳分支條件，產生決策樹 Copyright © Proprietary and Confidential. All rights reserved. 58

2. 尋找最熱門之保戶保單的關聯性 Copyright © Proprietary and Confidential. All rights reserved. 61

3. 尋找主力保戶客群之特徵 Copyright © Proprietary and Confidential. All rights reserved. 64

盈收貢獻度問題想了解購買三張保單，或累計投保 1000萬以上的主力客群特徵？ Copyright © Proprietary and Confidential. All rights reserved. 65

Copyright © Proprietary and Confidential. All rights reserved. 66

在各產業的應用 – 以零售通路為例 Copyright © Proprietary and Confidential. All rights reserved. 68

Slides: 73

Download presentation

Trend of Big Data 係指資料大量成長根據IBM的研究，全世界90%的資料是在過去 2年產生 Google、Facebook 等，就是站在Big Data上的範例巨大的數據源，將改變整個學術界，商界和政府依賴新的資訊科技來處理包括 capture， storage， search， analytics 等 Copyright © Proprietary and Confidential. All rights reserved. 4

Twitter 200 million tweets per day Peak 10, 000 per second How to analyze the data ? Zynga "Analytics company, not a gaming company“ 230 million players per month Harvest 15 TB data per day ‐ test new features ‐ target advertising 4 U box = 40 TB 1 PB = 25 boxes Copyright © Proprietary and Confidential. All rights reserved. 15

分群演算法 K-means 範例 (K=2) Pick seeds Reassign clusters Compute centroids Reassign clusters x x x Compute centroids x Reassign clusters Converged! 重點在計算資料相似性 (similarity) 視資料與群集多寡，通常做 3至 4回就大致穩定 Copyright © Proprietary and Confidential. All rights reserved. 23

Which attribute to choose ? choose the attribute that produces the "purest" nodes …and more informative 常見演算法 Information gain (ID 3, C 4. 5, C 5) ig(outlook) = average(3/5, 4/4, 3/5) = 0. 73 ig(humidity) = average(4/7, 6/7) = 0. 71 ig(windy) = 0. 63 = average(6/8, 3/6) ig(temperature) = average(2/4, 4/6, 3/4) Copyright © Proprietary and Confidential. All rights reserved. = 0. 64 29

利用標記 – 方式(1) 新增標記欄位 tag 1, tag 2, … ALTER TABLE content ADD tag 1 int, tag 2 int; 使用條件做標記 UPDATE content SET tag 1 = 1 WHERE content LIKE '*柯文哲*'; UPDATE content SET tag 2 = 1 WHERE content LIKE '*連勝文*'; 進行統計 SELECT sum(tag 1) as '柯文哲篇數', sum(tag 2) as '連勝文篇數' FROM content; Copyright © Proprietary and Confidential. All rights reserved. 35

利用標記 – 方式(2) 新增一張表 CREATE TABLE tag (id int, tag char(20), primary key (id, tag)); 使用條件做標記，新增紀錄 INSERT INTO tag SELECT * FROM ( SELECT id, '柯文哲' AS tag FROM content WHERE content LIKE '*柯文哲*' UNION ALL SELECT id, '連勝文' AS tag FROM content WHERE content LIKE '*連勝文*'); 進行統計 Copyright © Proprietary and Confidential. All rights reserved. 36

連續行為 Sequential pattern (續) 最熱門連續行為 Jurassic Park → Toy Story , Jurassic Park 2 : Lost World Jurassic Park → Terminator 2 : Judgment Day 行銷建議產品合購優惠方案櫃台人員主動推薦內部商品擺設建議 Copyright © Proprietary and Confidential. All rights reserved.