FASMI Fast Analysis of Shared Multidimensional Information Fast

¢ FASMI ¤ Fast Analysis of Shared Multidimensional Information ¤ Fast ¤ interactive system (5 ~ 20초 이내의 응답을 요구) ¤ keep a chain of thought ¤ Analysis ¤ Ad hoc. Analysis , Flexible end-user calculation ¤ Shared ¤ multiple access , one-write, multiple-read ¤ security 보장 ¤ Multidimensional ¤ MDB에서 처럼 데이터를 다차원적으로 관찰 ¤ eg) 판매정보의 차원 : 날짜, 품목, 대리점, 사원등의 관점에서 관찰

¢ 일반적인 작업 ¤ consolidation, aggregation ¤ Ad hoc query ¤ trend analysis ¤ drill-down, roll-up ¤ pivoting ¤ Charting Filtering Reporting Top. N/Bottom. N Running Sum/Running Avg. Ranking Sorting Stop lighting Exception Reporting

¤ Security ¤ Sceduling ¤ Workflow Management ¤ Data Mining ¤ Mathematical Modeling ¤ Custom Application Development Tool set ¤ Internet-Intranet Functionality ¤ Tuning Utility

¢ Basic Feature ¤ Multidimensional Conceptual View ¤ Intuitive Data Manipulation (Transparency) ¤Direct actions on cells in the view without recourse to menus or multiple actions ¤ Accessibility ¤Middleware Between heterogeneous data sources and OLAP front end ¤다양한 자료원천으로부터 완전, 일관성있는 업무모 델 구축

¢ Basic Feature ¤ Batch Extraction vs. Interpretive ¤ OLAP Analysis Model ¤static reporting ¤slicing & dicing with drill-down ¤‘what-if analysis’ ¤ Client-server architecture ¤ Transparency ¤ Multi-user Support

¢ Special Features ¤ Treatment of Non-Normalized Data ¤ Storing OLAP results ¤ Extraction of Missing Value ¤ Treatment of Missing Value ¢ Reporting Features ¤ Flexible Reporting ¤ Uniform Reporting Performance

¢ Dimension Control ¤ Generic Dimensionality ¤eg) time dimension ¤ Unlimited Dimensions & Aggregation level ¤ Unrestricted Cross-dimensional operation

¢ Relational OLAP(ROLAP) ¢ Multidimensional OLAP(MOLAP) ¢ Hybrid OLAP ¤ ROLAP + MOLAP

RDBMS SQL 분석 및 다차원 view 제공 OLAP engine 분석 및 다차원 view 제공 RDBMS User

Meta Data 질의생성(SQL) RDBMS 다차원 형태로 결과 변환(Cross. Tabulation 엔진) ROLAP engine 다차원 질의 결과

¢ 다양한 indexing 기법 ¢ Denormalization ¢ 요약 ¢ 분할 ¢ Star schema ¢ Snow-flake schema

Sales_person table (dimension table) customer_ID name Address Phone_number Time table (dimension table) Time_ID month display_date year sales table (fact table) Custom_ID Sales_person ID Time ID Product_ID Quantity_sold Total_amount 레코드수 : 5 , 000 person_ID name SSN Phone_number product table (dimension table) product ID Unit_price UPC_code

¢ 데이터 예제 Dimension Table

Metacube ¢ Platinum Info. Beacon ¢ Micro Strategy DSS server ¢ Brio Query ¢ Buisiness Objects ¢ Informix ¢ Information Advantage, Inc. Decision. Suite

RDB Field => dimension cell => record MDB

Dimension의 수에 따라 셀의 수가 기하급수적으로 증가 52 weeks 100 regions 520, 000 cells 100 products 예) 16 -dimension MDB에서 각 dimension의 항목의 수가 5라면 => 152, 587, 890, 625셀

10% MDB 50, 556 cells 90% RDB MDB one-cell fetch 속도 = RDB one record fetch 속도

¢ Local copy of Data warehouse ¤ small sized DW ¤ Focus on only one (or two) subject ¤ not contain ODS ¤ easy navigation ¢ DW 구축 방법 ¤ data mart => data warehouse

Data Mart Data Warehouse Data Mart

¢ 구성 절차 ¤ Requirement gathering ¤ Interview with end-user & DBA ¤ “what is the need or problem? ” ¤ data analysis ¤ Database design ¤ subject 수 결정 ¤ business requirement => physical database ¤ Data sourcing ¤ directly from DW or other source system (transaction system) ¤ Data delivery ¤ querying tool ex) web-based query interface ¤ Administration & support ¤ performance tuning ( when condition changes, or some users are added ) ¤ data refreshing

¢ Data Mining의 정의 ¢ 도입예제 ¢ Data Mining Process ¢ Data Mining의 요소 ¢ Data Mining의 응용분야 ¢ Data Mining 방법 ¢ Data Mining 알고리즘 ¤ Association ¤ Classification Rule ¢ Data Mining & Data Warehouse ¢ OLAP mining

• OLAP - Who are my best customer in region X? - summary data에서 질의 • Data Mining - What sort of customers should I target ? - detail 또는 summary data에서 질의

¤ Shallow knowledge ¤ easily retrieved from databases using a query tool(SQL) ¤ Multi-dimensional knowledge ¤ analyzed using online analytical processing tools(OLAP tool) ¤ Hidden Knowledge ¤ found by using pattern recognition or machine-learning algorithm ¤ Data Mining ¤ time-consuming processing => performance 향상이 관 건 ¤ Deep Knowledge ¤ can only be retrieved if we have a clue that tell us where to look

¢ Data explosion : automated generation & gathering => tremendous volume of data knowledge data but starving for ¢ Need to inference from data to generalized information automatically ¢ Computing power의 증가

¢ interactive , iterative ongoing processing

¢ A. 문제이해 및 목표 데이타(target data set) 의 선택 ¤ 기본 목표를 이해하여 mining 내용을 정의 => target data set의 결정 ¤ supervised learning의 경우 사전 정보들을 정의 data cleaning(데 이 타 preprocessing ¢ B. 세척) & ¤ 오류데이타(noise)의 제거, missing data field를 처리 ¢ C. transformation ¤ data reduction & projection & coding ¤ determine relevant attribute

¢ D. DM task와 method의 결정 ¤DM task : Knowledge Engineering tasks, Classfication tasks, Problem-solving tasks ¤DM method : machine learning ( decision tree. . . ), statistical approach, neural network model, rough set, fuzzy logic, visualization, multistrategy mining 등 ¢ E. Knowledge extraction ¤추출된 knowlege의 quality가 좋지 않다면, B, C, 또 는 D 단계를 반복

¢ Data set ¤training data : for learning ¤test data : for prediction ¢ Knowledge Representation ¤Decision Tree, Production Rules, Decision Lists, Ripple-down Rule Sets, First-order Semantic Nets, Frames, Neural Networks Logic,

¢ Search Algorithms( DM method ) ¤ Combination of different fields ¤ Database System + Artificial Intelligence + Statistics + Mathematics ¤ Data Mining 시스템 : 다양한 지식추출을 위해 여러 알고리 즘을 병합한 hybrid algorithm을 갖추는 것이 바람직 ¢ Tasks ¤ Knowledge Engineering Tasks ¤ knowledge-based system에 서 올 바 른 knowledge representation을 찾아내는 process ¤ Classification Tasks ¤ task of assigning a classification to a set of examples ¤ Problem-Solving Tasks

¢ Background Knowledge, Learning Interface ¤learning process를 효과적으로 수행하기 위해 추가 적으로 learning algorithm에 추가되는 knowledge ¤ex) Data Dictionary 정보 ¤ foreign key, composite key 정보 ¤ex) "Concept Hierarchy"

¢classification ¢association rule ¢sequencial pattern ¢similar time series ¢clustering ¢deviation detection

¢ 어떤 집합을 여러개의 disjoint한 부분집합으로 쪼 개어서 각각을 어떤 특정한 성질을 가지는 클래 스로 구분 ¢ 응용예) ¤ marketing ¤if ( buyers of expensive sports car ) then young urban professional ¤if ( buyers of luxury sedan ) then elderly wealthy person ¤ medical diagnosis ¤if ( smoke = Y & age >= 50 ) then "potential cancer" group

Training Data age <=35 salary <=40 B 미래의 데이타 salary <=50 G B G

¢ Input - hidden - output layer ¢ 예)

- to find relationship bet’n the presence of items Li : Large Item Set Ci : Candidate Item Set B, C => E (지지도 0. 5, 신뢰도 1. 0)

¢ Transaction : ( customer-id, transaction time, item ) ¢ discovery of inter-transaction pattern ¤cf) Association Rule : intra-transaction pattern ¤need not be consecutive ¤video대여의 sequential pattern : < "Star Wars" > < "Empire Strikes Back" > < "Return of the Jedi" > ¤당뇨병 환자의 10%는 시력이 점점 약해져 간다.

¢ capability to find time-series ( or portion ) that are "similar" to a give time series or to find groups of similar time-series on time-series data ¢ ex) ¤ identifying companies with similar pattern of growth ¤ finding products with similar selling patterns ¤ discovering stocks with similar price movement ¤ determine portion of seismic wave that has a particular pattern

¢ Process of separating a data set into components that reflect a consistent pattern of behavior ¤discover subsets of related objects according to some criteria or metric & find description of each subset ¢ Clustering according to similarity ¤ similarity function의 정의가 중요

진화 ¢ 적자생존의 원리 ¤ selection, crossover, mutation ¤ fitness function

¢ Discover and describe a set of data which diviate from major trend/characteristics ¤find significant change of data from previous data ¢ Method ¤find major and minor trends by data clustering and data distribution analysis ¤similarity matching

¢ Classification Rule의 형태 ¤ if A, B, C then P class ¢ Decision Tree ¢ Neural Network, Genetic algorithm

¤ DM을 위해서 DW를 구축할 필요는 없지만, DW가 DM을 위해서 효율적인 platform을 제공한다. ¤ DW은 clean, integrated, complete data를 제공 ¤ successful knowledge를 빠른 시간내에 발견할 수 있는 가능성을 증폭 ¤ Integration of DM in a decision support system ¤ DM에 필요한 데이터를 제공 ¤ Integration ( 두가지 측면 ) ¤ DM from the existing DW ¤ DM by extracting from the existing DW the part of the information that is of interest to the end-user and copying it to a specific computer, possibly a multiprocessing machine

¢ Integrated Data ¤ integrated data가 없다면, 데이터를 cleansing 또는 conditioning하는데 많은 시간을 소비하게 된다. ¤ What is 'Integrated' ? ¤ key의 재구성 ¤ encoded values의 통일 ¤ structures of data standardized ¢ Detailed and Summarized Data ¤ Miner can build on the work of others rather than build everything for the ground up ¤ 결국 시간 절약

¢ Historical Data ¤Important nuggets of information are hidden in huge amount of data ¤ex) trace the seasonablity of business ¢ Metadata ¤road map to miner ( exploration of the world of data )

¢ What is OLAP data mining ? ¢ Motive ¤ may miss key pieces of information ¤ may get incorrect results

OLAP DM engine OLAP engine SQL engine Summarized data Granular data

¢ DBMiner ¤ DM need to work on integrated, consistent, and cleaned data ¤ OLAP mining facilitates interactive exploratory data analysis ¤ often difficult for users to predict what kinds of knowledge to be mined

¢ Cubing operations ¤ drill-down/roll-up, slicing, dicing, filtering, pivoting ¤ generate a new data cubes