Types of Sparsity and Hierarchical Chunking in OLAP

Types of Sparsity and Hierarchical Chunking in OLAP 용 환승, 龍 煥昇, Hwan-Seung Yong Dept. of Computer Science and Engineering Ewha Womans University, 梨花女子大學校 Seoul, Korea http: //dblab. ewha. ac. kr/hsyong H. S. Yong, Ewha Womans Univ.

Contents • “Brain Korea Program”, Project “Chamois” and Human. Society@Internet Conference • Sparsity Issues on OLAP Cube • On chunking method based on dimension hierarchy • On going works – Implementation works – Metadata Management – User Interface 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 2

College of Engineering • Established in 1996. – The First Engineering college of the Women’s Univ. in the World • Consists of 4 Departments (established year) – – Dept. of Computer Science and Engineering (1981) Dept. of Information Electronics Engineering (1994) Dept. of Architecture (1994) Dept. of Environmental Engineering (1993) • Number of Faculty: 34 • Number of Graduate Students: about 200 • Number of Undergraduate student: 950 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 3

“Brain Korea 21” Program • Korean Government try to reform education system to have global competitiveness • Ministry of Education launched ‘Brain Korea’ program from Sept. 1999 to support R&D in University • Ewha’s mission – Provide Women professionals to Information Technology industry who can become immediately productive, rather than requiring lengthy and extensive retraining on the job – Develop advanced information technology and applications • Establish EIST(Ewha Institute of Science and Technology) – Dean: Dr. Won Kim 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 4

Project “Chamois” • A flagship project for the future of CSE department • Director: Dr. Won Kim, Dean of EIST. • “Chamois” means an antelope living in alps – after only a few steps, they can jump very high. • Research Area – Knowledge Engineering for internet based Business Intelligence – Major Sub Area: Data Staging, Data Warehousing, OLAP, Data Mining • Goal – Knowledge Engineering Platform Prototype System development within 5 years – Strategic Application development using the Prototype 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 5

Data Warehousing • • Data warehousing methodology Data mart-based enterprise data warehouse design Distributed data warehousing architecture Metadata management Data Quality Performance Evaluation & Metrics Visualization of data warehousing 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 6

OLAP • • • R-OLAP vs M-OLAP vs H-OLAP Usability Performance techniques and benchmarking for OLAP database primitives in DBMS Visualization of OLAP 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 7

Data Mining • • Taxonomy of algorithms Evaluation of existing DM algorithms Ease of use DM Process automation DM database primitives DM in multimedia data Development of new DM algorithms Visualization of DM 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 8

Internet/XML/Application Area • XML/HTML Internet Document Content Search and Analysis • XML Storage and Query System • XML based metadata management – RDBMS metadata exchange using XML – OLAP metadata exchange using XML • Development of Electronic Commerce Platform • Virtual Reality, Computer Graphics • Jointly work with Dept. of Electronics Engineering – Embedded Web Server, Mobile Computing etc. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 9

Human. Society@Internet Conference • Purpose – To view academically Social Impact of Internet on Human Life and Society – Identify “The Good”, “The Ugly”, “The Bad” face of the internet and provide solutions for human Cyber Society – The 1 st interdisciplinary conference on Internet in KOREA – Held on May 19 -20 at Ewha Womans University • Conference Logo 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 10

Human. Society@Internet Conference • Programs – Invited Speaker, Panel, Invited Presentation session – Initially we start with 10 Program Areas –Management & Economy –Publishing & Library –Electronic Government –Education –Law and Regulation –Entertainment –Medicine –Media –Electronic Commerce –Technology • Results and Future Plan – 400 persons attended with positive feedback on the conference – In 2001, will grow as an International conference from more than 10 countries. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 11

What is Sparsity • Why data can be sparse? – Based on origin of it, there are three kinds of Sparsity • Missing • Unknown • Inapplicable • Meaning of “Inapplicable” – Largely depends on application • Ex 1) All branch do not sell all kind of products. • Ex 2) Specific branch opens from March 2000. – Before opening, there are no sales data • What is the Problems due to sparsity – Cube will have many vacant cells but keep storage space. – Large expansion by aggregation 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 12

Sparsity Control in Current OLAP system • Most system do not have specific feature for sparsity control • Only some system have primitive level control – Classify each dimension as sparse or dense (True/False) – Based on this, they optimize storage using indexing mechanism – ORACLE, ESSBASE • But more work on sparsity is required – To refine sparsity control – So that we can get efficient storage space and fast response time • To do this – We analyzed characteristics of sparsity in OLAP 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 13

Type of Sparsity between Two Dimensions • random: Randomly distribute sales data • stripe: some product sells well only summer • cluster: Only specific brand sells well in specific area 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 14

Type of sparsity for 3 dimension 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 15

Type of Sparsity between 3 dimensions • Define Sparsity relationship – A □ B : Dimension A and B have sparse relationship – A ■ B: Dimension A and B have dense relationship • Transitivity of dense relationship – If A ■ B and B ■ C, then A ■ C – But Transitivity of sparse relationship is not true. • If A □ B and B□ C, then A□ C or A ■ C • Sparse priority relationship – If A ■ B and B □ C, then A □ C 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 16

Type of Sparsity between 3 dimensions • Possible combination of Sparsity in 3 Dimensions A and B B and C C and A Possibility ■ ■ ■ Yes. ■ ■ □ No. ■ □ □ Yes. (Slice) □ □ □ Yes. • Let’s focus on all sparse relationship based on sparsity type. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 17

Type of Sparsity in 3 dimension Cube Sparsity Type of A and B B and C Sparsity Type of Cube A and C Random Random Stripe Stripe Stripe Cluster Random Stripe Cluster Stripe Cluster 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 18

Current solution for Sparsity • In ORACLE’s Patent Document – Their solution can handle only stripe type sparsity • In ESSBASE’s Patent Document – Can handle ‘slice’ type sparsity • Best approach – Analyze sparsity type in OLAP Cube automatically. • Using OLAP model definition from user • During cube generation phase – Select appropriate storage technique and apply. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 19

On chunking method based on dimension hierarchy • Chunking in MOLAP – storage clustering technique of neighbor cells. – Treat every dimension equal, not biased. – For sparse cube, ROLAP is often used to save storage • ROLAP do not store sparse cells – MOLAP is more appropriate for dense cube • Research Issues – Multidimensional indexing technique which gives chunking effect • Cluster neighbor cells – Z-order curve was used for multimedia indexing for clustering large image files – Quadtree can handle this? 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 20

On hierarchical Chunking • Chunking Method – Proposed to store large multi-dimensional array – Do not give any favor on specific dimension during clustering on Disk space. – When we retrieve any subset of dimension in the array, almost equal number of Disk I/O is required. – Array is divided and stored into same size of chunk (block size) • Array items does not have any logical boundaries. • In storing OLAP cube – Leaf items of multidimensional data can be grouped by same parent items in the dimension hierarchy – Drill-down operation can be done on the same parent wise. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 21

On hierarchical Chunking • Example Cubes: 2 dimension with measure • Product – category/subcategory/brand/product name c 1 category subcategory brand c 2 sc 1 b 1 sc 2 b 3 sc 3 b 4 b 5 sc 4 b 6 b 7 • Store – state: st 1, st 2 – Region: r 1, r 2 – store name: s 1, s 2 …. . 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 22

category subcategory brand Chunking Example c 1 sc 1 b 1 Product Dimension P 2 sc 2 b 2 P 3 P 4 b 3 P 5 P 6 sc 3 b 4 P 7 c 2 P 8 b 5 P 9 P 10 sc 4 b 6 P 11 b 7 P 12 P 13 P 14 P 15 region S 1 state r 1 S 2 S 3 st 1 Chunk 0 Chunk 1 Chunk 2 Chunk 3 Chunk 4 Chunk 5 S 4 Store dimension r 2 S 5 S 6 r 3 S 7 S 8 st 2 S 9 r 4 2020 -11 -04 store H. S. Yong, Ewha Womans Univ. 23

category subcategory On hierarchical Chunking brand c 1 sc 1 b 1 Product Dimension P 2 sc 2 b 2 P 3 P 4 b 3 P 5 P 6 sc 3 b 4 P 7 c 2 P 8 sc 4 b 5 P 9 P 10 b 6 P 11 b 7 P 12 P 13 P 14 P 15 product region S 1 r 1 S 2 S 3 st 1 chunk 0 Chunk 1 S 4 Store dimension r 2 S 5 S 6 r 3 S 7 st 2 S 8 Chunk 2 Chunk 3 S 9 r 4 2020 -11 -04 store H. S. Yong, Ewha Womans Univ. 24

On hierarchical Chunking • Indexing is required to search appropriate chunks if given items Product Store Chunk# – Chunk Table • Category level for Product • State level for Store c 1 st 1 0 c 2 st 2 1 c 1 st 1 2 c 2 st 2 3 – How we can search for ‘subcategory’ or ‘brand’ name for chunk? • Chunk Index Table for each dimension is used. Product Name 2020 -11 -04 Store c 1 null c 2 null sc 1 sc 2 c 1 sc 3 c 2 sc 4 c 2 … Other dimension H. S. Yong, Ewha Womans Univ. 25

On going works and future plan • Implementation efforts – Storage system for Small OLAP examples • Existing MOLAP and ROLAP storage system • Main purpose is to learn technique – Implementations of proposed chunking system • Metadata Management – Goal: • Transfer OLAP metadata and data between OLAP systems – Work already done • Extract OLAP Metadata from MS SQL Server OLAP • Represent OLAP metadata using XML • Store it EXCELON XML DBMS and retrieval module via Web Browser. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 26

On going works and future plan • User Interface – Goal: • Develop more useful cube browser for higher dimensions • Other user interface issues – At present • Survey OLAP user interface on commercial product – Approach • 3 dimensional tool to browse OLAP cube • Automatic selection of appropriate graphic chart style • On-Graph OLAP operation – Direct OLAP operation (Drill-down/Roll-up) on graphic chart display • Compact Visualization of sparse data • Seek other metaphor such as spiral curve etc. 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 27

Conclusion Thanks you for your attention. 아리가도 고자이마스 2020 -11 -04 H. S. Yong, Ewha Womans Univ. 28