Chapter 1 Introduction to Data Mining Warehousing and

Chapter 1: Introduction to Data Mining, Warehousing, and Visualization Modern Data Warehousing, Mining, and Visualization: Core Concepts by George M. Marakas © 2003, Prentice-Hall Chapter 1 - 1

1 -1: The Modern Data Warehouse A data warehouse is a copy of transaction data specifically structured for querying, analysis and reporting Note that the data warehouse contains a copy of the transactions. These are not updated or changed later by the transaction system. Also note that this data is specially structured, and may have been transformed when it was placed in the warehouse © 2003, Prentice-Hall 2

1 -2: Data Warehouse Roles and Structures The DW has the following primary functions: It is a direct reflection of the business rules of the enterprise. It is the collection point for strategic information. It is the historical store of strategic information. It is the source of information later delivered to data marts. It is the source of stable data regardless of how the business processes may change. © 2003, Prentice-Hall 3

Position of the Data Warehouse Within the Organization © 2003, Prentice-Hall 4

Data Marts A data mart is a smaller, more focused data warehouse. It reflects the business rules of a specific business unit. The data mart does not need to cleanse its data because that was done when it went into the warehouse. It is a set of tables for direct access by users. These tables are designed for aggregation. It typically is not a source for traditional statistical analysis. © 2003, Prentice-Hall 5

Position of the Data Mart Within the Organization Data Delivery Data Mart Decision Support Information © 2003, Prentice-Hall 6

1 -3: What Can a Data Warehouse Do? Some of the benefits of a DW are: Immediate information delivery Data integration from across and even outside the organization Future vision from historical trends Tools for looking at data in new ways Freedom from IS department resource limitations (you don’t need programmers to use a data warehouse) © 2003, Prentice-Hall 7

Examples of Common DW Applications Sales Analysis Determine real-time product sales to make vital pricing and distribution decisions. Analyze historical product sales to determine success or failure attributes. Evaluate successful products and determine key success factors. Use corporate data to understand the margin as well as the revenue implications of a decision. Rapidly identify a preferred customer segments based on revenue and margin. Quickly isolate past preferred customers who no longer buy. Identify daily what product is in the manufacturing and distribution pipeline. Instantly determine which salespeople are performing, on both a revenue and margin basis, and which are behind. Financial Analysis Compare actual to budgets on an annual, monthly and month-to-date basis. Review past cash flow trends and forecast future needs. Identify and analyze key expense generators. Instantly generate a current set of key financial ratios and indicators. Receive near-real-time, interactive financial statements. Human Resource Analysis Evaluate trends in benefit program use. Identify the wage and benefits costs to determine company-wide variation. Review compliance levels for EEOC and other regulated activities. Other Areas Warehouses have also been applied to areas such as: logistics, inventory, purchasing, detailed transaction analysis and load balancing. © 2003, Prentice-Hall 8

What Does All This Mean? On a daily basis, organizations turn to their data warehouses to answer a limitless variety of questions. Nothing is free, however, and these benefits do come with a cost. The value of a data warehouse is a result of the new and changed business processes it enables. There are limitations, though. A DW cannot correct problems with the data, although it may help to clearly identify them. © 2003, Prentice-Hall 9

Comparison of Typical DW Costs and Benefits Costs Hardware, software, development personnel and consultant costs. Operational costs like ongoing systems maintenance. Benefits Added Revenue Will the new (business objective) process generate new customers (what is the estimated value? ) Will the new (business objective) process increase the buying propensity of existing customers (by how much? ) Is the new process necessary to ensure that the competition doesn't offer a demanded service that you can't match? Reduced costs What costs of current systems will be eliminated? Is the new process intended to make some © 2003, Prentice-Hall operation more efficient? If so, how and what 10

1 -4: The Cost of Warehousing Data Expenditures can be categorized as one-time initial costs or as recurring, ongoing costs. The initial costs can further be identified as for hardware or software. Expenditures can also be categorized as capital costs (associated with acquisition of the warehouse) or as operational costs (associated with running and maintaining the warehouse) © 2003, Prentice-Hall 11

Expenditures Associated with Building a DW Recurring Costs One-Time Costs Capital Hardware maintenance Software maintenance Terminal analysis Middleware Hardware Disk CPU Network Terminal analysis Operational Ongoing refreshment Integration transformation Data model maintenance Record identification maintenance Metadata infrastructure maintenance Archival of data Data aging within the DW © 2003, Prentice-Hall Software DBMS Terminal analysis Middleware Network Log utility Processing Metadata Infrastructure Integration/transformation processing specification Metadata infrastructure population System of record definition Data dictionary language definition Network transfer definition CASE/Repository interface Initial data warehouse population Data model definition Database design definition 12

Cost Are Highly Variable A company that spends less money for their data warehouse is often happier with it. The main justification for the development expense is that a DW reduces the cost of accessing the information owned by the organization. Since information has to be retrieved just once (when it is placed in the warehouse), DW users see a lower cost on each report generated. © 2003, Prentice-Hall 13

Typical Multidatabase Report and Screen Generation Data download and transformation contribute to retrieval costs for every report or screen generated Source System A Source System B Source System C Source System D © 2003, Prentice-Hall 14

Typical DW Report and Screen Generation Data upload and transformation costs occur just once. Retrieval costs are lower. Source System A Source System B Organizational Data Warehouse Source System C Source System D © 2003, Prentice-Hall 15

Farmers and Explorers Every corporation has two types of DW users. Farmers know what they want before they set out to find it. They submit small queries and retrieve small nuggets of information. Explorers are quite unpredictable. They often submit large queries. Sometimes they find nothing, sometimes they find priceless nuggets. Cost justification for the DW is usually done on the basis of the results obtained by farmers since explorers are unpredictable. © 2003, Prentice-Hall 16

1 -5: Data Marts and the Data Warehouse Legacy systems feed data to the warehouse. The warehouse feeds specialized information to departments. Legacy Systems Operational Data Store Finance Data Mart Sales Data Mart Marketing Data Mart Accounting Data Mart Operational Data Store Organizational Data Warehouse Operational Data Store © 2003, Prentice-Hall 17

The Data Mart is More Specialized Organizational Data Warehouse The data mart serves the needs of one business unit, not the organization. Corporate Highly granular data Normalized design Robust historical data Large data volume Data Model driven data Versatile General purpose DBMS Finance Data Mart Sales Data Mart Marketing Data Mart technologies Accting Data Marts Departmentalized Summarized, aggregated Organizational Data Warehouse © 2003, Prentice-Hall data Star join design Limited historical data Limited data volume Requirements driven data Focused on departmental needs Multi-dimensional DBMS technologies 18

1 -6: Foundations of Data Mining Data mining is the process of using raw data to infer important business relationships. Despite a consensus on the value of data mining, a great deal of confusion exists about what it is. It is a collection of powerful techniques intended for analyzing large datasets. There is no single data mining approach, but rather a set of techniques that can be used in combination with each other. © 2003, Prentice-Hall 19

1 -7: The Roots of Data Mining The approach has roots in practice dating back over 30 years. In the early 1960 s, data mining was called statistical analysis, and the pioneers were statistical software companies such as SAS and SPSS. By the 1980 s, the traditional techniques had been augmented by new methods such as fuzzy logic, heuristics and neural networks. © 2003, Prentice-Hall 20

A General Approach Although all data mining endeavors are unique, they possess a common set of process steps: 1. Infrastructure preparation – choice of hardware platform, the database system and one or more mining tools 2. Exploration – looking at summary data, sampling and applying intuition 3. Analysis – each discovered pattern is analyzed for significance and trends © 2003, Prentice-Hall 21

A General Approach (continued) Interpretation – Once patterns have been discovered analyzed, the next step is to interpret them. Considerations include business cycles, seasonality and the population the pattern applies to. 5. Exploitation – this is both a business and a technical activity. One way to exploit a pattern is to use it for prediction. Others are to package, price or advertise the product in a different way. 4. © 2003, Prentice-Hall 22

1. 8: The Approach to Data Exploration and Data Mining A The basis for all data mining activities is correlation. B A Perfect Correlation A B A Strong Correlation A B A Weak Correlation © 2003, Prentice-Hall 23

The Spectrum of Correlation 1 Perfect Correlation . 5 Moderate Correlation 0 No Correlation In general, a correlation coefficient is a number between 0 and 1 that shows strength of a relationship. Some types of correlation are signed (±) to also show the direction of the relationship. Even a weak correlation can be interesting, however, if it shows a trend over time. © 2003, Prentice-Hall 24

Methods to Determine Correlation The method used depends on the type of elements being correlated. A vs. B A vs. B BBB B Data element vs. data element Data element vs. unit of time Data element vs. data element groups Data element vs. geography A vs. Data element vs. external trends A vs. Data element vs. demographics © 2003, Prentice-Hall 25

The Data Warehouse and Data Mining Data mining does not require the use of a warehouse, but it may be the best foundation for mining. If multiple analyses are run in sequence, the data need to be held constant (as in a DW). In an operational database, data change often. Also important is that the data in the DW is integrated and stable © 2003, Prentice-Hall 26

Volumes of Data – The Biggest Challenge The largest challenge a data miner may face is the sheer volume of data in the warehouse. It is quite important, then, that summary data also be available to get the analysis started. A major problem is that this sheer volume may mask the important relationships the analyst is interested in. The ability to overcome the volume and visualize the data becomes quite important. © 2003, Prentice-Hall 27

1. 9: Foundations of Data Visualization One of the earliest known examples of data visualization was in London during the 1854 cholera epidemic. A map (next slide) helped to identify the source of the disease. Modern visualization techniques grew from the twin technologies of computer graphics and high performance computing in the 1970 s and 1980 s. One computer scientist who saw this trend arising was Douglas Engelbart in the 1950 s. © 2003, Prentice-Hall 28

Dr. John Snow used a map to show the source of cholera was a water pump, thus proving the disease was water borne. Broad Street Pump © 2003, Prentice-Hall 29

Opportunity and Timing Alternative input devices (light pen, sketch pad and mouse) began to appear in the 1960 s. In the 1970 s, flight simulators became much more realistic when graphics replaced film. In the same decade, special effects computers became entrenched in the entertainment industry. In the 1980 s, visualization grew more dynamic with applications like the animation of Los Angeles smog patterns. © 2003, Prentice-Hall 30

One of today’s more useful types of visualization is in simulators (both in games and in practice). This is the only way most of us will ever fly a Boeing 747. © 2003, Prentice-Hall 31

It is now both cheaper and safer to train commercial pilots on simulators. With good software, pilots can be placed in situations they may not ever see – until too late – in the cockpit. © 2003, Prentice-Hall 32

A Sequence of Frames Animating LA Smog Day 2 Offshore Winds – Moderate Smog Particles Day 1 Swirling Winds – Light Smog Particles Day 3 Head-on View of Smog Particles and Streamlines © 2003, Prentice-Hall 33

Number Crunching With a Difference In the 1990 s, rapid advances in chip technology, both at the CPU and the graphics processor, put data visualization everywhere. Imagine trying to understand DNA sequences from just the numbers! On the next slide, a Mapuccino display helps us see where the results from a text search come from. © 2003, Prentice-Hall 34

© 2003, Prentice-Hall 35