Data Preprocessing Data Cleaning An Overview Data Integration

  • Slides: 10
Download presentation
Data Preprocessing Data Cleaning An Overview Data Integration Data Reduction By Sandeep Patil, Department

Data Preprocessing Data Cleaning An Overview Data Integration Data Reduction By Sandeep Patil, Department of Computer Engineering, I²IT Data Transformation

Outline Ø What is Data Preprocessing ? Ø Major Steps in Data Preprocessing v

Outline Ø What is Data Preprocessing ? Ø Major Steps in Data Preprocessing v v Data Cleaning Data Integration Data Reduction Data Transformation and Data Discretization Ø Conclusion International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Why Data Preprocessing ? • Need of data preprocessing Some part of Data may

Why Data Preprocessing ? • Need of data preprocessing Some part of Data may have problems like – Incomplete (absence of data) – Inaccurate or noisy (other than expected values) – Inconsistent (containing discrepancies) – Timeliness (old version of data) – Believability (users faith in the correctness of the data) – Interpretability (simplicity in understanding the data) International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Major Steps in Data Preprocessing • Data Cleaning • Data Integration • Data Reduction

Major Steps in Data Preprocessing • Data Cleaning • Data Integration • Data Reduction • Data Transformation International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Data Cleaning • • • Filling Missing values Smoothing Remove Noisy data Identifying or

Data Cleaning • • • Filling Missing values Smoothing Remove Noisy data Identifying or removing outliers Resolving inconsistencies. International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Data Integration • Entity Identification Problem - Integrating multiple databases, data cubes, or files

Data Integration • Entity Identification Problem - Integrating multiple databases, data cubes, or files • Redundancy and Correlation Analysis • Tuple Duplication - updating some but not all data occurrences. • Data Value Conflict Detection and Resolution - for the same real-world entity, attribute values from different sources may differ International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Data Reduction • To obtain a reduced representation of the data set that is

Data Reduction • To obtain a reduced representation of the data set that is much smaller in volume • Numerosity Reduction - Parametric methods eg. Regression and log-linear models etc. - Nonparametric methods eg. Histograms, clustering, sampling etc. • Data Compression - lossless - lossy International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Data Transformation and Data Discretization • Data are transformed or consolidated into forms appropriate

Data Transformation and Data Discretization • Data are transformed or consolidated into forms appropriate for mining - Smoothing - Attribute construction or feature construction - Aggregation, - Normalization - Discretization - Concept hierarchy generation International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

Conclusion Although numerous methods of data preprocessing have been developed, data preprocessing remains an

Conclusion Although numerous methods of data preprocessing have been developed, data preprocessing remains an active area of research, due to the huge amount of inconsistent or dirty data and the complexity of the problem. International Institute of Information Technology, I²IT, P-14 Rajiv Gandhi Infotech Park, MIDC Phase 1, Hinjawadi, Pune - 411 057 | Tel +91 20 22933441 | Website - www. isquareit. edu. in | Email - info@isquareit. edu. in

THANK YOU For further information please contact Prof. Sandeep Patil Department of Computer Engineering

THANK YOU For further information please contact Prof. Sandeep Patil Department of Computer Engineering Hope Foundation’s International Institute of Information Technology, I²IT Hinjawadi, Pune – 411 057 Phone - +91 20 22933441 www. isquareit. edu. in | sandeepp@isquareit. edu. in

Data Preprocessing Agenda Why data preprocessing Data cleaningData Preprocessing Agenda Why data preprocessing Data cleaning
Data Preprocessing n Data cleaning n Data integrationData Preprocessing n Data cleaning n Data integration
Brewing Science Cleaning Brewery Cleaning and Sanitation CleaningBrewing Science Cleaning Brewery Cleaning and Sanitation Cleaning
Data Preprocessing Lecture 12 Overview of data preprocessingData Preprocessing Lecture 12 Overview of data preprocessing
Data preprocessing Data Preprocessing Mengapa data di prosesData preprocessing Data Preprocessing Mengapa data di proses
Data Preprocessing Data Preprocessing p Why preprocess theData Preprocessing Data Preprocessing p Why preprocess the
Data Preprocessing 1 DWDM Data Preprocessing February 21Data Preprocessing 1 DWDM Data Preprocessing February 21
Data Preprocessing 1 The need of data preprocessingData Preprocessing 1 The need of data preprocessing
UNIT 1 Data Preprocessing Data Preprocessing Learning ObjectivesUNIT 1 Data Preprocessing Data Preprocessing Learning Objectives