Data Analytics using MATLAB and HDF 5 Ellen

Overview § MATLAB support for Scientific Data § Big Data and Data Analytics Workflows § Functions and datatypes for Data Analytics § Example: File. Datastore for HDF 5 data 2

MATLAB Support for Scientific Data § Scientific data formats • HDF 5, HDF 4, HDF-EOS 2 • Net. CDF (with OPe. NDAP!) • FITS, CDF, BIL, BIP, BSQ § Image file formats • TIFF, JPEG, HDR, PNG, JPEG 2000, and more § Vector data file formats • ESRI Shapefiles, KML, GPS and more § Raster data file formats • Geo. TIFF, NITF, USGS and SDTS DEM, NIMA DTED, and more § Web Map Service (WMS) 3

MATLAB Support for HDF 5 § High Level Interface (h 5 read, h 5 write, h 5 disp, h 5 info) h 5 disp('example. h 5', '/g 4/lat'); data = h 5 read('example. h 5', '/g 4/lat'); § Low Level Interface (Wraps HDF 5 C APIs) fid = H 5 F. open('example. h 5'); dset_id = H 5 D. open(fid, '/g 4/lat'); data = H 5 D. read(dset_id); H 5 D. close(dset_id); H 5 F. close(fid); 4

MATLAB Support for net. CDF including OPe. NDAP § High Level Interface (ncdisp, ncread, ncwrite, ncinfo) url = 'http: //oceanwatch. pifsc. noaa. gov/thredds/ dods. C/goes-poes/2 day'; ncdisp(url); data = ncread(url, 'sst'); § Low Level Interface (Wraps net. CDF C APIs) ncid = netcdf. open(url); varid = netcdf. inq. Var. ID(ncid, 'sst'); netcdf. get. Var(ncid, varid, 'double'); netcdf. close(ncid); 5

Big Data and Data Analytics: Why MATLAB? Data Analytics 1 MATLAB Analytics work with business, scientific, engineering data DATA • Engineering, Scientific, and Field • Business and Transactional 3 MATLAB Analytics run in embedded systems developed with Model-Based Design Embedded Systems Developed with Model-Based Design 2 MATLAB lets domain experts do Data Science themselves MATLAB Analytics deploy to enterprise IT systems Enterprise IT Systems 4 6

Big Data Workflows in MATLAB ACCESS Access data and collections of files that do not fit in memory Datastores • Images • Spreadsheets PROCESS AND ANALYZE Purpose-built capabilities for domain experts to work with big data locally Tall Arrays • Math • Statistics GPU Arrays • Matrix Math • Visualization • Machine Learning • Image Processing Deep Learning • Image Classification • Tabular Text • Custom Files • SQL • Hadoop (HDFS) SCALE Scale to compute clusters and Hadoop/Spark for data stored in HDFS Tall Arrays • Math, Stats, Machine Learning on Spark Distributed Arrays • Matrix Math on Compute Clusters MDCS for EC 2 • Cloud-based Compute Cluster Map. Reduce MATLAB API for Spark 7

Data Analytics Workflows in MATLAB Access and Explore Data Preprocess Data Develop Predictive Models Integrate Analytics with Systems Files Working with Messy Data Model Creation e. g. Machine Learning Desktop Apps Databases Data Reduction/ Transformation Parameter Optimization Enterprise Scale Systems Sensors Feature Extraction Model Validation Embedded Devices and Hardware 8

Today’s Focus: Accessing, Exploring, Preprocessing Data Access and Explore Data Files Preprocess Data Business and Transactional Data § Repositories – SQL, No. SQL, etc. § File I/O – Text, Spreadsheet, etc. § Web Sources – RESTful, JSON, etc. Databases Engineering, Scientific and Field Data § Real-Time Sources – Sensors, GPS, etc. § File I/O – Image, Scientific Data Formats, Video, Audio, etc. . Sensors § Communication Protocols – OPC (OLE for Working with Messy Data Reduction/ Transformation Feature Extraction Process Control), CAN (Controller Area Network), etc. 9

What is a datastore? Serial PCT Local Workers MDCS MATLAB Compiler 10

Access Big Data through datastore § Datastore: easily access large sets of data – Object designed for accessing data – Preview data structure and format – Variety of types for different data sources: § Tabular. Text Datastore Spreadsheet Datastore Database Datastore Key. Value Datastore File Datastore § Image Datastore § § – Incrementally read portions of the data – Use with Parallel Computing tools 11

When to Use datastore § Data Characteristics – Data stored in files supported by datastore § Compute Platform – Desktop or cluster § Analysis Characteristics – Supports Load, Analyze, Discard workflows – Incrementally read chunks of data, process within a while loop 12

Example datastore code 1 2 3 4 5 6 7 8 9 10 11 ds = tabular. Text. Datastore('c: airlinedata*. csv'); max. Delay = 0; while hasdata(ds) data = read(ds); chunkmax = max(data. Departure. Delay); max. Delay = max(max. Delay, chunkmax); end % or use tall! ds = tabular. Text. Datastore('c: airlinedata*. csv'); t = tall(ds); max. Delay = gather(max(t. Departure. Delay)); 13

Datastores – the Key to Tall Arrays Databases … Images ds = datastore(…) T = tall(ds) Custom ds = datastore('s 3: //…', …) 14

What are Tall Arrays? tall data type introduced in Ideal for tabular/columnar data One or more rows can fit into memory Overall data size is too big to fit into memory Access Data • • • Text Spreadsheet (Excel) Database (SQL) Images Custom Reader Simulink Tall Data Types • • Table Timetable Cell Numeric Dates & times String Categorical Cellstr Machine Learning Preprocessing • • • Numeric functions Summary statistics String processing Table wrangling Missing data handling Visualizations: • Plot, scatter • Histogram/histogram 2 • Kernel density plot • Bin-scatter • • Linear Models Logistic Regression Discriminant analysis Classification Trees SVM K-means PCA Random data sampling “Tall” data types and functions for use with out-of-memory data 15

Execution Environments for Tall Arrays Local disk, Shared folders, Databases Run on Compute Clusters or Spark + Hadoop (HDFS), for large scale analysis Process out-of-memory data on your Desktop to explore, analyze, gain insights and to develop analytics Use Parallel Computing Toolbox for increased performance MATLAB Distributed Computing Server, Spark+Hadoop 16

Example: Working with HDF 5 data using File. Datastore § NASA’s Operation Ice. Bridge Aircraft Missions – Reference: https: //nsidc. org/data/icebridge/campaign_data_summary. html – Airborne Topographic Mapper LIDAR – Measures changes in ice surface elevation § Let’s look at the Antarctica Larsen D Ice Sheet datasets – Larsen D data collected on 10/18/14 and 11/18/2016 § Create a File. Datastore with a custom file reader – Read through the collections of files – Gather information on the datasets 17

Example: Working with HDF 5 data using File. Datastore § Create a File. Datastore ds = file. Datastore(h 5 Folder, 'Read. Fcn', @h 5 readall); § Scale to Map. Reduce – Map function receives chunks of data and outputs intermediate results – Reducefunction reads the intermediate results and produces a final result mapreducer(0); mr. Output. Folder = fullfile(pwd, 'output'); outds = mapreduce(ds, @count. Map, @count. Reduce, 'Output. Folder', 'output'); 18

Example: Working with HDF 5 data using File. Datastore § Read and view the computed data tbl = readall(outds); out. Table = horzcat(tbl. Key, struct 2 table([tbl. Value{: }])); out. Table. Properties. Variable. Names{1} = 'Filename‘ >> file. Datastore. Demo **************** * MAPREDUCE PROGRESS * **************** Map 0% Reduce 0% Map 10% Reduce 0% Map 21% Reduce 0% Map 31% Reduce 0% Map 42% Reduce 0% Map 53% Reduce 0% Map 63% Reduce 0% Filename Number. Of. Datasets File. Size Error. Datasets ___________________________________________________________________ _____________ '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_162307. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_162801. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_163343. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_163935. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_164516. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_165055. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_165637. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_170223. ATM 6 AT 6. h 5' '\mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_170810. ATM 6 AT 6. h 5' ‘ \mathworkshomeellenjice. Sheetn 5 eil 01 u. ecs. nsidc. orgICEBRIDGEILATM 1 B. 0022016. 11. 18h 5 FilesILATM 1 B_20161118_171357. ATM 6 AT 6. h 5' 19 19 19 1. 3913 e+07 1. 5699 e+07 1. 6593 e+07 1. 4693 e+07 1. 5862 e+07 1. 6317 e+07 1. 6681 e+07 1. 6438 e+07 1. 6231 e+07 1. 6502 e+07 0 0 0 0 0 19

Saving Preprocessed/Intermediate Data – MAT-Files § Saving preprocessed or intermediate results § In MATLAB, many people use. mat files for this § Binary MATLAB files that store workspace variables § MAT-File version 7. 3 are based on the HDF 5 file format! 20

Thank you! Questions? 21