MATLAB and Scientific Data New Features and Capabilities

The Leading Environment for Technical Computing § § § Numeric computation Parallel computing, with multicore and multiprocessor support Data analysis and visualization Toolboxes for signal and image processing, statistics, optimization, symbolic math, and other areas Tools for application development and deployment 2

Go Farther with MATLAB and Toolboxes Statistics and Machine Learning Toolbox Database Toolbox Image Acquisition Toolbox Signal Processing Toolbox Mapping Toolbox Image Processing Toolbox MATLAB Compiler 3

MATLAB and Scientific Data § Scientific data formats • HDF 5, HDF 4, HDF-EOS 2 • Net. CDF (with OPe. NDAP!) • FITS, CDF, BIL, BIP, BSQ § Image file formats • TIFF, JPEG, HDR, PNG, JPEG 2000, and more § Vector data file formats • ESRI Shapefiles, KML, GPS and more § Raster data file formats • Geo. TIFF, NITF, USGS and SDTS DEM, NIMA DTED, and more § Web Map Service (WMS) 4

Scientific Data Libraries § § § MATLAB R 2015 a Library Version in MATLAB Vendor Version HDF 5 1. 8. 12 1. 8. 15 HDF 4 4. 2. 5 4. 2. 11 HDF-EOS 2 2. 17 2. 18 Net. CDF with OPe. NDAP 4. 1. 3 4. 3. 3. 1 CDF 3. 3. 0 3. 6. 0 FITS 3. 27 3. 37 Developing formal upgrade cadence to stay current with vendors Work closely with vendors on testing new versions 5

HDF 5 § High Level Interface (h 5 read, h 5 write, h 5 disp, h 5 info) h 5 disp('example. h 5', '/g 4/lat'); data = h 5 read('example. h 5', '/g 4/lat'); § Low Level Interface (Wraps HDF 5 C APIs) fid = H 5 F. open('example. h 5'); dset_id = H 5 D. open(fid, '/g 4/lat'); data = H 5 D. read(dset_id); H 5 D. close(dset_id); H 5 F. close(fid); 6

Net. CDF § High Level Interface (ncdisp, ncread, ncwrite, ncinfo) url = 'http: //oceanwatch. pifsc. noaa. gov/thredds/ dods. C/goes-poes/2 day'; ncdisp(url); data = ncread(url, 'sst'); § Low Level Interface (Wraps net. CDF C APIs) ncid = netcdf. open(url); varid = netcdf. inq. Var. ID(ncid, 'sst'); netcdf. get. Var(ncid, varid, 'double'); netcdf. close(ncid); 7

New in R 2014 b/R 2015 a § HDF 5 version 1. 8. 12! – Read data with a third-party filter applied – Both our high-level and low-level interfaces provide support § Dates and Times – datetime, duration, and calendar. Duration – Support for math, sorting, comparisons, plotting, formatted display, timezones § Big Data – mapreduce and datastore functions – table and categorical powerful in conjunction with big data analysis § RESTful web server access – webread, webwrite, and websave – JSON objects represented as struct arrays 8

Reading HDF 5 Data with Dynamically Loaded Filter § § MATLAB can easily read datasets with dynamically loaded compression filters Example using BZIP 2 compressor % Set the HDF 5_PLUGIN_PATH environment variable >> setenv('HDF 5_PLUGIN_PATH', '/test/BZIP 2 -plugin/plugins/lib'); % Read data with our high-level interface >> my. Data = h 5 read('h 5 ex_d_bzip 2. h 5', '/DS 1'); % Read data with our low-level interface >> file. Id = H 5 F. open('h 5 ex_d_bzip 2. h 5', 'H 5 F_ACC_RDONLY', 'H 5 P_DEFAULT'); >> dset = H 5 D. open(file. Id, '/DS 1', 'H 5 P_DEFAULT'); >> my. Data = H 5 D. read(dset, 'H 5 T_NATIVE_INT', 'H 5 S_ALL', 'H 5 P_DEFAULT'); >> H 5 D. close(dset); >> H 5 F. close(file. Id); 9

Date and Time Arrays § datetime for representing a point in time § duration, calendar. Duration for representing elapsed time § Same data type for computation and display – Add, subtract, sort, compare, and plot – Customize display formats – Nanosecond precision § Support for time zones – Accounts for daylight saving time 10

Automatic Updating of Datetime Tick Labels 11

Big Data Capabilities in MATLAB Memory and Data Access § 64 -bit processors § Memory Mapped Variables § Disk Variables § Databases § Datastores Programming Constructs § Streaming § Block Processing § Parallel-for loops § GPU Arrays § SPMD and Distributed Arrays § Map. Reduce Platforms § Desktop (Multicore, GPU) § Clusters § Cloud Computing (MDCS on EC 2) § Hadoop 12

Options for Handling Big Data Platform Desktop Only Desktop + Cluster Desktop + Hadoop Cluster MATLAB Desktop (Client) . . … … … …. . … Hadoop Scheduler Data Size 100’s MB -10’s GB 100’s MB -100’s GB – PBs Techniques • • parfor datastore mapreduce parfor distributed data spmd mapreduce 13

RESTful Web Service Access Read historical temperature data from the World Bank Climate Data API >> api = 'http: //climatedataapi. worldbank. org/climateweb/rest/v 1/'; >> url = [api 'country/cru/tas/year/USA']; >> S = webread(url) S = 112 x 1 struct array with fields: year data >> S(1) ans = year: 1901 data: 6. 6187 14

View and Save Lunar South Pole Color-coded Topography >> url = 'http: //planetarynames. wr. usgs. gov/images/moon_sp. jpg'; >> data = webread(url); >> imshow(data) >> >> >> filename = 'lunar. South. Pole. jpg' options = weboptions. Timeout = 10; options. Content. Type = 'image'; out. File = websave(filename, url, options) out. File = c: LibrariesDocumentslunar. South. Pole. jpg 15

Demo: Webread meets HDF Server § § § § HDF Server: A RESTful API providing remote access to HDF 5 data Responses are JSON formatted text webread with weboptions provide data access Example: Coral Reef Temperature Anomaly Database (Co. RTAD) Version 3 Co. RTAD products in HDF 5 format 1. 8 G dataset Running h 5 serv locally >> options = weboptions( 'Request. Method', 'get', 'Key. Name', 'host', 'Key. Value', 'cortadv 3_row 04_col 14. hdfgroup. org ') >> data = webread('http: //localhost: 5000/' , options) data = last. Modified: hrefs: root: created: '2015 -07 -10 T 00: 41: 43. 681844 Z' [5 x 1 struct] '6 f 60 d 9 c 0 -269 c-11 e 5 -aa 56 -005056 c 00008' '2015 -07 -10 T 00: 38: 58. 799031 Z' 16

Questions? § § § § www. mathworks. com/matlabcentral Examples: Using the high-level HDF 5 Functions to Import Data Tackling Big Data with MATLAB Performing Numerical Simulation of an Oil Spill Reading Content from RESTful Web Service Thank you! 17

References § § § § www. hdfgroup. org https: //www. hdfgroup. org/HDF 5/doc/Advanced/Dynamically. Loaded. Filters/HDF 5 Dynamically. Loaded. Filters. pdf https: //hdfgroup. org/wp/2015/04/hdf 5 -for-the-web-hdf-server/ http: //data. worldbank. org/developers/climate-data-api https: //data. nasa. gov/data http: //visibleearth. nasa. gov/ http: //www. nodc. noaa. gov/sog/cortad/ http: //data. nodc. noaa. gov/cgi-bin/iso? id=gov. noaa. nodc: 0068999 18