WDSSII Training Module II Manipulating Data Sources Data

WDSS-II Training Module II Manipulating Data Sources

Data files n Data are stored in files n n n Separate files for each elevation scan, table, Lat. Lon. Grid slice, contour, etc Each file corresponds to one time-stamp Separate files as the data streams in Usually organized by product name Usually netcdf or XML format Details in data format section

Data Sources n The data files are organized n Based on the sensor(s) it is from n n Called a data source Each data source has an “index” file n n n Radar, satellite, model, METAR, etc. Combined from these 6 radars Lists which data are available How to get a hold of the data Usually all the data are in one directory n n But may be distributed The index will however be in one “well-known” spot

Indices: XML versus XMLLB n All index records are in XML format in one of two file types: n XML LB files (code_index. lb): can have up to 32, 500 records; handle notification; binary file must be accesses with lb_cat, etc. n n algorithms that use an xmllb: as input will never exit! XML text files: user-editable, can have an unlimited number of records n Algorithms that use an xml: index as input will exit when they reach the end of the file.

Example of an index record <item> <time fractional="0. 0"> 799875952 </time> <params>netcdf {indexlocation} Velocity 00. 50 19950507 -194552. netcdf. gz</params> <selections>19950507 -194552 Velocity 00. 50 </selections> </item> More details in session on data format …

Creating an XML index file n For files that follow the WDSS-II naming convention, n n n make. Index. pl /my/input/dir code_index. xml (very fast) Directory should have data in the directory structure: Data. Type/Sub. Type/Time, like: “Reflectivity/00. 50/20030511 -123400. netcdf. gz” Otherwise, use “make. Netcdf. Index” (slow but sure) n n make. Netcdf. Index can be run in real-time Monitors a directory and updates index

Cropping the index n w 2 simulator is a useful tool for playing back data. We can also use it to “crop” out a time section from an index file and put it in another: n n w 2 simulator –i xmllb: /data/mydir/code_index. lb –l new_index. lb –b 20010228 -160000 –e 200102281630 –f Accessing code_index. lb will give you all the data. Access new_index. lb will give you a 30 -minute subsection.

Backing up real-time data n The easiest way, if you want all of the data: n n Just copy the entire directory of interest, including the code_index. lb file (cp –r <source dir> <target dir> ; or scp). Connect to the LB in the new directory. If the first record of the code_index. lb file is too late (index doesn’t list all data files), then create an XML index file.

Backing up real-time data n archive. Products. pl – to save a subset of sources or a time period from a real-time data stream. n n Must have access to the machine’s data disk Usage: archive. Products. pl <start date> <start hour> <end date> <end hour> <source directory> <target directory> <sources (“list in quotes” or “all”) archive. Products. pl 20021110 18 20021111 04 /data/realtime/radar /date/mytargetdir “KLSX KILX KEAX multi” Creates a new code_index. xml and code_index. lb for each data source directory (with make. Index. pl & w 2 simulator)

XMLLB to XMLLB n XML to XMLLB: n n w 2 simulator –i xml: /data/mydir/code_index. xml –l code_index. lb –f XMLLB to XML: n replace. Index –i xmllb: /data/mydir/code_index. lb –o /data/mydir/code_index. xml

Referencing Data Locations (URLs) n Indextype: machine: /directory/index (Optional) xml: or xmllb: n n Path to data Usually code_index. lb or code_index. xmllb: anubis: /data/realtime/radar/KTLX/co de_index. lb xml: /data/archive/multi/code_index. xml

End of WDSS-II Training Module II Next: Real-time system setup Archived data playback