Writing a WDSSII Application using w 2 algcreator

Writing a WDSS-II Application using w 2 algcreator V Lakshmanan National Severe Storms Laboratory & University of Oklahoma lakshman@ou. edu 9/26/2020 http: //www. wdssii. org/ 1

WDSS-II – application development l WDSS-II provides a easy environment to develop new applications or algorithms. l Any algorithm developed in WDSS-II can be run either in archive mode or in realtime. l 9/26/2020 provided that the algorithm is fast enough to run in real-time and the machine can handle it! http: //www. wdssii. org/ 2

WDSS-II for Applications l What does WDSS-II provide your application? Easy ingest of data from multiple sensors l Easy ways to write out and visualize intermediate and final outputs. l A common framework for accessing data. l The framework is written in C++. l It’s best if you can write your code in C++ too. l But you can call Fortran or C code from C++ l 9/26/2020 http: //www. wdssii. org/ 3

Creating a new application l WDSS-II has a tool to create a new application quickly. l w 2 algcreator Need to create an input XML file specifying the inputs to your application. l Will create C++ main and a template for your scientific code. l l 9/26/2020 Your then write the scientific code. http: //www. wdssii. org/ 4

How WDSS-II works l WDSS-II applications connect to a source of data. l l Called an Index. Applications listen for new products in that Index. l l Decide whether or not to process that product. Create and process the data if needed. Write outputs (in netcdf or XML). Notify an Index about the new products available. l l 9/26/2020 For other applications listening to that Index May be a different index from the source. http: //www. wdssii. org/ 5

WDSS-II Applications l WDSS-II Applications are just executables. l launched on the command line. In deployed systems through scripts. Inputs are specified on the command-line. w 2 vil –i xmllb: /data/realtime/radar/KTLX/code_index. lb -R Reflectivity. QC -o /data/realtime/radar/KTLX/ -r 9/26/2020 http: //www. wdssii. org/ 6

Command-line options l l l l w 2 vil – algorithm executable name Vertical. Integrated. Liquid – algorithm full name (used in source code) Input Radar. Index specified by –i option Input data type is specified by –R option Output directory is specified by –o option Real-time is specified by –r option The options –r, -l (the letter ell), -o are reserved so that the deployment scripts work correctly. w 2 vil –i xmllb: /data/realtime/radar/KTLX/code_index. lb -R Reflectivity. QC -o /data/realtime/radar/KTLX/ -r 9/26/2020 http: //www. wdssii. org/ 7

Some conventions l If you need to provide domain extents, use the following letters: l l t: “top” to specify the north-west-top corner b: “bottom” to specify the south-east-bottom corner s: to specify the grid spacing If you need to process only a certain sub-type, use colons as a separator: l l 9/26/2020 -R Reflectivity. QC: 00. 50 -R Reflectivity. Maximum: vol http: //www. wdssii. org/ 8

Creating a new application l Need to create an XML file. l l l What is the algorithm’s descriptive name? l l l Use your favorite text editor. Save it somewhere. A template is available as example_alg. xml – you can edit this if you want. Used to generate class names in template. What is the executable name? What is the namespace that generated code should be in? Pick your group (casa) or name (tj) i. e. something that will distinguish your code from something someone else might create. <algorithmcreator name=“Vertical. Integrated. Liquid exec=“w 2 vil” namespace=“w 2 polar”> </algorithmcreator> 9/26/2020 http: //www. wdssii. org/ 9

Inputs l Next you need to specify inputs. l l Done within an “inputs” section. How many indexes do you need to connect to? l Each index or group of indexes will get a new index tag. l l Specify the history (in minutes) that needs to be maintained for each index (or group of indexes). l 9/26/2020 Give each index a name (Radar. Index, Model. Index, etc. ) so that a user can understand what needs to be provided to your algorithm. Supply a mnemonic option letter as well. The most important input index is by convention given the option letter i. Can be just 1 minute if you will never search backwards in your algorithm (only new data). http: //www. wdssii. org/ 10

Input Index specification <algorithmcreator …. > <inputs> <index history=“ 5”> <optionvalue name=“Radar. Indexes” letter=“i” /> </index> <index … > …. </index> </inputs> </algorithmcreator> 9/26/2020 http: //www. wdssii. org/ 11

Input Data specification The user may specify either just a single index or a bunch of index URLs. So each “index” tag actually corresponds to a group of indexes in general. l For each index, specify the data that you will be listening to in that index. l For each product listened to, specify: l l l How you will refer to the product (e. g: dbz) The default name of the product (e. g: Reflectivity. QC) l l l A mnemonic letter for this input What type of data is it? WDSS-II data are usually one of these types: l l 9/26/2020 This is the “official” name of the product that you can see on the display. Radial. Set Lat. Lon. Grid Lat. Lon. Height. Grid Data. Table http: //www. wdssii. org/ 12

Input Data Specification <index history=“ 5”> <optionvalue …. /> <data type=“Radial. Set”> <optionvalue name=“dbz” letter=“R” default=“Reflectivity. QC” /> </data> <data … > … </data> </index> 9/26/2020 http: //www. wdssii. org/ 13

Option l The optionvalue tag: Has a letter, name and default l The default tag is optional – if you leave it out the user has to specify a value on the command-line. l If the default is “false”, then the option is a boolean option i. e. yes/no. l 9/26/2020 http: //www. wdssii. org/ 14

Optional data l To specify an optional index, specify that the default value is an empy string. l E. g. you will use a model index if one is available, otherwise, you will run only on radar data. l You need to specify a default name for all input data types l 9/26/2020 otherwise we won’t know what to listen for. http: //www. wdssii. org/ 15

Other options l Besides the input, do you need any other parameters specified for your algorithm? l l For example, maybe you need to know the radar name. For each such option: l l 9/26/2020 Specify the name of the option (e. g: Radar. Name) Specify a mnemonic (e. g: ‘S’ for source. Radar, since –r and –R are both taken by realtime and Reflectivity respectively …) http: //www. wdssii. org/ 16

Other options <algorithmcreator …> <options> <optionvalue letter=“s” name=“source. Radar” /> <optionvalue …/> </options> </algorithmcreator> 9/26/2020 http: //www. wdssii. org/ 17

Use the w 2 algcreator l Now that you have the XML file: l Run the w 2 algcreator program l It will create a w 2 vil_main. cc and w 2 vil_Vertical. Integrated. Liquid. h file in the output directory. l Plug in the code for the two functions specified in the. h file: l l One function tells you the radar name, the output directory to write your outputs to and the LB to notify after you have written the outputs. The other function is called processdbz() l Your scientific code goes in there. w 2 algcreator –i name-of-your-XML-file -o output-directory 9/26/2020 http: //www. wdssii. org/ 18

Final steps l Fill in the scientific code. l Compiling: Use the example_Makefile that is provided to compile and link against the WDSS-II libraries. l Or put your source code into a subdirectory of the w 2 algs repository l Need to create Makefile. am l See w 2 algs/w 2 algcreator/README for details. l 9/26/2020 http: //www. wdssii. org/ 19

The autogenerated code l What does the autogenerated code do? l l It creates “listeners” to listen for the products your algorithm needs The user tells the algorithm the Index where those products may be found l l l The code has fault handling l l If you lose connection to an Index on a remote machine, the algorithm will reconnect. If your algorithm hangs or starts to eat up the CPU, it’ll stop providing heartbeat messages, thus allowing an external monitoring program to kill it and restart it. A good idea to specify “init. On. Start=true” for any product that should reread if your algorithm is restarted. As soon as it receives notification of a product, it tells your algorithm class about it l 9/26/2020 Also specifies that Reflectivity. QC_smoothed should be used where your algorithm expects a reflectivity May tell your algorithm to provide heartbeat messages Then, you do your thing! http: //www. wdssii. org/ 20

On a timer l W 2 algcreator is for data-driven applications l l l Best to process data as it comes in So, design your application to be data-driven as much as possible. To do things on a timer l l 9/26/2020 Use a Timed. Event. Handler (see API documentation) Provide a listener that will be called every N milliseconds. http: //www. wdssii. org/ 21

Exercises l The best way to learn is to write simple data-driven algorithms. l l l The next module on data formats will help you accomplish these tasks. May need to get input data from some source first. These examples should get you started: l Read in single-radar reflectivity data and write out an indexed scan with values below a user-defined threshold set to Missing. Data. l l Read in both reflectivity and velocity data from the same radar and index the reflectivity data at the velocity azimuths. l l l You will need to operate with Radial. Set data Wait for matching Reflectivity and Velocity scans (use the elevation angle to check) Form a histogram of VIL values to satellite IR temperatures. l l 9/26/2020 Hint: Polar. Grid is an indexed Radial. Set User-defined implies you need a command-line parameter Use Process. Timer to determine how long certain steps take. Use Data. Converter and/or Data. Remapper A multi-source algorithm. Makes a difference only when running, not when writing the algorithm! http: //www. wdssii. org/ 22

Questions? Email me: lakshman@ou. edu More documentation (including a more upto-date version of this document) is at: http: //www. wdssii. org/ 9/26/2020 http: //www. wdssii. org/ 23