AWIPS Continuous Technology Refresh CTR AWIPS Software CTR

AWIPS Continuous Technology Refresh (CTR) AWIPS Software CTR TO 9: Outbrief September 4, 2008 This document includes data that shall not be duplicated, used, or disclosed – in whole or in part – outside the Government for any purpose other than to the extent provided in contract DG 133 W-05 -CQ-1067. However, the Government shall have the right to duplicate, use, or disclose the data to the extent provided in the contract. This restriction does not limit the Government’s right to use information contained in this data if it is obtained from another source without restriction. The data subject to this restriction are contained in all sheets.

T 09 Outbrief Introduction Intro AWIPS-II T 09 Summary n AWIPS-II Architecture Update n AWIPS-II GFE Architecture n AWIPS-II Workstation (CAVE) Capabilities n AWIPS-II Hydro (Advanced Development) n AWIPS-II Stability Testing n AWIPS-II Architecture Evolution n AWIPS-II Demo Scenario n AWIPS-II T 09 Delivery Read. Me n 11/25/2020 Page 2

TO 9 AWIPS-II Task Order Second Migration Task Order with Emphasis on GFE n Summary Primary task is to migrate Graphic Forecast Editing (GFE) capability to the new AWIPS-II developed architecture – Performed GFE workstation capabilities migration – Performed server side migration to replace IFPServer – Text production generation infrastructure – Smart. Tool and Smart. Init interfaces Add data ingest capabilities for BUFR, Red. Book, and Text Warnings n Added python based derived parameter capability n Migrated Avn. FPS application into AWIPS-II n Performed additional risk reduction for Hydro n – Hydro View Perspective, SHEF Plug In with live data testing, limited time series display (Initial end-to-end shef ingest to IHFS to timeseries) n Limited T 08 DR fixes and numerous infrastructure improvements 11/25/2020 Page 3

T 09 Made Infrastructure Improvements n n n n Summary Added Python as u. Engine scripting language – Developed a JAVA bridge to Python’s “numpy” (Jepp+) – Linked to BLAS for high performance matrix operations Added Velocity Templates for automatic u. Engine script generation (enables both Javascript & Python) Performed 4 D data cube optimization – Improved Grid Tree Data Structure for performance improvements Improved Localization infrastructure – GFE uses localization dynamically which is a shift from AWIPS-I Improved CAVE alerting (notification) pattern with a new observer API Changed CAVE communication to HTTP from JMS for requests Added a HTTP data retrieve capability for light client operations (Note: IRAD prototype) 11/25/2020 Page 4

AWIPS-II: Development Sequence, T 09 is GFE+ 3 to 6 month fixed price task orders 2007 2006 2008 Today Summary 2009 2010 AWIPS II Development T 03 T 04 T 05 T 06 R&D Clean Sheet Start Architecture Development 2 Week Cycles Stability Testing Full Data Load T 08 Wx Workstation T 09 Forecast Editor T 010 Hydrology AWIPS-II Release 1 T 011 Comm Deployment Spiral Development with Interim Deliveries 11/25/2020 Page 5

AWIPS-II T 09 Services and Plug Ins ESB SOA Plug In Adaptable T 010 Improve Gfe. Purge. Srv Smart. Init Staging Ingest Hdf. Http Purging Index VTEC Archive Notify Adapter Utility Auto. Bld Product GFE py Proxy AWIPS-II Services Airep (22 total) Binlightening Bufrmos 801 Bufrmos Bufrua ccfp 268 Gfe 10808 Goessounding 547 Grib modelsounding 846 Obs Pirep poessounding 598 profiler 560 Radar Recco redbook 730 Satellite Sfcobs shef 3973 Taf Text 678 warning 1115 T 09 Architecture Future Eclipse RCP Visualization Plug Ins 14 Core 8 User Interface 22 Library 14 Application GFE 26944 Redbook 619 Python Editor 122 Hydro 10039 aviation(avn. FPS) 11539 CAVE Thick /Light Client Mode Enterprise Service Bus SOA Data Type Plug Ins 11/25/2020 Page 6

u. Engine Scripting Changed to Python Enables common scripting language, and numpy benefits u. Engine Automatic Script Generation Users IFPClient Plot. Model. Generator Architecture (i. e) Volume Browser Text Workstation Warn. Gen u. Engine Script Catalog. Query Script. Creator Py. Product. Srv Jep. Pool Script. Factory Jep execute. Script Velocity Template Engine Velocity Template Build Response 11/25/2020 Page 7

T 09 CAVE Eclipse Perspectives Enables Multiple User Interfaces with High Reuse Weather Workstation Hydrology Workstation Architecture Graphical Forecast Editing Python Editor • Perspectives leverages entire set of CAVE eclipse plug ins • Each perspective can have unique menus, toolbars, and dialogs • User can actively switch back and forth between perspectives 11/25/2020 Page 8

T 09 Alerting Pattern Update Architecture Added Observer Interface to enable multiple users CAVE Decision. Tree CAVE Startup EDEX Quinlan ID 3 Algorithm Ingest. Srv Alert. Job. Starter Menu data. URIs Data Item Build Data URICatalog Message Received Index. Srv Menu. Updater Data URI List ingested data. URIs Met. Watch. Monitor IAlert. Observer Avn. FPS quartz Timer Data URI Aggregator topic: edex. alerts 11/25/2020 Page 9

T 09 Data Cube Enhancements Redesigned data cube to enhance performance & flexibility HDF 5 Grid Repository Volume Browser Plan. View. Display Architecture Request. Job Loader Time. Matcher Browser. Dialog Browser. Data. Catalog <Job> Data. Cube. Manager Data. Cube. Container purge. Data. Cube refresh. Derived. Parameters examine. Attributes get. Data. Record perform. Time. Query List Grid. Tree set|get. Grid. Tree Dynamically loaded Python Derived. Parameter. Generator Derived Parameter Deriv. Param. Scripts. Loader 11/25/2020 Page 10

CAVE Localization Enhancements Uses Utility. Srv with a HTTP interface ESB Http Protocal architecture <<SOA Service>> Utility. Srv At CAVE Start Up & On Demand GFE Components that use localization CAVE Config Localization Manager Reference. Set. Manager load Sample. Set. Manager save Sample. Set. Manager Notify on updates Base Text. Formatter Smart. Tool. Manager Local Persistence Of Current Active Localization CAVE Menus Color Maps Localization Hierarchy Site User 11/25/2020 Page 11

GFE High Level Implementation Approach Plugins and Python with Java Bridge GFE Eclipse RCP Plug In Smart. Init Gfe. Purge. Srv Smart. Init. Srv GFE metadata HDF 5 Persistance EDEX Jepp+ Product. Srv Jepp+ Java Python Interface Extended to “numpy” Smart. Tools Text. Formatters GFE SOA Plug In Services Smart Init u. Engine Tasks Locking notification Dialogs (60) Perspective Actions (60) Query Grid Edit Tools Core Grid. Manager (51 classes) Interpolation Python Ed Display Resource Sampler Smart Tool Interface Text Formatter Launcher Color Bar Db access Data Types History Purging 307 Java Classes 26944 Lines 114 Java Classes 10808 Lines Metadata pattern extended to non-partitioned tables 11/25/2020 Page 12

T 09 GFE Perspective in CAVE GFE Menus & tool bar GFE Pydev Editor in Eclipse Perspective Grid Manager Implemented as Eclipse View Sample Sets as XML files. Managed by localization. Edit Areas stored as XML files. Managed by localization. Notification Pattern enables the updates Spatial Editor Leverages GIS rendering capability of CAVE 11/25/2020 Page 13

T 09 GFE Server (IFPServer reengineer) Use u. Engine by developing a set of GFE tasks Enables Mult. Threading Product. Srv u. Engine Gfe. Task Plug In Meta. Data Concept Extended to handle nonpartitioned tables Commit. Grid. Task Get. Parm. List. Task Get. Db. Inventory. Task Get. Site. Id. Task Get. Discrete. Definition. Task Get. Weather. Visibilites. Task Get. Grid. Data. Task Get. Wx. Definition. Task Get. Grid. History. Task Grid. Loc. Request. Task Get. Grid. Inventory. Task Request. Lock. Change. Task Get. Grid. Parm. Info. Task Save. Gfe. Grid. Task Get. Lock. Tables. Task Send. Notifications Get. Offical. Db. Name. Task Stage. D 2 DGrid. Data GFE HDF 5 Persistance Fcst Official Topo datauri Postgre. SQL Metadata Database Table: gfelocktable Table: gfegridhistory Extracted from AWIPS-I Server. Config. py Table: spatial_gfe 11/25/2020 Page 14

T 09 Text. Formatters: All AWIPS-I integrated Major Reeng Text. Formatter Minor Mods Grid 2 DBit. i 735 lines 162 lines IFPImporter. py AFPS. py C++ Python Interface(*. i): 66 Modules, 6767 Lines Grid 2 DBit. C GFE Replace AWIPS-I Interface 105 lines 3 lines AFPSsup. i Wx. Methods. py 57 imports 68 lines 380 lines Text. Formatter. py 1158 lines Site. Info. py AFPSDB. i C++ Modules 25 Modules 6847 Lines 35 imports 55 lines 789 lines Edit. Area. Utils. py 2702 lines Phrase. Builder. py 1198 lines 884 lines 173 lines Header. py Configurable. Issuance. py 178 lines 65 lines Vector. Related 954 Sample. Analysis. py Text. Rules. py Common. Utils. py Wx. Phrases 1906 2775 lines 65 lines Discrete. Phrases 2000 * 202 lines GUIs Vtec** Forecast. Narrative. py Generic. Report. py Process. Var. List. py Translator. py 4001 lines Config. Variables. py Table. Builder. py Time. Range. Utils. py 258 lines 310 lines Combined. Phrases. py Simple. Table. Utils. py Scalar. Phrases. py Text. Utils. py 30 lines Marine. Phrases 534 Fire. Phrases 905 677 lines GIS Interface 664 lines String. Utils. py 399 lines 273 lines 1618 lines Time. Descriptor. py Unit. Converter. py 716 lines 136 lines 127 lines NOW. py 40 Text 11 Harzard 65 lines 11/25/2020 Page 15

T 09 GFE notification design GFE Enables notifications with minimal impact on performance GFE Instance 1 SOA Data Plugin GFE IFPGrid. Database GFE Instance N Commit. Grid. Task Lock. Manager Eclipse RCP Plug In Grid. Manager. View Gfe. Purge. Srv Grid. Parm. Manager Grid. Manger. Visuals send. To. Observer <Eclipse Job> Notification. Manager msgs <Map> Listeners INotification. Observer add. Observer Gfe. Notification <JMS> topic: edex. alerts. gfe 11/25/2020 Page 16

T 09 GFE Parm Locking Design GFE Leverages RDBMS to improve lock management GFE u. Engine Tasks Get. Lock. Tables. Task Request. Lock. Change. Task Lock. Manager Lock. Database Postgre. SQL Metadata Database Lock Table: gfelocktable Lock. Notification <JMS> topic: edex. alerts. gfe 11/25/2020 Page 17

T 09 GFE Data Type Implementations GFE Capabilities of AWIPS-I data types ported into JAVA Abstract. Grid. Slice +valid. Time +grid. Parm. Info +grid. Data. History HDF 5 Persistance AWIPS-I Interpolation Functions For all data types ported to JAVA Fcst Official • Discrete Slice Reused for Wx Type Scalar. Grid. Slice +scalar. Grid 2 DFloat Vector. Grid. Slice +direction. Grid 2 DFloat Basic Type for Hazazrds Discrete. Grid. Slice +grid Grid 2 DByte +discete. Key[] Topo Discrete. Key +Discrete. Defintion +sub. Keys[<Discrete. Instances>] Discrete. Definition +mod. Attrs[] +modifiers[] +nouns[] Discrete. Instance +noun +discrete. Definition +mod. Map 11 Classes Used to Define Discrete 11/25/2020 Page 18

T 09 Warning Plug-In Enables GHGMonitor Leverages Warning Plug In Meta. Data to enable sharing SBN Ingest Endpoint. . /data/sbn/warning Replaces AWIPS-I VTEC “Active” Table EDEX SOA Plug In plugin-warning (1115 L) Extracts Metadata Warning. Decoder Warning. Separator GFE Vtec. Decoder GFE parse[SVS, SVRTOR, FFW, FLW, Warngen. FLW, Riverpro. FLW, FLS, Warngen. New. FLS, Warngen. Followup. FLS, Riverpro. FLS] build. Geometry build. Point. Str Warning. Record Extracts Warning Polygon out of Text Warnings Leverages GIS Capabilities of Post. Gre. SQL Table: warning_ugczone Meta. Data Table: warning 11/25/2020 Page 19

T 09 GHG Monitor Design Leverages Warning Plug In Meta. Data to enable sharing GFE • Common warning repository feeds both D 2 D and GHG. • VTEC elements are Stored as DB columns D 2 D Warning Rendering Meta. Data Table: warning 11/25/2020 Page 20

T 09 Redbook Graphics Capability Two New Plug-Ins (EDEX+CAVE) SBN Ingest Endpoint. . /data/sbn/redbook EDEX plugin-redbook (890 L) Extracts Metadata Decoder*** Message separating D 2 D CAVE. . /viz/redbook (652 L) Visualization resource Special Legend Handling Block parsing Rendering metadata . viz. ui. personalities. awips uri Redbook hdf 5 Persistance Menu XML Adding a Redbook item CAVE Bundle redbook. xml 11/25/2020 Page 21

T 09 Grid Derived Parameter Pattern New Pattern based on Python Scripts D 2 D Python Variables Set: Id, Name, Units Core Met. Lib FORTRAN ported to py Derived Parameters Python Scripts 11/25/2020 Page 22

T 09 Added Streamline Capability D 2 D Port D 2 D Fortran into AWIPS-II Meteo. Lib Right Click Legend Menu Fey <Java> Controller Meteolib strmpak. f <Eclipse Job> Contour. Manager 0 Dynamic Streamlining While pan and zoom Reuse CAVE Contouring Pattern Added D 2 D FORTRAN to Lib 11/25/2020 Page 23

Workstation Warning Generation T 09 Delivered T 08 Delivered Warn. Gen GUI D 2 D T 010+ Delivered Life Cycle Rules Dissemination VTEC Database tornado. vm severethunderstorm. vm Templates 3. . 19 GIS Capabilities Warning by Poly. Gon Text. DB UPDATE LIST Severe Weather Statement Store Set VTEC Generate Text Send to Text. WS Restart Text Workstation Editor 11/25/2020 Page 24

T 09 Hydro Advanced Development End to End Thread from SHEF ingest to Time. Series Hydro Eclipse RCP Plugin Dialogs (24) Time. Series (16 java classes) 101 Java Classes 10039 Lines . . /sbn/shef (ingest endpoint) EDEX IHFS Tables SHEF SOA Plug In Shef. Decoder Shef Posting to IHFS Shef Data Structures 22 Java Classes 3973 Lines Token Files 11/25/2020 Page 25

T 09 Integration Test Bed Cluster Metrics SEDA Load Balancing Performing Well Testing Cluster Balance Remaining Cluster Stability Issues are Network Related and are being worked 11/25/2020 Page 26

T 09 AWIPS-II Stability Run Results Build 15, 07 -Aug to 21 -Aug Average Data Ingest Profile Files/hr JVM 1. 6. 0_05 Stability Garage Collects to a stable level ~100 G Stable CPU Useage HDF 5 Persistance MBytes 11/25/2020 Page 27

T 010/11 AWIPS-II Infrastructure Improvements in Progress n n n Arch. Evolution Improve object serialization (Transition to JAXB and THRIFT) – Improve performance of request/response interactions – Improve persistence of binary objects to HDF 5 – Eliminates NFS for data access and enables “light client mode” Consider Migrating to a pure “C” Grib 1/2 Decoder from the Unidata One – GRIB 2 decoding using J 2000 is inadequate – Use GLUEGEN to generate JAVA interface to “C” decoder Upgrade ESB and improve ingest performance Improve HDF 5 locking under high transaction rates SOA Data Plug In Pattern upgraded to use “annotations” (JPA) for Meta. Data definitions – Eliminates Two XML files from plugin pattern, simplifies development Raytheon Investment: – Improve separation between EDEX core and Data plugins l Lowers maintance cost and facilitates partial deployment – Improve build process so that only selected set of plugins can be built and deployed – Improve separation of CAVE core plug ins from application level plug ins 11/25/2020 Page 28

T 09 Demo Scenario Features that illustrate the architecture n Demo D 2 D Features – Wind Streamlines (Zoom and Pan) – Redbook Graphics – Derived Parameters (Adding {i. e. Dp. D. py}, modifying) n GFE Features – Grid. Manager (Show eclipse “View. Part” features) – Running Smart Tools and editing, interpolation – Formatter Launcher (NOW product), Make Hazards n Hydro Perspective – Timeseries from ingested SHEF data 11/25/2020 Page 29

T 09 Delivery Read. Me n n If installing on Linux Fedora 6, the library installed by (yum install compat_lib 2 c-34) is required for Python module "numpy". If trying to run disconnected from a network, activemq multicast must be turned off. – Remove the XML element <transport. Connector name="discovery". . . from activemq. xml in. /activemq/conf n n n Ready for end to end testing – End-to-end testing will be available in T 011 T 09 Complete functionality – GFE: Spatial Editor, Color Legend Menus, Status Bar – GFE: Grid Manager, Interpolation – GFE: Edit Areas, Weather Element Browser, Data Pick Values – GFE: Pencil Tool, Sample Sets – GFE: Smart Tools, Smart Init – D 2 D: Streamline rendering for Wind parameters – D 2 D: Warn. Gen (Note: TRRs from delivery testing) Partial implementation – GFE: Formatter Launcher (Products: NOW, PNS, RFD, SPS, ESF) – GFE: GHG Monitor (Requires live warnings to be ingested) – GFE: Make. Hazard GUI (Run Button is in T 010) – D 2 D: Derived parameters (d. Z, RH, w. Sp, Dp. D, Dp. T, SHx, EPT, Po. T, mix. Rat, and TW) Note: Testing the adding of new derived parameters by just writting Python Scripts and adding them to the CAVE derived parameter folder – D 2 D: Profiler data, Model Sounding Data, Satellite Sounding, MOS, Redbook Graphics ingest – D 2 D: Redbook rendering (selected items under the NCEP/Hydro menu) – Avn. FPS: Ceiling Vis Dist, Configuration, Metars, Wind Rose GUI, TAFs, Weather Plot GUI 11/25/2020 Page 30

T 09 Delivery Read. Me n n n Advanced development – Hydro Perspective, Shef Ingest, Time Series (partial) Do not test – Anything that is comm related – GFE: ISC and Verify – GFE: Items under Hazard Menu except Make. Hazards GUI Issue: – HDF 5 locking performance in cluster for high transaction rates (fix identified and part of T 010) l Affects ingest of BUFR MOS 11/25/2020 Page 31