Engineering Reports Day 1 November 2017 EN Reports
Engineering Reports Day 1 November 2017
EN Reports • Day 1 – Build 8 a Release – Tool Working Group • Day 2 – Digital Object Identifiers – PDS Home Page Discussion – Search Discussion – 2018 Tech Session
Build 8 a Release November 2017
Executive Summary • Information Model V 1. 9 and core software services and tools were delivered to I&T on September 29; SR & DPH documents posted for V 1. 9 • I&T Completed on 10/23/17 • Deployment to EN Completed on 11/10/17 • V 1. 9 and software available to nodes, IPDA, and public
Build 8 a I&T Process SCRs (approved by CCB) IM I&T IM V 1. 9. 0. 0 candidate IM V 1. 9. 0. 0 Issues (in JIRA) Build 8 a Software Build 8 a System I&T Build 8 a Test Plan (test cases) Build 8 a Test Report Test Data
Build 8 a Artifacts
I&T Input/Output • Input – IM V 1. 9. 0. 0 as a result of Node testing of candidate IM based on approved SCRs by CCB – Build 8 a system software including upgrades and fixes of issues captured in JIRA – Build 8 a test plan with modified test cases for new features and regression test – Build 8 a test data • Output – New issues found have been added to JIRA – Fixed issues were closed in JIRA – Test result has been documented in the test report here (see next page)
Updated Test Cases • PRP. 1 – Added transformations to csv and PDS 3 files • PRV. 1 - Replaced test cases that used obsoleted schemas • PRV. 7 - Added steps to test more of validate -R's functionality • SRCH. 3 – Added steps to test search-analytics • HVT. 1 – Added steps to test harvest-search
I&T Results Total of 51 test cases exercised and passed, no failure. 97 requirements met. 0 major issues found, no show stopper. 3 minor issues and 3 suggested improvements created in JIRA. All 3 minor issues were fixed and regression tested during I&T. • Test report and data posted at: • • https: //pds-engineering. jpl. nasa. gov/content/build_8 a_deliverables
Details of Information Model Changes 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. CCB-162 - Move <md 5_checksum> from Object_Statistics to Byte_Stream CCB-165 - Ambiguity of ASCII_Numeric_Base CCB-166 - Deprecate bit_mask from IM CCB-171 - Split hardware-compatible ASCII numeric types from the unbounded CCB-172 - Add movie format(s) to list of acceptable documents. CCB-176 - Update Node names in the IM CCB-178 - Internal Reference Implementation CCB-179 - In PDS 4 'pds' namespace, use ASCII_Non. Negative_Integer CCB-184 - The xs: choice element does not validate as expected CCB-185 - Allow multiple instances of purpose and processing_level CCB-191 - Add Indian Space Research Organisation (ISRO) to the Namespace Registry CCB-192 - Add Product_Metadata_Supplemental (M. Gordon) CCB-193 - Add Geosciences and Radio Science as permissible values SCRs that did not impact the IM 1. 2. 3. 4. CCB-180 - Clarify whether namespace abbreviation in labels must correspond to namespace_id CCB-181 - Clarify whether newlines are allowed in PDS DSV 1 CCB-182 - Clarify whether CDATA is allowed CCB-183 - Rejected
PDS 4 Information Model Discipline and Mission Dictionaries
Build 8 a Software Components Tools • Ingest – Harvest Tool – Catalog Tool • Preparation – – – Generate Tool Transform Tool Validate Tool Core and PDS 4 Tools Libraries Inspect Tool (Prototype) Services • Registry and Search – Includes support for Tool and Data Dictionary content • Transport – Backward compatible with the PDS-D Product Server • Report – Generating monthly HQ reports
Major Software Changes • Validate Tool – Added support for data content validation of tables against the label description • Transform Tool – Added support for transforming PDS 3 labeled tables to CSV – Added support for transforming PDS 4 label to PDS 3 label (ODL) • Inspect Tool – Initial release supporting visualization of PDS 4 products • Tracking Service – Initial release supporting API access to tracking services. • Search Service – Added performance improvement for index generation.
Deployment • Build 8 a has been successfully deployed at the EN early November. • V 1. 9 and software available to nodes, IPDA, and public
Backup
Summary of Information Model Changes • Common dictionary – Implemented 13 approved SCRs – – – Detailed list in backup. CCB-184 requires the regeneration of LDDs (LDDTool) CCB-178 further restricts the values for <local_identifiers> Cartography LDD ingested Ingested Rings LDD updated • Files exported from IM for tools and services. – Added “abstract” and “deprecated” flags to the JSON file • Local Data Dictionary (LDD) changes. – – – Ingested new version of Rings LDD - V 1501_1800 Spectral LDD released – V 1800 Spectral Library LDD released – V 1000
Software Change Detail • New Capabilities – [PDS-243] - Generate: Add support to use Product Tools library – [PDS-495] - PDS 4 -Tools: Add URL support[PDS-506] - Transform: Add support for transforming PDS 3 tables to CSV – [PDS-507] - Transform: Add support for transforming a PDS 4 label to a PDS 3 label – [PDS-508] - Validate: Add support for validating data content of table products – [PDS-517] - Tracking: Develop a service-based interface for tracking product status • Modified Capabilities – [PDS-504] - Validate: Improve Operation guide examples[PDS-505] - Generate/Transform: Update PDS 4 label transformation for the latest model – [PDS-509] - Search: Separate the protocol implementation into a separate component – [PDS-511] - Transform: Add support for reading targets from URLs – [PDS-512] - Transform: Integrate the tool with the Tools Service – [PDS-519] - Generate: The tool outputs an unexpected warning – [PDS-521] - Validate: The -R pds 4. bundle option cancels -e "*. xml" • Corrected Capabilities – [PDS-467] - Core: Support passing in Schema jar URLs – [PDS-478] - Search: XSLT variable resolution is not working on the Windows platform – [PDS-501] - Core: Support bundle. xml and collection. xml file names to trigger their respective rule types – [PDS-515] - Validate: Handshake failure occurs when trying to validate against schemas located on the PDS machines – [PDS-523] - Transform: transformation of character table product to PDS 3 label fails
I&T Improvements Found open PDS-518 improve MOD. 1 Unexpected error messages closed PDS-519 improve PRG. 1 Unexpected warning messages open PDS-520 minor PRV. 1 Handle “+” in format closed PDS-521 minor PRV. 1 Option -R pds 4. bundle negates -e "*. xml" open PRV. 2 Less brutal error message for missing file PRT. 1 Transform fails on table_character to pds 3 -label PDS-522 improve closed PDS-523 minor
PDS Tool Working Group (PDS-TWG) November 2017
Agenda • • Status Overview Requirements Tool Activities Tool requirements and release plans – Inspection – Transformation – Validation • Open Source • Container Deployments • Next Steps November 29, 2017 Tool Working Group 20
Status Overview • Since last MC F 2 F, the working group has focused on: – Final requirements and priorities for inspection – Discussion of Node Tool – Exploration of deployment options – Discussion of open source approach • Continuing to populate the Tool Registry November 29, 2017 Tool Working Group 21
Requirements • Updated Level 1/2/3 requirements were approved April 2017 by MC and are posted • Individual requirements for validation, transformation, and inspection have been prioritized by the TWG – Mapped to builds November 29, 2017 Tool Working Group 22
Tool Activities • The team is tracking status for actively developed tools at the EN and the DNs • A traceability matrix is maintained to map Level 3 requirements to tools • EN has been discussing core tools • Nodes were asked to begin presenting DN tools • Nodes were asked to identify tools beyond the list (ongoing request) • This is facilitated with the following Google doc: – https: //docs. google. com/spreadsheets/d/18 oqtg 3 DEo 2 Krgv BOWLSOuq. F 2 u. Ztq 2 Xm. By. Jw. Ukn. YSZUQ/edit#gid=134066079 9 November 29, 2017 Tool Working Group 23
PDS Maintained Tool Status November 29, 2017 Tool Working Group 24
Inspection • The TWG has been discussing the inspection tool • The Inspect Tool will provides support for visualizing PDS 4 products – Eventually planned to replace NASAView • Requirements have been reviewed by the TWG – Comments are being incorporated into an updated version of the document • This was facilitated with the following Google doc: – https: //docs. google. com/spreadsheets/d/18 oqtg 3 DEo 2 Krgv. BOWL SOuq. F 2 u. Ztq 2 Xm. By. Jw. Ukn. YSZUQ/edit#gid=807742626 – Summary shown on backup slide • An initial version has been included in Build 8 a and will be tested by the TWG November 29, 2017 Tool Working Group 25
Transformation • Today, the Transform Tools supports approximately 29 transformations as of the last Build release (8 a) • The team was tasked with adding to and prioritizing the list of supported transformations for the Transform Tool • This was facilitated with the following Google doc: – https: //docs. google. com/spreadsheets/d/18 oqtg 3 DEo 2 Krgv. BOWLSOuq. F 2 u. Ztq 2 Xm. By. Jw. Ukn. YSZUQ/edit#gid=63 5102082 – Summary shown on backup slide 20 November 29, 2017 Tool Working Group 26
Transformation Releases Plan • The prioritization exercise identified the following as higher priority (assigned to builds): – Build 8 a (just released) • PDS 3 labeled tables to CSV • PDS 4 label to PDS 3 label (ODL) – Build 8 b • PDS 3 labeled tables to PDS 4 labeled tables • PDS 4 labeled tables to PDS 4 labeled tables (e. g. , binary to character) – Build 9 a • Array 2 D Map to Geo. TIFF • Array 3 D Spectrum to ENVI Cube • Array 2 D Image to FITS November 29, 2017 Tool Working Group 27
Validation • The Validate Tool supports the following as of the Build 8 a release: – Syntactic and semantic validation via the XML Schema and 350+ Schematron rules pertaining to PDS 4 label structure and content – Bundle and Collection referential integrity checking – Data content validation of tables against the label description • The team was tasked with prioritizing the validation requirements – These focus primarily on specific content validation rules, many of which need to be individually implemented and scheduled for release • This was facilitated with the following Google doc: – https: //docs. google. com/spreadsheets/d/18 oqtg 3 DEo 2 Krgv. BOW LSOuq. F 2 u. Ztq 2 Xm. By. Jw. Ukn. YSZUQ/edit#gid=1268101288 – Summary shown on backup slide 21 -23 November 29, 2017 Tool Working Group 28
Validation Release Plan • Upcoming priorities (from TWG discussions): – Build 8 b • Extended support for XML Catalog • Data content validation of arrays against the label description – Build 9 a • • Verify complex elements confrom to naming rules Verify simple elements confrom to naming rules Verify file names conform to naming rules Verify schema location accessibility November 29, 2017 Tool Working Group 29
PDS Validate Tool Support Validation Type Support Syntactic Validation Ensures accuracy of the grammar / syntax of the XML label against the appropriate schemas as well as any mission schema or schematron. Syntactic validation is largely complete. Semantic Validation Ensures accuracy of the semantic "meaning" of the XML label when run against all relevant schema and schematron files. PDS has about 350 schematron rules today that are in place. Content Validation Ensure accuracy of data against the description of the data in the label. Some content validation in place, but will be the focus of expanded support. Referential Integrity Ensures accuracy of relationships described in Bundle and Collection products in place today. Future Feature – Ensure that the relationships described, in and between observational and document objects described in the XML label, are consistent and represented. November 29, 2017 Tool Working Group 30
Upcoming Tool Topics • Search Tools – Tracking and integration of search tools across PDS. • Migration Tools and Support – Tracking and planning of PDS-wide tools and approaches to support PDS 3 to PDS 4 migration needs.
Open Source • All software related to validation has been approved for release to open source. – This includes the PDS 3 and PDS 4 validation libraries as well as the PDS 4 Tools library. • These packages will be migrated to the nasa-pds project [1] on Git. Hub. – Plan to have source code migrated in February 2018 and discuss at the Tech Session [1] https: //github. com/orgs/nasa-pds/ November 29, 2017 Tool Working Group 32
Container Deployments • EN is investigating the use of container technology for deploying PDS 4 software – Mainly focused on PDS 4 services and not necessarily tools • Containers are different from virtual machines (see next slide) • Goal is to simplify deployments at the EN as well as the Nodes • Docker is a technology commonly in use at JPL and will be used in the first prototype – Will be included in tech session November 29, 2017 Tool Working Group 33
VMs vs Containers November 29, 2017 Tool Working Group 34
Next Steps • Coordinate use and governance rules around open source • Schedule search tools discussions • Schedule migration tools discussions • Continue scheduling DN tool discussions • Evaluate container technologies to improve software installations • Develop PDS 3 tool maintenance plans November 29, 2017 Tool Working Group 35
Questions/Comments November 29, 2017 Tool Working Group 36
Inspection Requirements November 29, 2017 Tool Working Group 37
Transformation Prioritization November 29, 2017 Tool Working Group 38
Validation Prioritization November 29, 2017 Tool Working Group 39
Validation Prioritization cont. November 29, 2017 Tool Working Group 40
Validation Prioritization cont. November 29, 2017 Tool Working Group 41
Charter • Charter – – – – – Maintaining the PDS Level 3 requirements for tools; Reviewing and overseeing the PDS-wide tool inventory; Identifying gaps and overlaps in tools; Recommending tool priority and phasing for development; Reviewing and commenting on specific tool requirements; Maintaining PDS-wide Tool Schedule and Plan; Supporting beta testing of tools as part of a release cycle; Hosting a tool summit with the PDS technical group; and Provide regular reports at the Management Council Face-to. Face meetings. November 29, 2017 Tool Working Group 42
Members • • D. Crichton, Chair (E. Law as alternate) E. Guinness, Geosciences Joni Johnson, Atmospheres Todd King, PPI Jordan Padams, IMG (Moses Milazzo as alternate) Eric Palmer, PSI Tanya Lim, PSA * Sean Hardman, EN as ex-officio advisor November 29, 2017 Tool Working Group 43
Tool Registry • The application is available at: – http: //pds. nasa. gov/tools/tool-registry • Migrations of the PDS Search Tools and IPDA Tools have been completed. • Beyond migrated entries, approximately 28+ new entries have been captured (current total of 75). – A couple new entries in the last couple of months. • Submissions have been reviewed by the TWG. November 29, 2017 Tool Working Group 44
Types of PDS 4 Validation Type Definition Syntactic Validation Validate the grammatical structure / syntax of the XML label Semantic Validation Validate that the interpretation of the values in the XML labels are consistent with the circumstances and contexts being described Content Validation Validate that the syntactic and semantic content of the XML label accurately describes the digital bits in the data objects being described (i. e. , the physical structure and order of the bits are accurately described by the data objects in the xml label) Referential Integrity Validate that the relationships described, in and between digital objects described in the XML label, are consistent and represented November 29, 2017 Tool Working Group 45
- Slides: 45