NATIONAL CENTERS FOR ENVIRONMENTAL INFORMATION Send 2 NCEI

NATIONAL CENTERS FOR ENVIRONMENTAL INFORMATION Send 2 NCEI: Fostering Producer-Archive Propinquity. . Kenneth S. Casey, Ph. D Deputy Director, Data Stewardship Division John Relph Technical Development Team Lead, Software Engineering Branch, Data Stewardship Division National Centers for Environmental Information (NCEI) June 11, 2015

The NODC Mission - 2014 “… organized for the purpose of acquiring, compiling, processing and preserving oceanographic data for ready retrieval” National Centers for Environmental Information (NCEI) June 11, 2015 2 2

The NCEI Mission - 2015 6: National Services and Intl. Leadership 5: Authoritative Records NCEI Levels of Stewardship 4: Derived Products 3: Scientific Improvements Focus here today 2: Enhanced Access and Basic Quality Assurance 1: Long Term Preservation and Basic Access “… responsible for preserving, monitoring, assessing, and providing access to the Nation’s treasure of environmental data and information. ” National Centers for Environmental Information (NCEI) June 11, 2015 3 3

er! ind em Open Archive Info. Sys (OAIS) R Common Services Preservation Planning P R O D U C E R S Data Management Queries/Results Descriptive Info SIP Ingest Access Archival Storage AIP DIP AIP C O N S U M E R S Administration MANAGEMENT DIP/SIP = Dissemination/Submission Information Package National Centers for Environmental Information (NCEI) June 11, 2015 4 4

Ingest - Two Primary Methods Common Services Preservation Planning P R O D U C E R S Automati Data Management on Queries/Results Descriptive Info SIP Ingest Access Archival Storage AIP DIP AIP Send 2 NC Administration EI. C O N S U M E R S MANAGEMENT National Centers for Environmental Information (NCEI) June 11, 2015 5 5

Send 2 NCEI • A Producer-friendly interface for collecting metadata and uploading data files to the Archive as a well-formed SIP, and. . . • An internal set of services for inserting the wellformed SIP into the existing Archive workflow • Intended for “one-off” submissions National Centers for Environmental Information (NCEI) June 11, 2015 6

Archive “Swim Lanes” Producers Data Content Managers and Data Officers Consumers Data Content Managers (DCMs) do the bulk of the archive workflow. Data Officers (DOs) assign the submissions to DCMs, and provide an independent final review. National Centers for Environmental Information (NCEI) June 11, 2015 7 7

Archive “Swim Lanes” Making it simple here. . . … makes it harder here* * and some things, like file integrity checks, are basically impossible National Centers for Environmental Information (NCEI) June 11, 2015 8 8

Send 2 NCEI • Addresses two of the three SIP heterogeneity problems* ○ unstructured/missing metadata ○ variable transmission • Enforces structure while still attempting to keep it simple and easy for the Producer • Improves overall quality and integrity • Reduces the DCM level of effort (and to a lesser extent, * The third, varying file formats, is being addressed by the NODC net. CDF templates the DO’s) National Centers for Environmental Information (NCEI) June 11, 2015 109

Send 2 NCEI Producer UI Flow Create new account new submitter Edit Account Profile Track Submitted Login existing My Submission Packages Create New Edit Current forgot password Retrieve password Change password View Archived 1. People & Projects 2. Dates & Locations 3. Data Types 4. Package Description 5. Upload & Submit Store Metadata National Centers for Environmental Information (NCEI) Store Data Send Confirmation Email, Update Tables 11 June 11, 2015 10 1

Login • • • Login or create account (< 3 minutes) Meets numerous federal requirements (password complexity and recovery, “fine print” legalese, etc. ) Introduces user interaction model National Centers for Environmental Information (NCEI) June 11, 2015 11

My Submission Packages • • • “Control Panel” Establishes user presence “in app” Create new packages Edit/delete active packages View submitted packages Link to archived packages National Centers for Environmental Information (NCEI) June 11, 2015 12

1. People & Projects ● ● Well defined user interaction model (autocomplete, instant validation, repeating entries, etc. ) Multiple persons, multiple ISO roles Projects Funding Agencies National Centers for Environmental Information (NCEI) June 11, 2015 13

2. Dates & Locations • • Dates Locations Platforms Sea Areas/Regions National Centers for Environmental Information (NCEI) June 11, 2015 14 1

3. Data Types • • • Variables Units (UDUNITS) Instrument Observation category Can include methods and data quality info National Centers for Environmental Information (NCEI) June 11, 2015 15

4. Package Description • • • Title (optionally autogenerated) Abstract Authorship List for DOIs Purpose References National Centers for Environmental Information (NCEI) June 11, 2015 16

5. Upload & Submit • • • Upload files Can separately provide some or all files if needed Comment on overall package Validate package prior to upload/submit Newbies averaged less than 20 minutes! National Centers for Environmental Information (NCEI) June 11, 2015 17

new submitter Store Metadata Track Submitted Create new account Edit Account Profile Edit Current Store Data 5. Upload & Submit 4. Package Description 3. Measured Parameters 2. Dates & Locations 1. Responsible Persons/Projects Create New My Submission Packages existing Login Change password Retrieve password Send Confirmation Email, Update Tables View Archived forgot password Send 2 NCEI Internal UI Initial Appraisal by Data Officer of the Week (DOW) XML + data 1. S 2 N creates Help Desk Ticket (HDT) 2. HDT links to package metadata 3. DOW appraises and selects appropriate Unit/Team 4. Unit POC/DO assigns Data Content Manager (DCM) National Centers for Environmental Information (NCEI) June 11, 2015 18 1

Send 2 NCEI Internal Process Initial Appraisal By Data Officer of the Week (DOW) 1. S 2 N creates Help Desk Ticket (HDT) 2. HDT links to package metadata 3. DOW appraises and selects appropriate Unit/Team 4. Unit POC/DO assigns Data Content Manager (DCM) National Centers for Environmental Information (NCEI) S 2 N Stages Data and Metadata for DCM Normal Archive Procedures Followed 1. AIP created in ingest area 1. 2. Data and Producer’s ISO XML copied to accession DCM reviews and interacts with Producer as needed 2. ATDB Brief Access Record prepopulated DCM requests “archive-published” status from DO 3. HDT updated and DCM notified to begin work DO reviews and approves, or iterates 4. Producer emailed and My Submission Packages updated 3. 4. 5. Producer emailed and My Submission Packages updated June 11, 2015 19 1

Send 2 NCEI Internal UI National Centers for Environmental Information (NCEI) June 11, 2015 20 2

new submitter Store Metadata Track Submitted Create new account Edit Account Profile Edit Current Store Data 5. Upload & Submit 4. Package Description 3. Measured Parameters 2. Dates & Locations 1. Responsible Persons/Projects Create New My Submission Packages existing Login Change password Retrieve password Send Confirmation Email, Update Tables View Archived forgot password Magic Happens Here! Initial Appraisal by Data Officer of the Week (DOW) XML + data 1. S 2 N creates Help Desk Ticket (HDT) 2. HDT links to package metadata 3. DOW appraises and selects appropriate Unit/Team 4. Unit POC/DO assigns Data Content Manager (DCM) National Centers for Environmental Information (NCEI) June 11, 2015 21 2

What Magic? ● ● Human web interface only loosely coupled to backend processing All sorts of extensibility is possible ○ Program-specific “tabs” could require additional metadata ○ Other machine systems could bypass web interface entirely ○ Web interface could be used to establish new collection level record for a future automated data stream ○ Pre-populate fields from an ISO record or a net. CDF file ○ Other/additional vocabularies can be included in auto-complete fields. . . National Centers for Environmental Information (NCEI) June 11, 2015 22

Send 2 NCEI Status ● ● ● Paperwork Reduction Act approval granted Public comments collected and responses published Released on April 22, 2015: https: //www. nodc. noaa. gov/ s 2 n/ Integrate into NCEI Archive workflow Integrate into ATRAC? National Centers for Environmental Information (NCEI) June 11, 2015 23

John Relph Technical Development Team Lead, Software Engineering Branch, Data Stewardship Division Kenneth S. Casey, Ph. D Deputy Director, Data Stewardship Division National Centers for Environmental Information (NCEI) 151 Patton Ave, Rm 420, Asheville, NC 28801 National Centers for Environmental Information (NCEI) June 11, 2015 2 24

Producer Confirmation Email National Centers for Environmental Information (NCEI) June 11, 2015 25 2

Help Desk Ticket National Centers for Environmental Information (NCEI) June 11, 2015 26 2